While more relevant to OSR By Night, this could be applied to a more horror styled fantasy game as well.
These rules are intended to mainly affect the current situation, and are less gruesome than the effects in some popular horror role playing games.
When facing an encounter that is particularly horrifying, mind bending or otherworldly, whether a monster, a god or a particularly surreal location, you can make a Panic And Sanity check.
This is handled as a saving throw against spells. Class bonuses that apply to all saving throws, or to saves against mental attack specifically will apply, but magical items generally do not, unless the GM determines the specific item should.
Racial bonuses do not apply, unless, again, the GM determines they should.
Depending on the monsters level, three different effects may occur:
If the monster encountered is higher level/hit dice than the character, a save is required, to prevent panic for 1D6 rounds. While panicked, the character may take very simple, instinctive actions, such as firing a weapon blindly, running away, hiding or remaining motionless.
A companion may try to shake the character "out of it". This takes a round, and permits another saving throw to be attempted.
If the monster is 4 levels above the character, two saving throws must be taken. Both dice can be rolled at the same time.
If one of the saves succeeds, the character goes into a panic, as described above.
If both should fail, the character has a nervous breakdown. This is treated as a perpetual state of panic, though a saving throw is permitted to get 1D6 rounds of rational action before having to save again.
Nervous breakdowns take 1D6 days to recover from, where no monsters are encountered.
If a monster is 8 levels above the character or more, three saving throws must be made.
Passing 2 will inflict panic, as described above.
If only 1 save is passed, the character is subject to a nervous breakdown as above.
Should all 3 saving throws fail, the character goes mad. A mad character will act based on a warped logic, determined by the GM, though the players input should be taken into account.
In most cases, they will become ineffectual and can generally not use any skills, though they'll fight to defend themselves if needed.
Madness permits a 1D6 roll every week, with a roll of 1 letting the character recover.