Broadly speaking, horror caters to our fear of death and pain. But as the genre bleeds into fears that are more about sanity and our own sense of mental well-being, it begins to stretch into the subgenre of Psychological horror.
This subgenre mixes with nearly all other genres in some way, and often is enhanced by a nice mixture. The Shining is an example of a haunted hotel ghost story blended with a man slowly growing insane, i.e. psychological horror. We find similar examples in Roman Polanski films Repulsion and Rosemary’s Baby. The same can be said for more modern examples like The Babadook and The Lighthouse.
Perhaps the most common direction to take the genre is to depict someone slowly growing mad. But mystery horror such as Se7en and Silence of the Lambs depict psychological themes while focusing on the investigators rather than the insane killer in the middle of the story.