People like being scared. There's a thrill that comes with being on the edge of your seat, an adrenaline rush you get when you're creeping down a hall and you hear a sudden sound come from behind you. It's like a drug, and some, like me, have been addicted to it since we got our first taste. It's been years since a video game last gave me a nightmare, but I remember that dream even today. After a night of playing Resident Evil 3 with a friend, I dreamt the two of us were being chased through a labyrinthine house by a massive, hulking shadow. It was terrifying, and had pretty obvious Resident Evil influences, but a substantial part of me enjoyed it. Then, over the years, I grew to love that rush that only the horror genre offers.
The "fear factor" of any game is entirely relative to the person playing it. I played through Amnesia: The Dark Descent and enjoyed it, despite its impressive ability to be consistently terrifying even to someone who's played a majority of the games the horror genre has offered over the last decade and a half. After being thoroughly traumatized by Amnesia, I recommended it to a fellow fan of the genre. A week later, this friend came to me after finishing it only to tell me how boring it was. I was stunned, but that only proves that, like comedy, horror is something we all experience in different ways.
With that said, even if you don't necessarily find them terrifying, some games are surely scarier than others. If I were to break the genre into a few tiers, this is how I'd go about it. The first tier would consist of the newly established action/horror genre. This includes games like Resident Evil (post RE4), Dead Space, Left 4 Dead, and Condemned. These games tend to rely on "cheap" scares like sudden loud noises and things bursting out at the screen and/or your unsuspecting character. These are horror-themed action games that, more often than not, give you control of an extraordinarily overpowered character. The scares are numerous, but short-lived, and unlikely to inspire any real nightmares.
Then, things start to get much darker.
In the second tier, we have games that give you a usually underpowered character. You can defend yourself, sure, but sometimes it's better to cut your losses and embrace the flight over fight instinct. Fatal Frame, the early Resident Evil games, and the Silent Hill series, sans Homecoming, are all known for this. The scares are less about sudden thrills and more about instilling you, the player, with a deep sense of fear. The scares are psychological, like the iconic Pyramid Head, which doesn't really look like anything in reality. Of course, you could say that about almost every creature that calls Silent Hill its home, but a few really stand out.
As I mentioned earlier, games like these have given me nightmares. I can remember a few, like the one inspired by my time with Resident Evil 3. Even so, there's a tier above these that freaks me out every time.
The third and final tier would consist of the scariest games, like Amnesia: The Dark Descent, where you're utterly defenseless against the things that hunt you. I'm particularly fond of the games that leave a majority of the scares to your imagination, because my imagination has become twisted by all the horror films and games I've seen and played since I was six. In The Dark Descent, you're punished for lingering too long to get a look at the monsters as every second spent looking at them weakens your already loose grip on your sanity.