A video game doesn't need to be realistic, but it should at least make sense. That doesn't necessarily refer to the game's story - although things like plot holes are obviously to be avoided. Rather, the fundamental features of how the virtual world reacts and is reacted to by the player must be believable, in order for full and sustained immersion to take place.
As humans we have established expectations around how the real, physical world is and how it reacts around us - even from early on in infancy. For example, even young babies generally understand that one physical object cannot pass through another, or an object bigger than a particular container cannot fit into that container.
These intuitions are carried over into the game worlds we interact with, too. So, in a game’s virtual world physical events ought to make sense, even if the virtual world itself or its characters are unrealistic in their appearances (i.e. are cartoonish) or have fantasy elements. They have to be credible, or otherwise they run the risk of distracting a player and thus breaking their immersion in the world.
So take for example, the sounds of the character's footsteps on metallic surfaces should differ clearly from those on gravel or sand. Or a car driving into things should reflect that it's getting visibly damaged over time. If small things like these are executed correctly, the player won't even notice that they're happening. They will serve to smooth out our experience of a game by satisfying the very basic - often subconscious - expectations we have about how the physical world acts and reacts.
Of course, there are reasonable exceptions to this rule. If a game's character is a ghost, then moving through solid objects is expected of them. And if a ghost is seen acting in accordance with how lowly earthbound beings act – like opening a door or grasping and eating an apple – then that wouldn’t be credible, thereby breaking immersion.
This is the kind of thing that happens in The Elder Scrolls: Oblivion, for example, where ghosts shuffle and plod along the ground in exactly the same way as their living counterparts do. But it's hard to be too harsh considering the game's age, and the fact that it's so great in almost every other aspect. Sometimes a game's entertainment factor can be strong enough that we can forgive little gripes like this.
Ultimately, though, such gripes can add up. Players can become fixated on them, to the extent that that delicious sense of immersion and presence within a virtual world will be interrupted. So these little things? They make all the difference.
Source: The Gamer's Brain