Hand tracking was a new feature revealed for the original Oculus Quest during the Oculus Connect presentation last fall. It was announced as an experimental feature for the Quest with a tentative 2020 release date but ended end getting released in December as an experimental feature. With the release of the Quest 2, hand tracking has now been implemented as a feature right out of the box. Hand tracking presents an interesting opportunity for new ways to interact with games, but there's still got a long way to go before it's anything more than a novelty.
Hand tracking works by way of the 4 cameras positioned on the corners of the headset. As you can imagine, the tracking works best when the cameras can see your fingers. When the cameras can't see your fingers, however, things quickly fall apart.
When you boot up the Quest 2 you can use hand tracking immediately to navigate the menu using simple gestures. Holding your hand up directly in front of you creates a cursor you can use to select menu options. Touching your finger and thumb together (make the OK symbol) will click on something as long as your cursor is hovering over it. The gesture can change where your cursor is pointing though, so it may take a couple of tries to click accurately. You can also pinch the screen with the same gestures to scroll around, though scrolling too fast can make things go haywire.
In fact, doing anything too fast with hand tracking can cause it to completely fail. You need slow, deliberate actions to interact with the virtual screens, remembering to keep your fingers in full view of the cameras. Bringing your hands too close or too far from the headset will also lead to poor tracking. The sensitivity and inconsistency of the tracking makes it feel less like Minority Report and more like trying help a drunk ghost keep its balance.
You can browse through menus to your heart's content, but eventually, you'll want to actually open an app and play a game. As soon as you do, you'll be prompted to pick up your Quest 2 controllers. What's surprising is that this goes for streaming apps as well. Amazon Prime, Youtube, and Netflix all require you to pick up at least 1 controller to navigate their menus and start watching shows. If I could put the Quest 2 on and start watching a show without ever needing a controller I would applaud the hand tracking, but that simply isn't the case and I can't for the life of me understand why. Perhaps this is something Netflix and Amazon need to implement on their end, but you'd think Oculus would be keen to have that ready for the launch of their new and improved headset.
Of course, the more interesting application for hand tracking is in unique gameplay experiences, and for that, we have Waltz of the Wizard: Enhanced Edition for Oculus Quest 2. There are other hand tracking-enabled games (I'm particularly fond of the Curious Tale of the Stolen Pets) but Waltz of the Wizard is by far the best tech-demo for what hand tracking can offer VR games, as well as what its limitations are.
Waltz of the Wizard is one of those classic playroom-style VR games that excels at demonstrating the possibilities of new technology. What Job Simulator did for modern VR, Waltz of the Wizard does for hand tracking. At times, Waltz of the Wizard can be quite profound. I had some real moments of awe while playing and it renewed my hope in the potential of hand tracking. It's also clunky, janky, and highlights some of the most glaring issues that hand tracking still needs to overcome.
The first big problem is locomotion. Without a stick to push, a new way to walk around needed to be invented. In Waltz of the Wizard, you walk around by holding your hand in the finger-gun position with your palm up. This creates a line you will draw on the floor ahead of you with your index finger. When the route you want to walk is complete you then pinch your thumb to your index finger and you'll begin walking along the path. When you want to stop moving just put both hands out in front of you like you're pushing someone.
It's strange, unintuitive, and I suspect this won't be the control scheme that sticks around in future hand tracking experiences, but it works. I'd prefer to simply bow my head to move forward or something because I never quite got the hang of this process.
Once you're standing in front of the Wizard's table, things don't get much better. You'll first need to pick up objects and combine them in a cauldron to create spells. Picking things up is by far the worst application of hand tracking. With your fingers naturally pointed away from the cameras and nothing to actually grab onto, the game does a horrible job of determining when you want to pick something up. Even though you can see your ghost hands wrap around the object, grabbing them and picking them up is a frustrating nightmare. It's better to hover your hands over the object until they highlight and then make a fist to pull them towards you.
Casting spells is a mixed bag, but some of them feel really satisfying. You can snap your fingers to make objects explode, for example. It's not the easiest spell to aim but it's such a unique interaction. It feels like actual power when you make something explode just by snapping, and it doesn't hurt that the tracking is really consistent with this gesture.
My other favorite gesture is extending my index and pinky finger to create a slingshot that stretches between them. The in-game fingers extend unnaturally long and you can use your other hands to pinch the slingshot, pull it back, and fire little balls of light at objects around the room. It's the only spell that changes the form of your hands but there is something about that transformation that really tickles my brain every time it happens.
As expected, the less reliable gestures are the ones that aren't nearly as much fun. Anything that requires fine motor skills like picking things up, touching things, or throwing things can make you feel like you aren't really in control of your own body. On the other hand, shooting lightning from your outstretched fingertips or magically teleporting objects to your outstretched hand feels incredible.
If hand tracking games lean into simple, reliable gestures I think there's a lot of opportunity for more interactive experiences like Waltz of the Wizard. Maybe there's even space for one-controller games that let you hold the controller in your right hand to represent holding a gun but keep your left hand free to interact with the world naturally. Count me in, especially if I can still snap my fingers and make things explode.