Tracking in Virtual Reality and Beyond – VR 101: Part III

Tracking is one of the key elements of VR. It’s a big reason why virtual reality and other XR technologies are so immersive.  

We discussed the concept of immersion in part I of our introduction to VR and the headset specs that contribute to immersion in part II. Now we’ll explore how tracking lets you interact with the VR environment. 

Get ready to discover all the different ways you can move in virtual reality!

What is VR tracking?

Tracking in VR is the continuous detection of the position and orientation of objects within a physical space. Those objects may include a VR headset, its controllers, external devices like trackers, or even parts of your body.  

As each object’s position and orientation change over time, this is registered as movement virtually, and tracking can be thought of as how a VR system collects information about your many movements. 

Accurate, precise tracking is an important component of spatial computing and necessary to enable realistic interactions in VR, AR, and MR. It’s for this reason that innovations in tracking technology like the VIVE Ultimate Tracker are key to enhancing extended reality experiences. 

The degree of immersion offered by a VR headset is correlated with its tracking capabilities.

How does VR tracking work? 

By pinpointing exactly where the HMD, controllers, and other accessories are located and how each is oriented, a VR headset can translate your body’s movements in the real world to corresponding movements in the virtual world. 

For example, if you were gazing down at a river inside a rainforest in VR and then turned your head upwards, tracking is what would cue the display to shift the image from the water in front to the tops of the trees above. 

If you then raised up the controller in your hand, tracking would allow you to see your hand in VR reaching for the clouds. 

There are many types of tracking in VR and different ways to achieve them. Some tracking systems only read the movements and positions of your head, while others may track your hands, eyes, and even facial expressions! These can make navigating VR more intuitive and enrich your overall experience. 

Tracking can be achieved internally (e.g., by the headset itself) or externally (e.g., by an object or set of objects separate from the VR headset). The ways a tracked object can move physically and virtually are expressed in degrees of freedom. 

Positional tracking devices like the new VIVE Ultimate Tracker can enhance VR gameplay.

What are degrees of freedom (DoF)?

Degrees of freedom (DoF) refers to the number of axes of physical movement that can be tracked by a VR headset. Real-world movements must be tracked before they can get translated into corresponding movements in VR.

An axis of movement is a unique way of moving through a 3D space. There are two categories of movement, rotational and translational, with each having three axes related to the x-, y-, and z-axes you already know.

Rotational axes are the three different ways a physical body can rotate.

When you tilt your head so that your ear approaches your shoulder, you are performing a kind of rotation called rolling.

When you look left and right before crossing the street, your head is rotating in a manner known as yawing.

And when you look up at the sky or down at your feet, this rotation is called pitching.

Rolling, yawing, and pitching are the three rotational axes that degrees of freedom can measure.

Sliding and rotating an object in a 3D space may help you better understand degrees of freedom.

Translational movement is advancing in one particular direction.

When you shuffle your body to the left or to the right, this kind of translation is called strafing.

When you jump up and come back down, your body is elevating.

And when you step forward or backward, this is called surging.

Strafing, elevating, and surging are the three translational axes that degrees of freedom can measure.

What does 3 degrees of freedom (3DoF) mean?

In VR, 3DoF means that only three axes of movement can be tracked by the device — namely, the rotational axes of rolling, yawing, and pitching described above.

In other words, the 3DoF VR headset you’re wearing will register all the different ways you may turn your head to allow you to look around the 360-degree environment freely. However, since translational movements are not tracked with 3DoF, the device will not recognize when you step left, right, forward, or backward, or when you jump up or crouch down.

Many kinds of VR experiences are perfectly suited to 3DoF. For example, 360° videos present a set series of visuals that you can enjoy from any angle — you wouldn’t be walking into them, though.

A 3DoF VR device could also be used sitting in a car or on a train since your movement in the virtual world wouldn’t be disrupted by your being propelled forward in real life.

What does 6 degrees of freedom (6DoF) mean?

A VR headset with 6 degrees of freedom (6DoF) can track all six axes of movement: rolling, yawing, pitching, strafing, elevating, and surging.

The inclusion of translational movements quite literally adds a new dimension to the way you can enjoy VR. You could take a step forward to make your avatar walk in VR or shift to the right to dodge a virtual obstacle coming your way.

Having 6 degrees of freedom makes virtual reality a lot more immersive, but it also means you need to choose your movements in the real world a bit more carefully — they could all affect your location and position in the virtual space.


Precise tracking lets everything in your view appear where it should be.

What is inside-out tracking?

Inside-out tracking is any kind of tracking in which a VR device detects its own movements in space. This is generally achieved through several outward-facing cameras or sensors on the headset that gather data from the environment, which the headset uses to determine its location and the locations of any other tracked objects (like controllers). Typically, this type of tracking does not rely on any external sensors and is considered an all-in-one solution.

You can see inside-out tracking in standalone VR headsets like the VIVE Focus 3 and VIVE XR Elite, which conveniently don’t need to be connected to a PC or any other equipment. It is, however, also possible to get inside-out tracking in PC VR headsets like the VIVE Cosmos Elite and even in VR accessories — as with the VIVE Ultimate Tracker.

The opposite of inside-out tracking is, not surprisingly, outside-in tracking.

What is outside-in tracking?

Outside-in tracking is the detection of a VR system’s orientation and position by one or more external devices. External tracking devices are typically placed at strategic locations in the surrounding environment and use sensors to capture information about your movements within that space. This information is relayed to your headset in real time.

The advantage of outside-in tracking is that it is not limited by the range of a VR headset’s built-in cameras or sensors, which, during a virtual experience, might get blocked by real-world objects or your limbs (a problem known as occlusion). To be effective, the external tracking devices need to be placed where they won’t get in your way, such as in the top corners of your room, and even then, their sensors are still at risk of being occluded by you or things in your environment.

You may have seen the Lighthouse tracking system used by HTC VIVE PC VR headsets cited as a classic example of outside-in tracking. While this system does rely on external devices placed in the player’s immediate area, those devices don’t actually collect any information themselves.

Let’s take a closer look at the Lighthouse system to show you what we mean.

What is Lighthouse tracking?

Also referred to as SteamVR tracking, Lighthouse is a positional tracking system in which small, rectangular devices called base stations flash a play area with precisely timed infrared lasers, which are detected by objects like HMDs (VR headsets) and used to determine their location and orientation.

The base stations’ infrared lasers flash across a room several times per second in alternating horizontal and vertical sweeps. Each tracked object, meanwhile, has an array of sensors called photodiodes that recognize this light — as well as an internal chip that continuously calculates the object’s position by using the known physical layout of its photodiodes, the known timing of each infrared laser flash, and the specific time at which each particular photodiode is hit.

While the light emitted by base stations is invisible to the human eye, it can be detected by SteamVR-compatible devices like the VIVE Pro 2 headset. The consistent movement and frequency of each laser’s movement allows time to be used as a reliable measure of the angle and distance from the base station to the tracked object.

Provided that the play area is properly set up, base stations enable high-precision tracking within larger environments than are possible with other tracking systems, allowing users to roam freely within them. This kind of experience is known as room-scale VR.

The VIVE Pro 2 combines power and precision for incredible PC VR experiences.

What is room-scale tracking?

Room-scale tracking refers to the ability of a VR system to track the movement of objects within and throughout a 3D physical environment, with space in the real world corresponding to space in the virtual world.

A simple example of room-scale tracking is a user moving about a VR environment by actually walking around his or her play area. Taking a step forward in the play area also moves the user a step forward in VR, and this kind of movement is possible all throughout the defined play area.

The freedom offered by room-scale tracking differs from that of stationary experiences like those you might find in an arcade. In stationary VR, which features 3DoF tracking, you can look around with your headset and move your controllers around, but you can’t (or shouldn’t) leave the spot within your play area that you began playing in. Doing so would not only have no desirable effect on your virtual location but also put you at risk of colliding with something physically!

Without room-scale tracking, moving around a virtual space may require teleportation, which we’ll explain next.

What is teleportation?

Teleportation in VR is a means for you to instantly get from one virtual location to another. This could involve moving to another part of the same virtual area, such as from the bottom of a staircase to the top of it, or relocating to entirely different space, like when you leave a meeting room to return to your VR home.

The act of teleporting is achieved by pointing your cursor in VR either to a portal or to the specific spot you want your avatar to stand or sit in. Not every virtual location can necessarily be teleported to, though. Some kind of indicator, perhaps a ring of light on the floor, will typically show you where you are able to teleport to and where is off limits.

The movement of the cursor used for teleportation is typically determined by that of a controller in the real world.

There are many different approaches to tracking movements in VR. 

Do you need controllers for VR?

Using a pair of VR controllers is perhaps the most popular way to navigate virtual worlds and operate a VR headset, but different VR devices come with different control methods, and some headsets may only make use of one controller (or none at all).

Controllers are useful because they can be held comfortably and manipulated intuitively. In the VR environment, your controller’s movements and the pushes of buttons or triggers are all used to interact with objects and navigate menus.

Controller-free VR may involve using your sight to select points (such as by staring at them for extended periods of time) and move around (such as by turning your head). Generally, relying on only this kind of control method limits the extent to which you can interact with the virtual world.

A more promising alternative to controllers is the use of your bare hands. This requires a special kind of tracking that can read the movements and positions of your fingers.

Using your hands to navigate virtual reality can make some experiences a bit more intuitive.

What is hand tracking?

Hand tracking is the detection of the position and orientation of your hands and fingers, which can be used as a substitute for VR controllers and as a model for your VR avatar to copy.

With hand tracking, you can activate specific commands using pre-defined hand gestures like the “OK” sign instead of pressing a button. You can also intuitively point at and select things in VR and express yourself through body language. That means your friends can see you wave hello, clench your fists, or give a thumbs-up!

To replicate your real-world hand movements virtually, VR headsets must be able to see and recognize your hands and approximate where they are when they go out of sight. This can be achieved with a combination of onboard cameras and artificial intelligence (AI).

How is AI used in tracking?

AI can play a critical role in forms of tracking that use cameras. Computer vision is a kind of AI that allows a device to understand what it is looking at.

Training a device’s computer vision to learn how to decipher images fed to it from a camera involves another kind of AI called machine learning. Together, computer vision and machine learning help devices like VR headsets and trackers follow the movements of objects or parts of your body.

Eye and facial tracking show how VR and AI can work together to improve virtual interactions.

What are VR trackers?

Virtual reality trackers are devices that offer various kinds of tracking functionality. They allow VR headsets to track additional points in the real world and replicate those movements virtually.

VR trackers generally fall into two categories:

1. Devices that can have their own positions in a 3D space tracked by a VR system.
2. Devices that collect positional data about a separate object or a part of your body.

By pairing your VR headset with trackers, you can enjoy new kinds of interactions with your virtual surroundings. Movements recorded by trackers can serve as inputs to control XR experiences like VR games.

Object tracking adds a new level of realism to virtual reality experiences.

What is eye tracking?

VR eye tracking refers to the positional tracking of your eye movements for a virtual experience. It is achieved through cameras placed close to your eyes and the display.

By following your gaze, a VR headset with eye tracking can update what you see in real time and let your eye movements serve as inputs for more intuitive interactions. For example, eye tracking could allow you to select an item from a menu just by staring at it. It also serves as a way to animate your virtual avatar more realistically, showing others around you what direction you are looking in.


By bringing eye movements into VR, avatars gain a more human-like quality.

VR eye tracking can even save computing resources by rendering high-quality visuals only where your eyes are looking, a feature known as foveated rendering. Since you are unlikely to notice things in your peripheral vision, there is little need to render images there in high detail.

Some VR headsets like the VIVE Pro Eye come with built-in eye tracking. Others require an eye tracker, which is a kind of VR accessory that attaches directly to a headset.

The VIVE Pro Eye’s built-in eye tracking enabled new kinds of interactions in VR.

What is face tracking?

Face tracking involves following your facial movements for virtual experiences. By replicating your facial expressions and lip movements through your avatar, you can allow others in the virtual environment to better understand you and communicate more naturally.

Face tracking allows your virtual self to smile, laugh, frown, and speak just as you do in the real world, enabling true-to-life interactions. It is ideal for situations like VTubing where you present a digital persona to others.

Like eye tracking, face tracking is achieved through computer vision. Whereas eye tracking focuses specifically on your eyes, however, face tracking also involves your nose and mouth.

We use the term “face tracking” to describe positional tracking of the entire human face and the term “facial tracking” when talking specifically about tracking for your nose and mouth. These kinds of functionality either come built into a headset or are unlocked through VR accessories called face trackers or facial trackers. Like eye trackers, these add-on devices attach directly to a VR headset.

Face trackers make conversations in VR as natural as those in the real world.

What is full-body tracking?

Also known as whole-body tracking, full-body tracking is the virtual replication of your entire body’s movements, including those of your legs. It can be experienced using tracked devices like the VIVE Tracker (3.0) and VIVE Ultimate Tracker, which are worn on the body part you want to move in VR. These devices are sometimes called full-body trackers, but they can also be used for the tracking of objects.

To understand why full-body tracking is important for immersion, consider that most VR systems are only capable of tracking the movements of your head (via the HMD itself) and arms (via the controllers, wrist trackers, or hand tracking). This works fine for many games and immersive experiences, but limits how your virtual self can move in others.

If you were playing a soccer game, for example, it wouldn’t be very intuitive to kick the ball with your hands. And it would certainly be awkward to tie the controller to your feet! Body trackers offer you a simple and comfortable way to bring your kicks into the virtual field.


Just a few correctly placed VIVE Trackers are enough to realistically animate a full-body avatar.

Of course, your body is made up of many more points than would be practical for any modern VR device to track. For most experiences, however, a faithful replication of your body’s movements can be achieved by tracking just 6 points on the body: the head, left hand, right hand, waist, left foot, and right foot. You could use your headset and two controllers for the first three points and three body trackers for the remainder.

A VR headset can figure out how the rest of your body is positioned in between these tracked points using a process called inverse kinematics. This still involves a bit of prediction, though.

For even more precision than you’d get with 6-point tracking, you could also attach trackers to your chest, left elbow, right elbow, left knee, and right knee. This would be an example of 11-point tracking, which is helpful in cases where you want to faithfully recreate real-world movements through your avatar, such as for dancing or VTubing.

Try out different kinds of tracking in VR to see which serve your needs best!

What do I need for VR tracking?

If you’re ready to discover all the different ways you can move in virtual reality, the first thing you’ll need is a headset or pair of immersive glasses!

Beyond that, it all really depends on what kind of virtual experience you’re interested in. You can do a lot with just controllers or wrist trackers, but for more intuitive interactions with others, a complete face tracker may be the thing for you. And if you’re looking to perform or utilize your legs in VR, full-body trackers are what you’ll want.

(Speaking of full-body tracking, check out what you can do with the VIVE Ultimate Tracker, our new self-tracking accessory that's powered by computer vision.)

This inside-out tracker is sure to redefine how we interact in XR. 

We hope this overview has given you a clearer understanding of the critical role tracking plays in virtual reality. In our next post in this series, we’ll dive into the different kinds of VR that are available out there. Stay tuned!