Face and Eye Tracking (Avatar)

This is for avatar setup, for software/hardware setup, see: Face and Eye Tracking (Setup)

This page is mostly a bunch of notes to gather info about the systems related to face and eye tracking. This is not (yet) a tutorial.

Due to the wide range of avatars that exist, setting up facial tracking can be either fully automatic or quite the complicated exercise. This page seeks to detail how the Face and Eye Tracking work inside Resonite and how to set it up to work with avatars of different types. As this It also helps with Create and maintain documentation for face/eye tracking So any input is welcome.

On avatar equip, AvatarMouthDataSourceAssigner/Component:AvatarEyeDataSourceAssigner makes sure the raw data your facial tracker gives is sent via MouthTrackingStreamManager/EyeTrackingStreamManager to the appropriate driver component. From there it drives either bones, blendshapes or can be accessed to make creative use of via Flux.

Avatar Setup

At creation

When creating the avatar, ensure that "Setup Eyes" and "Setup Face Tracking" is selected in the Avatar Creator#Avatar Creator Panel

Avatars set up via this process have the required components added to the same slot as the body mesh and the head. (?)

Check to make sure the blendshapes are detected and set up correctly and change as required. This process will not work with avatars with 2D eyes or with unconventional names.

After creation

Component:AvatarExpressionDriver has: "AutoAssign Blendshapes" button

The avatar needs to be re-equipped after these components have been added to populate the DataSourcefields and enable tracking.

Components

Most useful

• Component:EyeManager - For bone based eyes, tracked and simulated.
• Component:EyeLinearDriver - For converting EyeManager to blendshape and texture based eyes.
• Component:AvatarExpressionDriver - Drive mouth/face blendshapes directly using tracking inputs
• Component:AvatarRawMouthData - Show/debug incoming mouth data as well as an easy way connect ProtFlux
• Component:AvatarRawEyeData - The matching component for eye data.

Handy

• Component:MockupMouthTrackingSource - A way to simulate having face/mouth tracking.

Flux

Example of using AvatarRawMouthData to drive flux examples, for missing blendshapes on simplified models.

Or if a model only has Visemes for example.

Avatar Examples

Default Avatar

FaceBot

All of these avatars have set up face and eye tracking:

• Default Avatar - Smiles too much. Has no tongue
  • This is an example of an avatar with a mostly 2D based expressions.

• Face Tracking FaceBot Avatar World - NeosVR Themed, fully functioning
  • This avatar uses conventional bone and blendshape based expressions.

• "Flat Face Tracking Avatar" - Unchecked
  • Gearbell: http://redx.adamski2003.lol/parent-link.brson?ownerId=U-GearBell&id=R-a878a956-3df2-49d7-bcce-c556ea4f3afe&v=2&depth=2

THE MISSING 2D FACE TUTORIAL

Tutorial:2D Viseme Textures

External Links

References

• Face Tracking Conversion w Resonite added (Google doc)
• VRCFaceTracking Unified Expressions
• facetracking_cheatsheet_NEOS_VERSION.png (Discord)
• List of surported face tracking names (Discord)
• Component:AvatarExpressionDriver#Expression For a list of surported expressions with examples.

Resonite GitHub

• Add VRCFT Unified Expression names to Face Tracking Heuristics

See Also

Category:Components:Users:Common Avatar System:Face - All facial related components

Category:Components:Users - The components that source the data used