Visemes

Making your avatar’s mouth move and respond to how you are talking is a common avatar feature. One method of doing this is to use Visemes. A Viseme is a blend shape that's designed to resemble the way a person's mouth moves when saying a specific or set of specific Phonemes.

Resonite take's your incoming voice information and analyzes it, based on that analysis it then controls your Viseme blend shapes on your avatar to make your avatar's mouth try to match your actual mouth.

In Resonite we use a fairly standard list of Visemes loosely based on similar platforms and products in the space, The Visemes we use can be found below:

Viseme Name	Phonemes	Example Phrases / Words
AA	A	Car
CH	tS. dZ. S	chair, join, she
DD	t, d	tip, doll
E	e	bed
FF	f. v	fat, vat
IH	ih	tip
KK	k, g	call, gas
NN	n, l	lot, not
OH	oh, o	toe
OU	ou, u	book
PP	p, b, m	put, bat, mat
RR	r	red
SS	s, z	sir, zeal
TH	th	think, that
Silence	neutral	Silence

Viseme Allocation

Resonite supports two options when it comes to setting up Visemes for your avatar:

• Automatic - Requires specific blend shape names.
• Manual - Can use any blendshape names

Automatic Allocation

Automatic allocation, will happen when you create the avatar using the Avatar Creator, ensure you check “Setup Face Tracking”.

Then when you create the avatar, Resonite will try to guess the correct blend shape for each Viseme. Our algorithm, aims to find most common naming schemes including from other platforms and we improve it overtime adding new items into the algorithm to aid users.

To help explain how the automatic allocation works, here’s a table that details what we look for:

Viseme Name	Accepted Names	Example Names we recognize
AA	Any segment contains aa, a,ah or あ	vrc.a, vrc.ah, vrc.a, a, ah, ah, あ
CH	Any segment contains ch or sh	vrc.ch, vrc.sh, ch, sh
DD	Any segment contains d or td or dd	vrc.d, vrc.td, d, td
E	Any segment contains e, ee, え	vrc.e, vrc.ee, e, e, え
FF	Any segment contains f,ff,fv	vrc.f, vrc.ff, vrc.fv
IH	Any segment contains ih, い	vrc.ih, ih, い
KK	Any segment contains kk,k	vrc.kk, kk, vrc.k, k
NN	Any segment contains nn,n,ん	vrc.nn, vrc.n, n, ん
OH	Any segment contains o,oh,oo,uh,お	vrc.o, vrc.oh, vrc.uh, uh, oo, uh
OU	Any segment contains ou,u,う	vrc.ou, vrc.u u, ou
PP	Any segment contains pp,p	vrc.pp, vrc.p, p, u
RR	Any segment contains rr,r	vrc.rr, vrc.r r
SS	Any segment contains ss,s	vrc.ss, vrc.s s ss
TH	Any segment contains th,h	vrc.th, vrc.h, h, t
Silence	Any segment contains silence,sil	vrc.sil, silence, vrc.silence

When reading the table, keep in mind the following:

• Everything in the table below is case insensitive.
• We split all blend shape names by any non-letter character into segments for searching.
  • For example “vrc.ah” gets split into two segments “vrc” and “ah” for searching.
  • For example “cheese is a good fruit”, would get split into segments that would make that blend shape get allocated to “AA” below.
  • For example “blendshape_monkey_a”, would get split into 3 segments with the last one “a”, matching the AA Viseme above.
• vrc prefixes are NOT required, they’re just really common. Do read #Allocation Prioritization though.
• You do not need to follow any naming, you can always do manual allocation instead.

Allocation Prioritization

Due to the prevalence of the prefix vrc. on a large amount of commonly found avatars, when a blend shape that contains the prefix vrc. is found, it takes preference over all over blend shapes for that one Viseme.

For example, if you have an avatar with a vrc.aa blend shape and an a blend shape, then the vrc.aa one will be allocated as it takes preference.

Re-running the Automatic Allocation

You can re-run the automatic allocation process at any time, by finding an Avatar’s DirectVisemeDriver and pressing the “Auto-assign targets” button.

Manual Allocation

TODO

See Also

• VisemeAnalyzer
• DirectVisemeDriver