Visemes

From Resonite Wiki
Revision as of 06:15, 17 June 2024 by ProbablePrime (talk | contribs)

Making your avatar’s mouth move and respond to how you are talking is a common avatar feature. In Resonite we use a fairly standard list of Visemes loosely based on similar platforms and products in the space, The Visemes we use can be found below:

Viseme Name Phonemes Example Phrases / Words
AA A Car
CH tS. dZ. S chair, join, she
DD t, d tip, doll
E e bed
FF f. v fat, vat
IH ih tip
KK k, g call, gas
NN n, l lot, not
OH oh, o toe
OU ou, u book
PP p, b, m put, bat, mat
RR r red
SS s, z sir, zeal
TH th think, that
Silence neutral Silence

Viseme Allocation

Resonite supports two options when it comes to setting up Visemes for your avatar:

  • Automatic - Requires specific blend shape names.
  • Manual - Can use any blendshape names

Automatic Allocation

Automatic allocation, will happen when you create the avatar using the Avatar Creator, ensure you check “Setup Face Tracking”.

Then when you create the avatar, Resonite will try to guess the correct blend shape for each Viseme. Our algorithm, aims to find most common naming schemes including from other platforms and we improve it overtime adding new items into the algorithm to aid users.

To help explain how the automatic allocation works, here’s a table that details what we look for:

Viseme Name Accepted Names Example Names we recognize
AA Any segment contains aa, a,ah or vrc.a, vrc.ah, vrc.a, a, ah, ah, あ
CH Any segment contains ch or sh vrc.ch, vrc.sh, ch, sh
DD Any segment contains d or td or dd vrc.d, vrc.td, d, td
E Any segment contains e, ee, vrc.e, vrc.ee, e, e, え
FF Any segment contains f,ff,fv vrc.f, vrc.ff, vrc.fv
IH Any segment contains ih, vrc.ih, ih, い
KK Any segment contains kk,k vrc.kk, kk, vrc.k, k
NN Any segment contains nn,n, vrc.nn, vrc.n, n, ん
OH Any segment contains o,oh,oo,uh, vrc.o, vrc.oh, vrc.uh, uh, oo, uh
OU Any segment contains ou,u, vrc.ou, vrc.u u, ou
PP Any segment contains pp,p vrc.pp, vrc.p, p, u
RR Any segment contains rr,r vrc.rr, vrc.r r
SS Any segment contains ss,s vrc.ss, vrc.s s ss
TH Any segment contains th,h vrc.th, vrc.h, h, t
Silence Any segment contains silence,sil vrc.sil, silence, vrc.silence

When reading the table, keep in mind the following:

  • Everything in the table below is case insensitive.
  • We split all blend shape names by any non-letter character into segments for searching.
    • For example “vrc.ah” gets split into two segments “vrc” and “ah” for searching.
    • For example “cheese is a good fruit”, would get split into segments that would make that blend shape get allocated to “AA” below.
    • For example “blendshape_monkey_a”, would get split into 3 segments with the last one “a”, matching the AA Viseme above.
  • vrc prefixes are NOT required, they’re just really common. See the note on prioritization too!
  • You do not need to follow any naming, you can always do manual allocation instead.

Allocation Prioritization

Due to the prevalence of the prefix vrc. on a large amount of commonly found avatars, when a blend shape that contains the prefix vrc. is found, it takes preference over all over blend shapes for that one Viseme.

For example, if you have an avatar with a vrc.aa blend shape and an a blend shape, then the vrc.aa one will be allocated as it takes preference.

Re-running the Automatic Allocation

You can re-run the automatic allocation process at any time, by finding an Avatar’s DirectVisemeDriver and pressing the “Auto-assign targets” button.

Manual Allocation

TODO

See Also