Visemes: Difference between revisions

From Resonite Wiki
No edit summary
No edit summary
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
Making your avatar’s mouth move and respond to how you are talking is a common avatar feature. In Resonite we use a fairly standard list of Visemes loosely based on similar platforms and products in the space, The Visemes we use can be found below:
Making your avatar’s mouth move and respond to how you are talking is a common avatar feature. One method of doing this is to use Visemes. A Viseme is a blend shape that's designed to resemble the way a person's mouth moves when saying a specific or set of specific Phonemes.


Resonite take's your incoming voice information and [[Component:VisemeAnalyzer|analyzes it]], based on that analysis it then controls your Viseme blend shapes on your avatar to make your avatar's mouth try to match your actual mouth.
In Resonite we use a fairly standard list of Visemes loosely based on similar platforms and products in the space, The Visemes we use can be found below:
<!-- We need photos here, but we need licensed photos. Do not just stick google results here -->
{| class="wikitable"
{| class="wikitable"
! Viseme Name
! Viseme Name
Line 69: Line 74:
== Viseme Allocation ==
== Viseme Allocation ==


Resonite supports two options when it comes to setting up Visemes for your avatar. ### Automatic Allocation Automatic allocation, will happen when you create the avatar using the [[Avatar Creator]], ensure you check “Setup Face Tracking”.
Resonite supports two options when it comes to setting up Visemes for your avatar:
* Automatic - Requires specific blend shape names.
* Manual - Can use any blendshape names
 
=== Automatic Allocation ===
Automatic allocation, will happen when you create the avatar using the [[Avatar Creator]], ensure you check “Setup Face Tracking”.


Then when you create the avatar, Resonite will try to guess the correct blend shape for each Viseme. Our algorithm, aims to find most common naming schemes including from other platforms and we improve it overtime adding new items into the algorithm to aid users.
Then when you create the avatar, Resonite will try to guess the correct blend shape for each Viseme. Our algorithm, aims to find most common naming schemes including from other platforms and we improve it overtime adding new items into the algorithm to aid users.
Line 147: Line 157:
** For example “cheese is a good fruit”, would get split into segments that would make that blend shape get allocated to “AA” below.
** For example “cheese is a good fruit”, would get split into segments that would make that blend shape get allocated to “AA” below.
** For example “blendshape_monkey_a”, would get split into 3 segments with the last one “a”, matching the AA Viseme above.
** For example “blendshape_monkey_a”, would get split into 3 segments with the last one “a”, matching the AA Viseme above.
* <code>vrc</code> prefixes are '''NOT''' required, they’re just really common. See the note on prioritization too!
* <code>vrc</code> prefixes are '''NOT''' required, they’re just really common. Do read [[#Allocation Prioritization]] though.
* '''You do not need to follow any naming, you can always do manual allocation instead.'''
* '''You do not need to follow any naming, you can always do manual allocation instead.'''



Latest revision as of 06:20, 17 June 2024

Making your avatar’s mouth move and respond to how you are talking is a common avatar feature. One method of doing this is to use Visemes. A Viseme is a blend shape that's designed to resemble the way a person's mouth moves when saying a specific or set of specific Phonemes.

Resonite take's your incoming voice information and analyzes it, based on that analysis it then controls your Viseme blend shapes on your avatar to make your avatar's mouth try to match your actual mouth.

In Resonite we use a fairly standard list of Visemes loosely based on similar platforms and products in the space, The Visemes we use can be found below:

Viseme Name Phonemes Example Phrases / Words
AA A Car
CH tS. dZ. S chair, join, she
DD t, d tip, doll
E e bed
FF f. v fat, vat
IH ih tip
KK k, g call, gas
NN n, l lot, not
OH oh, o toe
OU ou, u book
PP p, b, m put, bat, mat
RR r red
SS s, z sir, zeal
TH th think, that
Silence neutral Silence

Viseme Allocation

Resonite supports two options when it comes to setting up Visemes for your avatar:

  • Automatic - Requires specific blend shape names.
  • Manual - Can use any blendshape names

Automatic Allocation

Automatic allocation, will happen when you create the avatar using the Avatar Creator, ensure you check “Setup Face Tracking”.

Then when you create the avatar, Resonite will try to guess the correct blend shape for each Viseme. Our algorithm, aims to find most common naming schemes including from other platforms and we improve it overtime adding new items into the algorithm to aid users.

To help explain how the automatic allocation works, here’s a table that details what we look for:

Viseme Name Accepted Names Example Names we recognize
AA Any segment contains aa, a,ah or vrc.a, vrc.ah, vrc.a, a, ah, ah, あ
CH Any segment contains ch or sh vrc.ch, vrc.sh, ch, sh
DD Any segment contains d or td or dd vrc.d, vrc.td, d, td
E Any segment contains e, ee, vrc.e, vrc.ee, e, e, え
FF Any segment contains f,ff,fv vrc.f, vrc.ff, vrc.fv
IH Any segment contains ih, vrc.ih, ih, い
KK Any segment contains kk,k vrc.kk, kk, vrc.k, k
NN Any segment contains nn,n, vrc.nn, vrc.n, n, ん
OH Any segment contains o,oh,oo,uh, vrc.o, vrc.oh, vrc.uh, uh, oo, uh
OU Any segment contains ou,u, vrc.ou, vrc.u u, ou
PP Any segment contains pp,p vrc.pp, vrc.p, p, u
RR Any segment contains rr,r vrc.rr, vrc.r r
SS Any segment contains ss,s vrc.ss, vrc.s s ss
TH Any segment contains th,h vrc.th, vrc.h, h, t
Silence Any segment contains silence,sil vrc.sil, silence, vrc.silence

When reading the table, keep in mind the following:

  • Everything in the table below is case insensitive.
  • We split all blend shape names by any non-letter character into segments for searching.
    • For example “vrc.ah” gets split into two segments “vrc” and “ah” for searching.
    • For example “cheese is a good fruit”, would get split into segments that would make that blend shape get allocated to “AA” below.
    • For example “blendshape_monkey_a”, would get split into 3 segments with the last one “a”, matching the AA Viseme above.
  • vrc prefixes are NOT required, they’re just really common. Do read #Allocation Prioritization though.
  • You do not need to follow any naming, you can always do manual allocation instead.

Allocation Prioritization

Due to the prevalence of the prefix vrc. on a large amount of commonly found avatars, when a blend shape that contains the prefix vrc. is found, it takes preference over all over blend shapes for that one Viseme.

For example, if you have an avatar with a vrc.aa blend shape and an a blend shape, then the vrc.aa one will be allocated as it takes preference.

Re-running the Automatic Allocation

You can re-run the automatic allocation process at any time, by finding an Avatar’s DirectVisemeDriver and pressing the “Auto-assign targets” button.

Manual Allocation

TODO

See Also