This is a transcript of The Resonance from 2025 January 5.
0:00: streaming I'm going to post the announcement
0:05: posting there we go f hours uh post live stream
0:14: announcement where's live
0:19: stream there we go live stream there we
0:26: go let me make sure I have it open too so I can make sure it's
0:31: working okay this open hello hello
0:40: hello can you hear us
0:45: hello yes I'm still a bit sick hello hello sunp it's a
0:56: glch a c chat
1:02: hello everyone people are should be coming in the Stream is live hello
1:07: welcome to the everybody's sick stream oh God sorry everybody's
1:14: sick hello is the audio for both of us okay test one two can you hear me hello
1:23: is going like about the same levels in OBS I assume it's uh it should be okay okay I can hear you so hello and
1:30: everyone hello and welcome to the first episode of the Resonance of
1:36: 2025 of this new year um the resonance it's a weekly sort of office hours uh SL
1:44: podcast where you can come you can you know ask any questions about resonate whether it's the development engineering
1:49: whether is you know how is the platforms going where goals are you know what the histories like whatever you want to know
1:56: about the platform um and so all the questions that can end up like going into little bit of a tangent or like
2:02: longer Rambles where um we kind of dwell a little bit more into you know the
2:08: philosophy of it like where is it going what do we want to do with uh you know with this project uh where it's heading
2:14: to give you better and sort of like you know higher level overview of like you
2:19: know what the platform is and what our goals are um if you'd like to ask any
2:24: questions um make sure you put question mark at the end of it or you know
2:30: somewhere in the middle of it um then that way it pops up like on our pin messages so that Mak sure like we don't
2:36: miss it uh also we are coming to you from this like beautiful world this is the new um Creator gam 2025 New Year's
2:45: world is very pretty so like U like it's going to be like very fitting to stream from here um it's like going to be like
2:52: kind a good backdrop I'm not kind of get into the habit of trying to like use different worlds for for these streams
2:58: um just kind of go around hello F hello fuzzy
3:06: hello ter scale I'm sorry if I'm mispronouncing that um so hello everyone
3:13: um so with that like I think we could get started uh the a is like you know
3:18: usually go usually just go like for two hours because we got plenty of questions by the time it kind of gets going um so
3:26: hello blue Bo blue
3:32: blue blue I'm like it sounds like blue W but
3:37: uh it's blue but it's B which is the sound like
3:42: a it's a that's that's a very different dialect it's a different dialect of it
3:48: anyway we got a question uh actually clear this one uhb 8272 is asking will
3:55: sharing videos in game on the spot be improved maybe an option to load it if friend wants to watch it alongside for
4:01: now I have preload MP4 by saving it in the world itself um so I'm actually not fully sure
4:10: what do you mean by that like like when you bring a video into the world um it
4:16: it automatically loads for everyone in the world at least like it should um I'm actually not quite sure I
4:23: understand this one um perhaps say so perhaps maybe you've
4:30: just had bad luck there is currently a bug that happens where um if there's a
4:36: lot of uh asset transfers in a world there's a chance that asset transfers may end up getting stuck that is
4:41: transfers between you know teers players um it could be that but it's like know
4:50: cumal so I'm not sure maybe you've just been having bad luck with that but if you bring an mp4 into the world it
4:55: should upload to the host and then the host distribute to everyone else in the session yeah I don't want to like Rite that
5:02: system because like it is PR to kind of like just get things clogged and then Stu like stuff that's like locally imported doesn't like transfer anymore
5:09: um so it's um you know it's yeah like but like it's not like something that's
5:15: specific to videos like it's anything it's you know it's pictures it's you know like what whatever assets you bring into the session they are not like safe
5:22: to Cloud yet they will be synchronized you know sort of um between like the clients through the session and when it
5:28: clogs you know just stop transferring I I want to like Rite that one at some point um um
5:35: yeah also the sorry the question is like you know like also like where are you importing
5:42: the video from are you like importing you know is it from YouTube is it like low class it and so
5:48: on yeah it might be it might be just that like that system kind of needs right it's kind of uh
5:55: they they did mention that um the like loading indicator on the video player only showed one out of two um that
6:05: thing well the question is like is the is the video actually loaded because there's also another B indicator where
6:10: might not show it correctly so like yeah does the video play for the other person
6:16: but also like maybe knowes are just clock so it could be either the two bugs yeah it was a local
6:22: file um yeah like this isem it's one of those
6:27: things I really want to rep at some point um so that's going of because that's like it makes things kind of just weird
6:33: because like one on you start like you bring stuff um on you kind of like you
6:41: know bring stuff like uh you could be bring like pictures and memes and videos
6:46: and like other stuff we want to show people andly kind of stop showing it kind of gets weird because if usually
6:51: restart a session so um I really want to touch that one at
6:57: some point always also asking perhaps an option for uploading access directly to the cloud inventory so it can load
7:03: faster for everyone I you can already do that you just save it to your inventory and then spawn it from there um so there
7:10: already an option you know and usually what people do is like if if this sinking is like you know stuck usually
7:16: that we just tell people like just save it to inventory wait for it to syn and spawn it out that way it loads from the
7:21: cloud because once you save it to your inventory um the URLs are Rewritten to
7:27: the cloud instead of like local um so it kind of like you know gets around it but um the main thing is you
7:34: know just fix it so like the syncing doesn't break uh we usually don't do like we
7:40: usually don't add features to work around a buag we just want to fix the bug because like it's otherwise you just
7:48: end up like you know with lots of kind of bugs that still stay there and you're just kind of working around them and it
7:54: kind of makes the Bucks faster which is like you know not good long term
8:01: um the next question is from Jack the foxter uh what's the scope of the upcoming audio system rework both things
8:08: will uh get worked on added replaced so right now the audio system in resonite
8:14: is already like a hybrid um we like f engine it handles uh a lot of lot of the bits of
8:22: audio processing like for example we do our own audio decoding you know like when it's decoding wave files or Fleck
8:29: files you know or OG like vbus um all of that is you know our own same like you
8:35: know with stuff like the voice like this being like transmitted that's like fully Forks engine um the parts that LE unit is
8:42: handling right now is the spatialization so we actually just give Unity we give
8:47: Unity the audio buffers actually this might be like a good I might like draw
8:52: this a little bit just to kind of give you a better idea um I'm going to go over here I made a spot for drawing
9:00: there we go it's not it's not a whiteboard but um It's Kind nice you
9:06: know oh um it's nice like whiteboard uh in the
9:15: sky so which is kind of bit weird because I don't have like any reference to draw but we'll see how it works
9:22: um so right now like you know say you have a bunch of uh audio sounds you know
9:28: you have like audio s here we have another one here you know and we have another one
9:36: here and these are like you know all playing like different sounds um you know say like this one's
9:43: like a wave this one's like uh I might actually
9:49: a little bit the camera is a little bit rotated I might move it a little bit
9:55: better I guess like having a whiteboard would be better but uh this should work so a bunch of different
10:01: audio effects you know say this one's OGG you know it's like this one's like flag it's just a bunch of sounds in the
10:08: world you know and you have like a listener you have like an ear I don't know if I can throw an ear but you have
10:13: like your ear here so what we do is we have a system that actually decod the audio for this
10:21: you know uh so this is like the audio data and like this is the buffer and every
10:27: like audio update it like you know decodes the bit of that audio and it's already done on the FR engine so this is
10:33: you know on our side of FR
10:40: engine um is it's still kind of in frame it's kind of weird I probably should
10:46: have like found like a flat place or something um let me see if I can find the
10:52: Whiteboard yeah it's one of the downs I'm going to move the camera a bit
11:00: uh I can actually move a little bit out the
11:06: way there we go uh so this is like you know FRS
11:15: engine uh and then like you know we have the unity side and right now what
11:21: happens is um FR engine will kind of you know synchronize the positions of them
11:29: um and this is unit day there we go we got a whiteboard
11:34: there we go it help I mean it doesn't fully there we go is it FL camera there
11:42: we go I think that's
11:48: fine I'm going tojust the camera a bit uh let me just move it
11:56: oh oh wait folks we're a little out of it it today why is the camera attached to the
12:04: Whiteboard what what oh it's the uh is it the
12:09: frustrum snapping to the Whiteboard oh May that's weird uh
12:16: no uh BR okay this this work so let's just thr
12:23: with this so then we have like you know the unit this side um and units this
12:28: side we have have to like you know duplicate U can actually move way um you know on the UN side we actually
12:36: say you know there's s here there's AIO Source here there's audio Source here um
12:42: and then like we have to transmit like we have to give United the audio data for each one of these so we give it
12:48: buffer for this one we give it buffer you know for this one um and then we
12:53: give it buffer for this one and then unity and we also tell it like you know the ear is over here here so Unity can
13:01: figure out you know how this should sound based on the position of the ear
13:06: with the spatialization um and this is like one of the things
13:12: like which makes it like you know a little bit more entangle with the unity because we're giving it audio buffers for each individual audio output um so
13:20: one of the things we'll be doing is that like all the specialization because like Unity resolves this and then it produces
13:27: you know final mix of the audio then then goes you know to your actual
13:34: speaker you know and then like you then hear that um so this you know then goes
13:39: like to your you know whatever like your headphones your speakers whatever um
13:45: what we'll do by having our own audio system because um say like you know we
13:51: have the ear here and these sounds are like you know near you you they will actually kind of collect you know these are like the
13:57: close sound effects going to render these out I'm going to specialize them and maybe there's also like you know
14:03: there's like a Reverb Zone here that we you know told it about
14:09: um you know there's like one on here and we just kind of sync that um there's the
14:15: here so there going to apply you know extra effects um we want to like resolve all of this here so unit is not even
14:22: going to need to care about this so what our system will do instead of having
14:27: Unity you know or essentially F mode like which is the system it uses to this
14:33: part um will have a system uh which is very
14:40: likely going to be based on the be physics uh where each of the know s effects it has like you know it has like
14:48: a range where you can hear it and then like when we say okay there's a listener
14:53: here we want to collect all the sound effects or like you know all the sounds that should be
14:59: hearable from this point in space um and want like you know hear them um we want
15:08: to kind of hear them like you know we want to like resolve them so it's going to collect all of them like in so it's going to be okay this one's near and
15:13: this one's near we already have like system 4 decoding in audio so then it's just matter of like you know taking
15:19: these and running it through spatialization which is you know like um they're going to be still using steam
15:25: audio except like we're going to be using directly now so we're going to ask theme audio you know to make the sound
15:32: as if you know they are at this position relative to The Listener this ones at this position relative to listen
15:39: um what's cool about this is like you know like once we also like resolving audio footage over the sound effects we
15:45: can now control stuff like attenuation so for example we can say this is you know this far from the user so maybe you
15:52: know it's going to use you know this this kind of curve you know or you know
15:58: this kind of curve or maybe like we have we're going to have like a different you know type of curve or whatever like you
16:05: know we want because we can just plug whatever function we want to compute um
16:11: to compute you know the um the actual iation um this going to you know do like
16:18: stuff like feature parity and when there's also Reverb Z they going to be like okay like the user the listeners
16:24: within the Reverb Zone like we found they like inside so we're also going to apply you know extra sound effects to
16:29: this audio Data before like mixing it together and once this is done we actually compute our own final audio
16:37: that's all mixed and resolved for this listener so this is like little final audio that's been like resolved from all
16:43: the sound effects whatever other effects are in there and we we're going to produce it ourselves instead of having
16:49: you to produce it and it becomes a question because like either uh we can just send this send this to Unity and be
16:57: like you know can you pipe it you know to the speakers or we just um we use the
17:03: library we might just like you know pipe it the speakers ourselves so like just we just pipe it in and you is not even
17:10: going to know there's any audio being produced because we just do it with our own process um what do uh by having you know
17:19: the control over the system uh what does what allow there's going to be you know a few new features that are going to
17:25: come in as part of the system uh the main goal because like you know this is done as part with
17:30: performance updates is feature priority which is making sure you know all the stuff that works right now reasonably
17:36: works with a new system um but there's like a few things that
17:42: are like relatively simple additions that we're going to add along the way especially because there are also things
17:47: that people have been asking about one of those is having multiple listeners so
17:53: right now like you know I'm streaming to with a camera uh if I switch to audio to specialized actually I might like you
17:58: know even do live demonstration for this um so I'm going to switch CRA back to
18:03: you know specialized so SRA if you say
18:09: something you hear him coming from the wrong side because it's actually coming from my side I would have to like you
18:15: know turn this way now you say something hello so now here little from the right side because like you're
18:21: actually hearing it from my viewpoint we do have a feature where I can switch it on the camera um there's uh render audio
18:30: from camera Viewpoint so now s if you say something
18:36: hello oh that's so weird so now you see it correctly from the camera but I hear him from over there which is incorrect
18:43: like and it kind of it messes with my brain okay I switch it back I'm going to switch them back to broadcast so that's
18:50: one of the limitations you can either have the audio coming from the camera's Viewpoint or from my viewpoint but not
18:57: like you know you cannot have two separate once because with unity we can only have like one listener like one
19:04: virtual ear in the scene um with uh having our own system we can
19:12: essentially have as many ears as we want so we can you know have another one that represents the camera and just be like
19:19: we want to also resolve all the audio you know from the Viewpoint of this ear and it's going to be the camera and this
19:24: is going to be piped into a different audio device so this is you know this one goes to your speakers so this is
19:31: what you hear and this goes to the camer and maybe there a different audio device you pipe it to OBS and now we have
19:38: accurate audio you know from from recom Viewpoint we can do like a bunch of
19:43: extra features as well like we can for example render your own voice So like um
19:48: you could be hearing me spatialized relative to your Viewpoint even so like you know I'm the one
19:53: streaming uh which is also going to make it a little bit easier you know for the setup because with obs won't have to
19:59: like you know I won't have to do my voice separate capture my voice separately from the game audio my voice
20:06: is still is actually going to come from the game audio with this system um
20:11: there's some other things we could do too uh because for example when we are resolving the
20:17: audio um one of the things that's added um uh that I'm kind of like adding like
20:25: in background is spherical harmonics and spherical mons like I mentioned before one of the things they're used for is
20:31: ambisonics which is a way to encode directional audio that means in a scene
20:39: um I'm going to like remove some of this clutter just so it's like not too
20:46: cluttered it's a way of having audio um
20:51: that sounds different based on the direction you're hearing it from so say
20:57: there's a i AIO Source here and this is ambisonic one I'm just going to like do
21:02: like I'm going to visualize it like this so you might actually this might
21:09: sound different if you're like hearing it from this angle and might sound different if you're hearing it from this
21:15: angle or this angle or it can also be you know the other way so like it can be kind of
21:21: coming from everywhere and you kind of inside of it and as you turn your head you know depending on which angle like
21:28: head is the audio is going to move around you it's a really good way to like do like you know environmental
21:35: audio so you can um you can you know have like sounds of like environment and
21:40: as you move around it doesn't stay fixed to your head but it actually kind of moves around it's good good way to do
21:47: Ambiance uh and because we have a de code for you know resolving spherical harmonics and because we'll have like
21:54: full control over resolving the audio we can just you know be like we just pipe the direction into the ambisonics
22:01: compute what the audio should be and compute you know the audio buffer and then you know we do whatever processing
22:06: and piping we need on that um so just having like you know the full control the system that's going to open up you
22:13: know L of like possibilities the one thing I don't want to mention is the main focus you know is just kind of
22:19: collect audio like collecting um and then it's like uh you
22:25: know resolving the spatialization um people people kind of expect you know
22:30: some things like stuff like with audio decoding and so on but it's stuff that we already to do like that's why it's
22:36: like currently why is it a hybrid because we already handle our own
22:41: decoding uh of like different like audio files so like this really going to get touched um there's also uh things like
22:50: um you know doing audio processing like DSP that's a separate project that's going to be the protox audio DSP so you
22:58: can do like you know effects like um you know splitting channels you know pimping them into different things and so on
23:04: that's going to handle stuff like that uh the audio system specifically is just you know um resolving the spatialization
23:12: calling the audio sources that are like you know around you building like you know uh the system will to build like a
23:17: priority list too so for example if you have you know um if you have uh lots of
23:25: audio sounds like in the same area so you know we
23:31: have like your listener and there's like you know say each one of these is you know an audio
23:36: Source you know there's so many
23:42: around the system cannot like you know reasonably play them all without like you know losing
23:48: performance um so this you like build a list of them you know builds like a list
23:54: it's like you know these are all the these are all of the these sound sound
24:00: sources within the proximity of the ear and it's sorted list it's like you know okay like this one has higher priority
24:07: and you know kind of ranks them and decides I'm going to play only
24:15: these ones and these ones are going to be quiet but we also have like in a control over that logic now so we can
24:21: you know add whatever rules we want we can add whatever behaviors we want it's going to open up a lot of you know
24:27: options for expansion uh for the kind of system so I don't hope this kind of covers things um you
24:34: know what the audio reward will kind of like involve what can can what can you expect from it uh if you get know any
24:40: followup questions like you know feel free to ask um we're going to move back to the other
24:49: area there we go oh there another of the rambl oh let me
24:56: POS this um there Al like you know bunch of things that we'll do like in the future
25:03: just like by implementing our own versions of some of these systems you know we get full control of our system
25:09: because right now some of it is more of a black box inside of unity and by
25:14: having our own code that handles it you know it's no longer a black box we have we control it and that opens up like new
25:20: possibilities too and makes it easier to fix certain issues or out certain features that otherwise would be a lot
25:26: more difficult so Pro kind of covers that
25:32: question uh next question from SNB 8272 another question uh will we be
25:39: getting any linear curves for color Alpha over Lifetime on leg a c particles
25:44: back uh so I'm not what do you mean bu like a
25:51: cereon parf course we should already have the like
25:57: the the arrays for the for the curve Keys should already be there so if you have any previous particles that ended
26:02: up using those they should be fine there was previously there was there was also
26:09: previously no way to edit those and if you saw a way to edit those then that was likely you joined someone's session
26:14: and they had a mod that yeah added added like a like hack hack UI to like edit
26:20: them so it should convert like the like all of the modules like they should be there for Feature Part if something's
26:27: not converting right like you know make sure to make a GitHub issue so we can kind of look into it but um yeah like
26:34: like all the modules for feature par should be there so we probably need a little more kind of
26:41: clarification um there's also like new modules you can for example do color an alpha over lifetime using a texture as
26:47: well which can make it like easier to edit but um if you're like using like a mod or something to edit the
26:55: Aras um they will require you know whoever made the mod or whoever made the thing to like update it for the new
27:01: system but the actual like you know system like is in
27:06: there uh next question is the chip Forge any plans to support Opus OG file
27:12: decoding I think last time I tried to upload OG file that had a encoded in Opus it would just have a blank way for
27:18: Man play which is is funny because all the voices from user seem to be Opus um so it's something we could do
27:24: it's just like there's not been too much you know request for it I I would recommend like
27:30: making a GitHub issue because that can be a good way to kind of gauge like you know people interested in that it's just
27:36: m of you know putting time into it um it's doable it's just you know needs men
27:44: hours to happen um for the voices of users the isn't like we can of just use
27:50: the part like you know this is used for voices is because uh while the voices
27:55: they use the Opus codec they don't use the OG container the data is like you
28:00: know is essentially send pretty much Raw it's like raw Opus frames that are being
28:06: transmitted through our like custom like system for like hand like audio buffering which is one of the ways like
28:12: you know we achieve like a really long like AIO Laten of it voices um but it
28:17: means you know like we can just use that same code for you know Opus OG playback
28:23: because what you need to do is you like either like we would have to
28:28: um use codee to like you know decode the Opus frames from OGG and then you know
28:34: use the Opus decoder to decode the actual audio data or we just you know use the library that kind of does it
28:40: like both at the same time we don't have to like worry we just kind of give it and gives us audio data bag so but
28:46: essentially you you need like two pieces you need uh something to extract the OGG
28:52: you know frames from the OG stream and within those you have uh you know the
28:59: Opus frames which is the actual kind of raw encoded audio data because the the
29:04: the codec it also handles like you know some other stuff like for example seeking you know like each of the O OG
29:10: frames it has like you know stuff like time stamp and so on and that's one of the things you need to kind of keep track if you want to you know move in
29:17: the audio file like you know like you know jump a minute ahead or want to jump a minute back um stuff like that you
29:23: cannot really do that with voices because the voices is just a constant stream and if if you miss a packet like
29:29: it's gone like and you cannot really go back uh so it's kind of different uh but
29:35: yeah like if if you want to like import files um I would recommend making a GitHub if you getting an approach on it
29:41: and that way like oh your keyboard is sounding
29:47: um um they can you know what I the music in the world I
29:54: haven't used it um but but I do recommend like making
29:59: it isue um there's also another possibility I do eventually want us to switch to a different kind of library
30:06: that handles a l of the decoding because we're using uh we're using a lot of kind
30:11: of C like implementations of these things and there's like some li like libraries been kind of looking at as a
30:16: potential replacement some of those libraries they do support like Opus like
30:22: with an OG container so by making that switch in the future like you know it might just happen implicitly
30:28: um but we'll we we'll see how that one kind of goes um so this one uh this one's not
30:38: related uh nun isn't there an open standard for VR avatars and worlds why
30:45: hasn't it been implemented in resite and will it ever be um I don't think there's
30:51: like a standard for like VR like I mean there's like some for stuff like you have like for example
30:56: vrm uh um which is sort of like you know just G GT files with some extra stuff
31:03: one of the things that's been kind of like blocking us there is because it includes like licensing information and right now we don't have like a good way
31:09: to kind of preserve that like enforce that so like we've been kind of like waiting on that um especially since like
31:15: information we gotten from people where like you know it's better to have like we don't robus licensing system before
31:21: supporting that but you know we might want to reevaluate that um I don't think
31:26: there's like any standard for worlds like at is I'm not like really aware of any um the problem is also like you know
31:33: it's a little bit hard to like have thunder that kind of covers everything like you end up like with something that's like um you end up with something
31:41: that's essentially like uh like the least common denominator
31:47: like because there's a lot of stuff you can do in res that you can do in other platforms which means it cannot be part
31:53: of an standard you know and vice versa with other platforms like every platform has its own different ways of doing
32:00: stuff uh and if you want to like you know there still the things that can be shown
32:06: between platforms like it needs to be some common share functionality which ends up being just a small subset of
32:12: what the different platforms support um but we do support like you
32:17: know number of like formats like you can import like GF like directly that's like an open you know form like for mod files
32:25: so um it was looking to add like more for mons as the time goes just you
32:31: know like with everything is just a matter of like um men hours that need to be put into it and also kind of like you
32:37: know kind of interest from people um or like I should say like you know
32:44: the interest like you know from people is is what helps to determine which
32:49: things we invest mon hours in to to implment so if you're like aware like any standards like you would like to see
32:55: you know make sure there's like a geub issue and make sure just like know pH on
33:01: it uh V how much is the FRS how do I give FR a coffee a
33:09: coffee I don't know how are you giving me a
33:14: coffee I mean you like go to Starbucks I guess I
33:22: don't uh Zan 5250 is asking do we have or we have any
33:29: interface protox to manipulate audio buffer for creating custom sound effect audio filter audio ciz algorithms like
33:36: the Callback uh mono behavior on AIO filter read and unity engine so there's
33:42: a Planned feature uh that's called the audio DSP for Proto flag uh there's a
33:48: GitHub it that kind of like has a bunch of kind of details and essentially what it will do is like you know what you're asking is so you can do a bunch of like
33:55: AIO paring that's going to includeing know a bunch of existing filters so like you know little common filters you want
34:01: to do with audio that are going to be like you know just nodes and you just plug it through them there also going to be mechanisms where you can specify and
34:08: say um my like you know where you can specify
34:14: uh uh where you get like an audio buffer and you can just kind of loop through it and do whatever you want with it uh to
34:20: make your own filters as well uh so it's going to come at some point um if you
34:25: like you know if you like to see that like if that going to be sure like an it's going to make it easier for the
34:31: justify prior prioritizing it sooner it's one of the things I'm kind of looking forward to working on because I
34:36: think it'll open like you know lots of new cool options um but yeah it's it's it's on
34:43: the it's under the road map uh it's going to happen
34:49: eventually uh next question from chck the fox author I've been busy the past
34:54: week getting into plugin development and I'm curious what are your plans for improving expanding plug-in system the
35:00: ability to make user space core plugins is powerful and fun to work with already but some things still seem to be missing
35:06: like call bikes um events that get called after the engine start stops I
35:11: may get a isue about it um it's look kind of depends because like I'm I
35:16: always like um there's like several things because like plugins to me
35:22: they're like how do I put it like uh
35:29: you can pretty much like use them potentially to add anything into the engine the problem is whatever you add it makes the it makes you incompatible
35:37: with other clients and one thing could be done is you know like you have like other users install a plugin but they
35:43: requ to go website and and so another thing and so on um you could
35:49: theoretically have the plugin automatically download and then run but that's a very very bad idea because it
35:55: can be malicious code and you know you don't want want to just run that automatically it can be malicious code if you download it you know from some
36:02: websites too and install it but at least like you know that requires user action to do uh so it's like less likely to be
36:10: exploited so in my idea like I think plugins are most like suited for things that don't touch the data model um uh I
36:18: think you're going to be like you know great like I do want to like expand uh support for um um you know making it easier to add
36:27: like stuff like device drivers you want to add you know support for another device can make it easier to do that you
36:33: want to add stuff like new importers and exporters for file formats you know plugins can be like really great for
36:39: that too um it's sort of like providing like you know specific ways to add
36:44: different types of things to the engine that can expand it well um and sort of
36:50: like you know provide like different kind of hooks and call bags that make it easy to plug into these systems um that way like you know
36:58: uh we could also like you know take a lot of our own code um like one of the things I've kind of wanted to do for a while is for
37:04: example you know take the Importer exporter system and rewrite it to be more modular but also open source doors
37:11: modules so we can publish the code that does the importing of models you know uh
37:16: and does the importing like you know exporting of other stuff because that way you can both use it as a reference
37:22: to make your own or make your forks that um you know better suit your specific needs needs uh or you can even make like
37:29: you know contributions to the official on so like if there's like you know bugs and if there's you know some issues with
37:35: something import and like you have fixes for it you could like you know help us with those and help like you know
37:41: improve things um that way uh plus you know can just build your
37:46: own importers you want to you know support this like super obscure file format you know um WR like WR like an
37:53: import plug-in for it um the other is like you know having sort of like patch
37:59: processes so like if you have like you know tools that like um I don't know say like they go through the whole hierarchy
38:05: of the world and they do bunch of like optimizations on it that's another I think like really good use for plugins
38:10: because like that doesn't need to introduce new data into the data model
38:15: um so it can be you know another thing like we provide sort of nice interfaces for um for like you know that like add
38:23: stuff to the data model I feel it's better to like um you use stuff like uh
38:28: you know web assembly because that's much much easier for like soundbox is
38:34: designed to be secure um so we can like you know we can like synchronize that code to other
38:40: users without worrying you know it's going to cause exploits or like you know the likelihood of an exploit is like way
38:48: like way lower like it's never ever going to be zero you know but um the
38:53: assembly is like designed you know for the internet is designed to like you know run in your web browser
38:58: um so it's already kind of like you know hardened against that versus like you know 6 plugins like they're
39:06: not um but yeah like there's probably going to be like you know things for um
39:12: supporting specific you know kind of like use cases for plugins uh one of the things is also going to be so you can
39:19: more easily like you know like when you when you do have stuff thatat model if
39:24: you really want to do that is like when you start a session you decide which ones of them are going to be activated
39:30: for that particular session uh because right now like you know you can't really select that um so
39:37: when you start a session and you want to use like you know some like components you made or something you got to select it and then anybody else who has those
39:45: they'll be they'll still be able to join but also like you know you're not forced to use those automatically so you can
39:52: have those plugins installed but you can still join sessions hosted by people who don't have those plugins
39:57: they're just not going to be know usable within that session you won't be able to bring those in so there's another kind
40:03: of improvement that's going to be done to the system as well um oh that kind anwers that one uh
40:12: let's see I actually actually wanted to comment real quick actually okay um
40:18: there is actually already a uh hook you can use to uh run like a function or
40:25: whatever um after the engine is initialized I think you can it's called like engine. current. onpos it or post
40:33: it or something I can't remember the exact name but if you give that a delegate that will run after the engine
40:39: has initialized um so you do already have that one I know for a fact so I'm sorry I forgot to plug in my
40:47: headset the power uh come
40:53: on yeah there's kind of like a few events like it's not super nice but like oh je there we go I can't plug it
41:02: in uh but you can like you know once you can of hook into like a method you can like un access the engine you can then
41:08: like you know hook hook into like whatever other things you need okay so should be good
41:17: um so next uh uh next question is from shining Hero
41:23: 1 uh what sort of audio system are you looking at play Unity wants something in house or a pre-made one like steam audio
41:29: so steam audio that's not uh it's not an audio system it's like a audio spatialization so it does part of it um
41:37: I think this question was probably before like I did like the long explainer something like I'm going to go into details but um steam audio is
41:47: uh um this part that we're going to be using for specialization and stuff like that but there you know other parts are
41:52: going to be custom we're going to be using you know other like for example for the calling of the audios um we're
41:59: going to use you know beu uh because it kind of has like you know stuff like for accelerating special queries and stuff
42:06: like that um uh but it's going to be like you know custom all like the gluing and
42:13: everything uh next question check the fox Duncan I want to do dunin again yes
42:19: Duncan when when to your friends like I mean ja like we want to dunan because somehow they have Duncan in
42:26: Germany and it was right at the train station so my thing's freaking out where
42:31: is the battery freaking out there we go um SNB any's going with their ha is
42:39: blowing my of it I think it's the transform over right uhbe just disable
42:45: motion there we go okay I should make it better
42:50: um next question is from shining her one uh uh there an idea how does a plugin
42:57: store for how does a plugin store for resite sound kind of like how demoing Discord
43:04: has with at list of most but officially around um so oh do you mean like like having
43:12: sort of like a place where you can find plugins and so on like a workshop I guess yeah I don't know if it'll be a
43:19: workshop like the thing is like maybe the problem is like do with plugins it's
43:26: like they're inherently less safe so like like we're little bit like if you
43:31: make it like very easy to download them it potentially opens a vector for abusing them and I'm rather much like
43:38: you know um Implement you know like the Vib
43:44: assembly based ones uh which we can be like an nor kind of certain like this is not going to cause like security
43:50: concerns so I don't know if it be I don't know if it do it for plugins like it might be opening like a
43:57: kind of worms that we would might not want to uh nukon is asking will this custom
44:05: modules be open format like can we bring them to other programs um
44:11: exp oh no like Source modules no like I
44:17: mean generally no like because like the Importer like what essentially is a job
44:23: of the important and exporter uh for an importer you know it whatever format uh
44:28: you like you know whatever format you want and it converts it into resonite equivalent exporter takes you know the
44:36: resonate equivalence and translates it into whatever format you want which means it's you know it's specific to
44:42: resonite by it nature um so you you cannot really take like an importer
44:47: that's for reson height and just bring it to a different program you have to like kind of rewrite it for that program
44:53: you could make like you know an importer and exporter that like you know Imports and exports format that works in in a
45:00: different program that supports like you an open format but you have to make that a module but those modules themselves
45:08: they're still specific to resonite because you know they work with it data structures so there's not really a way
45:14: to like have those work in different
45:20: programs [Music] um so the next question is the red Nico
45:27: hey FRS what's some of other systems other than audio and particles that keep us from detaching version from
45:33: munity um so ma for like any systems that are like you know big system big
45:39: systems uh it's pretty much just the particles and audio uh there's a bunch of smaller bits like in particularly the
45:47: inte like how the integration with the unity Works uh so that's uh another system that needs to be kind of reworked
45:53: before like we actually move it into separate process um but that's pretty much it um like uh
46:02: in the past like FRS engine you know it was tied into unit in more ways like for example like you know it was using its
46:08: UI system which was like you know bad and then like um it got like uix which
46:13: is our own UI system which handles all the UI generation completely um uh you know completely
46:21: like on the um side of like you know FS engine and it just sends like you know
46:26: generated like meshes you know to unit that are in out uh so there's like more systems but like been kind like slowly
46:33: like moving them piece by piece also R was asking so not input system yeah we
46:38: we already kind of like have like our own input system um this kind of Lely handled so we just uh the main thing is
46:45: like we need to like pipe data into it um but like we already do you know we
46:50: have our own input system and we just like piping data into it so we just keep
46:56: doing that um but it's not really like a big
47:03: thing so it kind of cleared out like all the questions right now um let's see
47:08: what the time is um I think I'm like vibrating because I want
47:15: to see a Splat so bad yes so it's one thing like we're actually running on a
47:21: build that supports cian spot thing it's not super like tested yet because I'm kind of just working on it um sort of
47:28: like a fun thing but uh do you guys want to see like a gasi in Splat say
47:36: Yes um the only thing we'll have to go to a separate uh
47:42: uh oh never mind there's a question can do the [Music]
47:48: Splats like like you made s said we'll answer this question and I'll show you
47:54: um I'll show you I'll show you some Splats uh Cas blades off topic question
48:01: but did you see Contact track if so what AR I
48:07: haven't seen it I know what it is do you know what it is I'm not really sure what that word
48:13: means yeah I don't know what it means you have to clarify but question is if if you see if we saw it then it's no
48:21: don't know what it is also did you spot the Sho I did I don't have it in portrait um so let's actually move like
48:28: uh I sh this word is a little bit too heavy um is a little bit too heavy for
48:35: like you know this world especially because there's also like you know this's like mirror and everything so like it's like rendering this play like
48:41: multiple times and what spawning it kind of killed me so we're going to move to a grid space for a second um I'm just
48:48: going to go grid there we go there we go come
48:57: oh I still have these I need to like get rid of them s should be able to join me it's there we go it's public
49:04: so um let me also bring the chat here uh
49:09: the chat should still be running in the other place so it's going to collect questions but just so we can kind of see people's
49:15: reactions uh twitch we go we're going to be collecting questions in this world
49:22: um so I do have uh some
49:30: um let's see I do some SPL that I like know
49:36: um that I have uh already imported so I can just kind of Spawn them in let me
49:42: adjust the camera or do want actually let's let's I'm going to show you the import process um let me just make sure
49:48: I got stuff ready um let's see uh
49:57: I'll show how the importing works if I can find it um 3D scanning
50:05: where's uh give me a second so right now we support like
50:12: importing SPL in the py
50:18: format um it's kind of imported into like our own kind of like internal format which actually tested it is is a
50:25: little bit smaller because we have probably a bit of a compression on the data um okay so let me also switch on
50:33: private UI can see my Dash there we go
50:38: uh SM POV there we go so I have the Lego boyer. P yes I'm
50:47: importing that one I'm just going to copy this file I'm going to paste it here and I press it I missed it there we
50:56: go and the thing is you know uh this format uh py can also be 3D model but
51:01: this one's gasan split so I have to sell gasan split otherwise it w import properly and you can see it's converting
51:07: the Splats now there's about there's three million of them oh my God splatting SP thing so it converts them
51:14: and then it like needs to encode them I actually went a little bit overboard and I like made it report like you know what is encoding at the time
51:22: too right now we also don't um there we go
51:29: colors um right now we don't
51:34: um do like any compression data yet that's going to come like later so like they're kind of heavy on VM like this is
51:41: going to eat like it's going to like 700 Megs of vrm plus like the Sorting
51:48: which is another like it's going to be about a gig of vrm so there have finalizing in code and it should appear
51:54: in a second it might take a while to show for you actually which I didn't realize oh there it
52:00: is um and it's kind of hurting because of the camera
52:05: too uh let me switch to the person and I actually don't need do private anymore
52:12: uh there we go but it this is a gas in
52:18: Splat oh this is much heavier with the camera too I think my laptop's kind of crying
52:25: right now actually let me see if this is going to
52:32: be this is kind of like um I hav really much testing so
52:40: this it's kind of you know just showing you of the CF
52:46: um well this is surprisingly heavy like it's it's big so it's probably transferring through you and it's taking
52:52: a while uh ch finished already give it a
52:58: bit wait it already transferred was fast yeah yeah it finished really quick it
53:04: might be processing the stuff like give it um okay keep an eye on my
53:13: stats oh this is junky I wonder if I disable I'm going to disable the mirr to
53:19: display for a bit and close the camera oh it makes it nicer um I might
53:25: need to like add like a thing like where it caps how many sorts it does you know per
53:31: frame but yeah it's like it's here it
53:36: works still work to do but um so there's a question will be able to
53:43: edit PL within resite funny thing you ask so I actually did add like a basic
53:51: editing functionality it also might be there's like a bug like with transferring them over net work so
53:57: like oh there is a bag it won't show to you because
54:03: uh because uh when it's imported locally um it needs to sync the metadata
54:11: for it but I haven't added code for that which means it probably blew up and
54:16: you're stuck not seeing it I'll spawn this one um what I'm just going to use this
54:24: one uh I'm going to point it from the cloud which means you'll be forced to compute it locally so I'm just going to
54:30: gra in splits uh which one I'm going to do one so this one should load for you um
54:40: there we go very do you see this one that is that
54:45: is that is quite chunky damn yeah they are they are they are
54:53: heavier it would be like uh it would be faster like if we were using your
54:58: version of unity so we could like use more efficient soring algorithms
55:04: um I still want to like do stuff like where I can kind of limit you know how how many because like when you render
55:11: the plats you essentially have to sort all of these based on the you know based on
55:17: the distance and you see there's a lot of them and like these are being you know constantly sorted based on like
55:23: where I'm viewing them from um and because of the version of uni like the Sorting options are more limited to less
55:30: efficient algorithms so I do want to add like mechanisms where you'll be able to
55:37: limit how many sort events going to happen per frame so it like you know doesn't hurt as much
55:42: um holy crap M my GPU is at 100% oh yeah this all
55:50: this will tax your GPU like this is very very GPU I've got a 7900 XT just for
55:57: context so it's no SL and yeah um but anyways like there was a
56:03: question about think it's actually kind of like where also like you know it's going to help so I'm going to select this up I'm going to open it in the
56:10: inspector and you see there's like a bunch of new things there's like you know static gassian split there's gasan
56:15: SPL render um can actually also see
56:21: like can change the oppacity which is weird and I can also make it way oh this is weird
56:29: um or I can you know change the size of them which I should be careful because
56:34: the that hurts it can make you know all the supp like be way smaller you can
56:40: kind of see you know how is it constructed because like you know you see like everything's made out of these like fuzzy color
56:48: blobs and there's a lot of them in here you know this whole thing is made of the blobs and now they're just smaller than
56:54: they should be sting covered the whole area but it gives you a better idea of the structure of it uh so make them back
57:00: to normal size sure I should just under so it's perfectly one there we go
57:09: Um can also make it sort less often but like you know if I do that it still
57:18: works but if I if I move too fast it might take a little bit to adjust you
57:25: see like when I Circle
57:30: it like if I like kind of works it's actually not too bad it's not
57:38: too bad espe not in VR because it makes your because since it makes your game a little faster than it the uh you have
57:45: more let me make it s there we go oh there we go you see and I it's taking a
57:51: while to SAR so like they're in the wrong order and just there we go now would sarch it
57:56: um so can going to see the effect of it and if I go on the other side you know
58:02: like now it's not sorted again so it's there we go now it's the resorted um make it s a little bit more
58:09: often do you happen to have a uh a Splat that shows off like some cool Reflections or refraction um I'm not
58:17: going to complicate things right now I don't want to showcase the other thing I was showing
58:23: um so uh what I was saying is you know there's like a bunch of different things ging sper on the static ging split
58:31: there's some ex methods exposed uh the protox so if I go and
58:37: grab a protox tool um there we go I can ex
58:46: oh what I wanted I'm just going to there's a splot
58:52: that was grabbing uh so I can expose the method I'm going to proxy this and I
59:01: need to Fed it like in a bonding bonding box so I go pounds
59:09: math let's see uh where is it do you know where the bonding box
59:15: ones are transform bounce bounce
59:22: uh compete bonding box
59:27: is that one want no don't touch that one it's component
59:33: menu oh accident here you go is this the one no it is not the one
59:41: uh no that's not the one I want uh transform bands encapsulate bands
59:51: no hold on was uh
59:56: that's really weird why is it like where's the stuff it's not showing
1:00:03: everything um it is what is it under transform so
1:00:12: transform bounce uh there we go from Center size
1:00:17: that's the one I want from Center size uh so I just compose like one and kind
1:00:26: of have to like you know just guesstimate it but I'm going to use like you know one that's 4X 4x4 and then I run it and this this will
1:00:34: take a little bit but this a essential processing that's going to clip um
1:00:39: that's going to clip all the um all the gsan that are you know outside of this
1:00:45: bounding box there we go and you see how it's clipped so there's like very basic
1:00:52: editing options or rather like you know there's like editing options they're not like
1:00:58: exposed through like intuitive tools yet uh but they're there and like we can add
1:01:05: more and we can add like you know visual tools where instead of you know having to do this the tool like let's you draw
1:01:10: a box and then like you triggers it and it triggers this to process the ging
1:01:15: spad so you can do basic everything I would like to add more as well in the future and you know make it more
1:01:22: intuitive to use but there's like some things that you can um you know you can
1:01:28: build your own tools for anything also um is asking is the bonding box clipping
1:01:35: reversible or is it permanent I mean pretty much with anything like it's it's permanent if it's permanent if you save
1:01:41: this and delete the original so like as long as you have the original you can always go back to it
1:01:47: um is you know it's same like processing any asset on right like if you don't delete the original then you can always
1:01:53: go back to it uh but if the original is gone then you know it's gone you've deleted it um it do makes it much
1:02:01: lighter you know to process also one thing you can do with this you can actually export this so I can you know
1:02:07: go files I've already tested this but um it does support exporting actually this
1:02:14: one does oh this one doesn't because um this is an old import before that I have to set it up but you can export the
1:02:22: edited C and spli out of resite 2 there actually one thing I can show you I can show this one in the other world because
1:02:28: it's going to be too heavy um let's switch
1:02:34: back there we go hello I'm going to switch I'm going to turn the camera back
1:02:39: on as well M
1:02:45: display so I also got a bunch of like questions accumulated um
1:02:51: but uh where did go there's another thing I wanted to
1:02:56: show you uh there is a procedural gaussing SPL and this is just kind of to show you
1:03:03: what gassian splits look like so if I create empty object
1:03:11: um I go into assets there's procedural gaussian Splats diagnostic Gan Splat and
1:03:18: this is like one of the things I use like you know when working with rendering you can literally add the splits one by one uh I'm going to set
1:03:26: the render for this and I'm going to add a gassian Splat there we go this is a
1:03:32: single gassian spat and you can see it's essentially like a color
1:03:43: blob and this is like you know just the unit size so it's like one by one like by in every it's kind of grayish too and
1:03:51: if you look a bunch of different parameters for you have like the position so you have like you know the white position so I can you know make it
1:03:57: move up um you also have rotation that's not going to do anything because it's uniform size but you have the scale so I
1:04:04: can actually make it be you know 0.1 on the x- axis which means you know
1:04:09: it's kind of thin like this but if I look at from the side um this is a good one because there's a bunch of stuff
1:04:15: there there we go you see like now over this is it's still kind of like um it's
1:04:21: essentially more of a disc but each split has like you know a 3D Dimension to
1:04:27: it uh and if I change the rotation you know that's going
1:04:33: to uh it's not the right Axis what AIS do I need this one there
1:04:40: we go there we go so now now it's kind of rotated so each Splat can also have
1:04:45: like you know rotation so like they can be like you know stretched they can be thin they can be big uh I can you know
1:04:51: make it squished on another axis so it can be 0.1 on this axis and now it's likeing know this like thin
1:04:59: Splat um and they also have color and the color is specified using something
1:05:04: called spherical harmonics so I can put color here and then I can put also color
1:05:10: on like you know some of the other um you know some of the other
1:05:15: bands of the spherical harmonics I'm just going to put some random values there
1:05:21: um just to kind of see what it does and now like if I like move around
1:05:27: it I might have done wrong
1:05:32: bands I might have done like vertical ones or something um I'm actually do some of the lower bands that's going to
1:05:38: make it a little bit more obvious there we go um let me also change the default one
1:05:44: to so now when I move around what did I do these are kind of
1:05:52: hard Red by hand so I'm just going to like you know yeah I think it's up and down for for
1:05:58: this let me do this thing two in here going I do zero on this
1:06:07: one this is hard to like get the
1:06:14: effect why is much maybe these other
1:06:23: bins let's see if I fly up
1:06:30: uh let's see let me just return this back to what it
1:06:37: was so if I change this one this one should
1:06:45: be which direction is this
1:06:53: in these are puzzling let me throw this
1:06:58: one it's not really super human readable for
1:07:04: this that looks like red from this way and blue from this way does it didn't update for me yeah
1:07:13: it's right from here and then when I look at it from this way it's blue that's weird why is it not oh
1:07:21: there's something oh I think there's a bug
1:07:27: okay this looks like a bug because like it's gray for me but it's red on the
1:07:33: camera so the camera is rendering okay yeah so it's that is odd that's very odd
1:07:38: there's something wrong I see it okay more bags although it like it's
1:07:44: literally like for my viewpoint like if I disable this it's
1:07:49: gray but on the camera is red so there's a bug with the rendering so I was kind
1:07:55: of confused because it looked like most of it wasn't like doing anything um but yeah you can you can see it on the
1:08:01: camera now it's like you know it changes color based on the angle we view it from
1:08:07: and essentially all the G and Splats they compos from these and this is like a new type of primitive you know the way
1:08:15: like you have triangles for the traditional geometry you have you know you have U
1:08:21: gassian Splats for you know gassian SPL thing and because of this um this
1:08:26: representation they're you can see like all the Gan spells they have like you know this very kind of fuzzy Edge on
1:08:32: them which means like they're really good at representing fuzzy details like you know like fur or plants or
1:08:38: transparent things and so on so um it's it's like you know this new
1:08:44: primitive that kind of gives it like the ability to like you know make really photo realistic looking scenes and
1:08:49: people of of times ask like you know can we just like make a model out of it and I always tell them you know that kind of loses the point point because the cian
1:08:57: Splats look so good because they're gassian Splats you know because of how they work because of how they rendering
1:09:03: works so like if you converted this to just the typical model you know it's not going to look as good
1:09:09: anymore but yeah hopefully this kind of gives you like a bit a better idea it still needs like a little bit more time
1:09:15: to work on but I want it to give you you know little bit like sneak peak on
1:09:20: things that have been worked at and there's still like you know bags and stuff that need to be sorted through
1:09:26: um let's go back to the other questions because we have got a bunch of them kind of piled up um also need to enabl The
1:09:35: View on the camera there we go so hopefully hopefully you know that
1:09:40: kind of gives you a bit of a kind of sneak PE and a little bit better understanding like you know what Gan SPL are and how they're going to like you
1:09:46: know work in resite just just just a little sneak um so going back to the questions
1:09:54: let's Al double check on time still got a bit of time um so the uh the red Neco
1:10:03: is asking I had a second one that related audio system will there be increase in the number of audio outputs
1:10:09: playing once before the audio in the breaks in the new system um potentially
1:10:15: like we'll have to see like you know how it kind of performs um uh it's usually it's essentially just kind of like you
1:10:21: know restricted like you know by how much the power can have before the audio breaks
1:10:27: down uh so if it like you know performs well we can kind of increase the defaults it also might be kind of tricky
1:10:33: like um because we could like you know make it a set thing you know based on our Hardware but then the problem is if
1:10:39: you build a world and you assume you know everybody can hear as many sounds as you can maybe some people can and I
1:10:45: like you know they just kind of broken for them so like that's one of the three character things approach uh so we'll see we'll see how
1:10:52: it kind of goes I don't want to you to make like any commitments there right now uh next question is from dwy morph
1:11:01: speaking of splits will compress G and split format be supported as well or just P files um it depends what do you
1:11:09: mean by compressed format uh which format do you mean specifically there's
1:11:15: like one format I've been looking at called the spz uh we just need to like you know have a wrapper for it and make
1:11:20: like you know um have like a compile version of it and then it should be relatively easy to add support for it
1:11:29: um so the one probably going to be supported but I don't know if that's the same format you know you're thinking of so like if um it's the spz one then
1:11:38: probably has there's also like other part like is like having the SP compressed and VM which is going to like
1:11:44: know make them a little more efficient to render because it's going to use less VM um that's one of the things I also
1:11:50: want to add just to like because they take a lot of VM um
1:11:55: so that's also going to come at some point uh but it's kind of separated you
1:12:01: know from the file format compression that's like you know how to compress in the video memory
1:12:07: um next question is from shining hero one how Bly would Unity react to having the underlying mon version replaced with
1:12:13: even a slight update uh you can't really do that like the unity is built you know
1:12:19: with its own like mono version so like and they have their own Fork of it too so like you know like can't you can't
1:12:25: just update it even if you could it would react badly to say the least yeah
1:12:31: like and was one of the reasons like they don't update often is because they they haven't made a lot of changes to it
1:12:36: which means like every time they want to update it you know they need to like like backport
1:12:43: like all all the changes and everything so it takes them like forever to do it and even then like you know there's you
1:12:49: don't really get super much benefit from doing it because even the new versions of mono are like they're not potentially
1:12:55: faster than you know the other ones um the next question is from the
1:13:02: red Neco uh resending c not in crew oh this was we already answer that one
1:13:09: um uh Alex topi FR Vol or SPL and there I support movies in Splats no you need
1:13:16: you need all the your am for that also I don't know like what movies what do movies and Splats
1:13:22: mean I don't even know what it mean like basically like just like think of like
1:13:27: gifts but for Splats I guess no I thought it's like it's a it's a movie that's in it's each Splat each Splat
1:13:33: each Splat is a movie texture oh God I don't know uh I don't like I
1:13:41: thought that was just a joke one but like if you have like like more one like I mean like you could like do animation
1:13:47: with SPL there's like some things for that but like until like there's something we can like you know easily
1:13:52: adapt and use like I don't think we're going to integrated it like until then um I'm probably not going to do a bunch
1:13:59: of kind of research because like they're already like very heavy in memory and for movies like you need a lot of data
1:14:05: um so like that there there needs to be a lot of work into those kinds of
1:14:11: formats um next question is from uh uh KRA blades I don't know much about Gan
1:14:17: splits uh would you be able to like cut some of oh yeah like we showed that uh
1:14:26: um uh that is it got some of the fuz bits got only have the good bits of the
1:14:32: models it can be integrated with other stuff in SPL mode as cies for social place um yeah you can like cut them out
1:14:39: the only thing is in some Splats you actually need the bit because like they do stuff like you know like Reflections
1:14:46: because sometimes like often times they do Reflections where it's literally sort of like a mirror version of the world
1:14:51: like you know that's on the other side so if you cut those pieces off you actually lose bunch of the visual but
1:14:57: yeah like like like we showed earlier you can kind of cut them off um do I more yeah like we show this
1:15:05: one will'll be able to edit plus with res yes like you'll be able to I you kind of like you know you'll be able to
1:15:11: like with a simple like prolex method but uh we want to add like nicer tools too uh Shing here want ask I hope there
1:15:19: so trim cing spots as well uh yeah like you can do it like you know um you can
1:15:25: do it in some of the tools it's a little bit harder to do in desktop because like you know like in the cool thing in VR is
1:15:30: like you know I can just literally grab a tool and be like you know I'm just going to use like here I can be like you
1:15:36: know I want to crop this and we just kind of draw the Box around versus on desktop you have to be like okay I have
1:15:41: to be like I have to align here and then I have to make sure it's aligned here and in VR is like more kind of intuitive
1:15:48: like you know to do it it's even harder if you want to like you know just clean them up so like you know there's a bunch of like Splats over here
1:15:55: like in VR you can just grab a tool and be like okay going to remove here and going to remove here and a little bit
1:16:01: here like it gives you like you know that kind of like spatial ability to work like specially
1:16:07: which I think is like you know really beneficial and make things a lot easier
1:16:13: um next question shining here one also what what's that glitch for doing the family get death pose yes he was he does
1:16:20: that a lot it's been cute
1:16:26: next question from dagy morph uh also what do you use to make your Splash
1:16:32: direct with camera scanner or from images and videos I make my from like photos I take I use a software called
1:16:39: jaw set post shot to like reconstruct them uh I usually take like R photos and
1:16:44: I kind of process them like do like d noise and some extra processing and then
1:16:49: process them like you know the the casting split software uh um Jack the fox Au is asking
1:16:57: are gas plus part of the performance update they seem to help with Shifting the but L to the GPU they're not it's
1:17:03: just like a fun project kind of on the side um they do like you know they're
1:17:08: very they use the GPU a lot but they use it for themselves and they don't really help like you know they don't help with
1:17:16: any other stuff so like I would considered part of it there's like one
1:17:22: oh sorry say there's no sorry you you there's like one B like
1:17:29: there's like few kind of things that are that actually end up like being related to the performance update one of them is
1:17:36: you know adding the support for the spherical harmonics which will use like um for a bunch of like different things
1:17:42: you know like Photon does there going to be new effects and so on but also to kind of like migrate more stuff from Unity into FKS engine like you know for
1:17:49: example the ambient lighting which also uses spherical harmonics the same that go in Splats do
1:17:55: um this vehicle Harmon is also going to be used you know for audio system for ambisonics and so on so there's like
1:18:01: stuff that's going of related there um if I can also show that one I should
1:18:08: probably shouldn't right now but um we'll see um because like one of the
1:18:13: things is like you know the ambient lightting in the world it actually is also uh encoded using spherical
1:18:20: harmonics which is what the gasan SPs you know use for the like you know the individual pieces so uh the code either
1:18:26: for spherical harmonics for the gassian Splats that's also useful for other things some of them related to you know
1:18:33: the performance update like the audio system and like moving some of the lighting so it goes you know from being
1:18:38: computed on the unity side you know the FR engine side um so there's like bits
1:18:45: there um one of the things that like where actually it might end up being
1:18:51: sort of related is because I had to like you know work with like um
1:18:57: make plus they're rendered using compute shaders um so I've kind of like you know
1:19:04: had to like integrate that like into the kind of pipeline that we have you know especially with the one that resonite
1:19:09: uses uh because it has to you know do all the soring of them but also has to compute all the properties for the
1:19:14: supplies and shade you know Gan shade the spherical harmonics for each car in
1:19:20: um I might end up like you know writing a comput Shader to do similar thing for the particle system for Photon dust uh
1:19:28: because there's still simil like kind of performance issues with Photon dust like when it comes to submitting the simulated data to
1:19:35: Unity um I don't know if it's necessary yet I still need to kind of look into it
1:19:40: deeper but it's possible we might need to like you know bypass like just kind of not use unit this like particle
1:19:46: rendering at all and just have our own using compute shaders So like
1:19:52: um uh working C SP like kind of helps like you know get a B more familiar with
1:19:58: with that so it might help like you know in that regard of like not super
1:20:03: directly but kind of indirectly kind of helps fuz bip calling me a blow
1:20:10: rud talking about in specific uh oh we that one um Alex topi
1:20:21: I ask about possible support for movies plus and resonite um I think we already answered that yeah
1:20:27: we answered that look can we bake a set of these diagnostic gas PL as one could be interesting to make brushes of these
1:20:34: uh you can bake it you couldn't really make brushes of it like
1:20:39: um like I know what it would do like it's not really a thing like I would use for a brush I don't think it's like
1:20:45: super useful for that also like you know like You' have to like like they don't one of the things
1:20:52: the cting SP they will not blend with each other um like they kind of like rendered
1:20:58: as its own like unit so you have two of them like they're not going to blend you know any like super useful
1:21:05: ways like it's not useful for that um next question is from the jet
1:21:11: Forge is it possible to SPL photo realistic avatars so for the rigging part you could actually do it um like
1:21:20: like the basis of it like you know you do have like each calcian is determined by Point um the rotation and the scale and
1:21:29: technically you know you could have you know a matrix that binds it to a bone
1:21:34: and you just recalculate that you know and move it around um move it around you
1:21:40: know as the bones move you to calculate position so theoretically yes you can rig them uh you would need like you know
1:21:48: some kind of tool set to do it I don't know if there's any software that lets you do that right now um in terms of like the math of it
1:21:56: like you know it's not a problem in terms of the math uh I don't know if there's any
1:22:01: tooling for it but even if there was tooling for it I don't think it's
1:22:06: necessarily super useful for avatars because there's been like there like some Pro Plus where they like you
1:22:13: know they actually do them but it's like you know specific to that project and you can kind of move them around but the
1:22:19: problem is gas and SPL they don't interact with the lighting in the scene all the lighting is you know baked which
1:22:24: means your avatar uh for one the Avatar is going to be really
1:22:30: heavy you know like you're you don't want to like you know like use us amounts of GPU just to
1:22:37: render your avatar um it's not going to blend well with other avatars because like know
1:22:43: mentioned earlier the G plus they don't blend with each other um you could make them blend with each other like with
1:22:49: lots of extra effort lots of extra performance but you know it doesn't right now uh it also inter with the
1:22:55: lightting so like you know if you're in dark environment you're going to be glowing if you're brand new environment you know still might not fit in because
1:23:01: the light the lightting is effectively baked into it so like I I don't think it's a other than like you know using it
1:23:08: as a joke I don't think it's useful for you know that kind of stuff it's is
1:23:13: really most useful right now for capturing you know Real World scenes and
1:23:18: then like you know rendering them in VR so we can kind of look at them and preserve like you know their graphical fidelity
1:23:25: um but I don't know like how useful is it for interactive stuff like you know
1:23:31: avatars and so on um next question is from Alex topi uh
1:23:41: there's new paperwork people are starting to create moving videos in Splat RS yeah there's like there's like
1:23:48: some research the only thing is like you know until it makes it into some like um
1:23:54: code you know that's like public and it's can be used and some format uh that we can like easily ad out like you're
1:24:01: probably not going to see it because we' have to like you know um we have to do a
1:24:07: lot of that kind of work ourselves and it can take months and we're not going to we cannot afford to dedicate you know
1:24:12: months like to doing ging pling kind of research ourselves you know to have it
1:24:18: in here uh that's not like main focus so the best thing is like you know like
1:24:23: once CH make it into some kind of like you know more standardized formats once there's more like standardized kind of
1:24:29: tooling we can adapt it but we're probably not going to spend it amount of time like you know implementing that uh
1:24:35: all this kind of ground workor ourselves um next question is from Zan
1:24:42: 50250 I'm trying to research for technical details about gasing splitting may I ask this feature related to paper
1:24:47: 3D G sping for real time um R Field rendering publishing
1:24:53: August 23 uh yes yes it's related to that paper I
1:25:00: mean that one I think that's like the first one maybe that like you know kind of published or like one of the first
1:25:05: ones so it it is like you know that function and kind a lot of papers kind
1:25:11: of researching and so on like the rendering we have is like you know essentially just rendering the basic like static place and there's like a lot
1:25:18: of research like doing like a different things but like you know it's still within the REM of the research or like
1:25:23: you know very specific ABS um so next question is from SNB
1:25:31: 8272 uh will there or have there been any attempts on getting system for converting shaders from VRC to
1:25:38: resonite um that's very unlikely to happen
1:25:43: shaders like shaders they're essentially code and conversing code from one system
1:25:49: to another like that's very complicated it kind of task you know like where like
1:25:57: you really get systems that will be able to convert like all possible code into a different
1:26:03: system and if it's still work um we so like right now like even
1:26:08: the issue is like you know we don't support like custom shaders um which means all the shaders
1:26:16: like kind to like pre-built and one of the reasons for that or the main reason for that is because while we're still
1:26:22: tied to Unity we cannot guarantee that the code will keep
1:26:29: working like it won't keep working like only once we switch the graphics engine because it's um it
1:26:37: works Fair bit like you know like it's going to work like very differently
1:26:44: um and like one of the things we want to do with everything on resonite is make sure we can maintain compatibility
1:26:50: whatever you build we want to make sure it can keep working and we can adapt it
1:26:56: to keep it working like you know with whatever changes we make to the system you know this for example with the
1:27:01: photon dust like we have a system that converts the Legacy particle system into the new one and like you know most of
1:27:08: the times like it just works and there's like still few cases to be fixed but it pretty much like you know things will
1:27:13: look and feel the same um and we want to make sure we can do the same for shaders
1:27:20: if we implemented custom shaders right now we have to use unit system system
1:27:25: but there's not really a reasonable way to convert those shaders into whatever new engine will switch to eventually
1:27:31: which means all those custom shaders would break and is one of the reasons we don't have the support right now and the
1:27:37: reason like you know we the reason like it's not possible to keep supporting them is because you can't really convert
1:27:45: code automatically to a completely different system um so there's not a system I
1:27:52: don't think like anybody like tried one I don't think it's even like you know possible like not like with like
1:27:59: reasonable kind of success
1:28:04: right um
1:28:10: yeah I think it's like uh it's one of the things like where um
1:28:17: like you know like there's like some tools that can like you know can get you like you know certain amount of time
1:28:23: like the they can get you a little bit there like they kind of preon convert some patterns and still have to like edit it
1:28:29: and yeah it doesn't really work like the the closest like the most successful
1:28:36: you'd probably get with like the least amount of effort is like like let's say
1:28:41: you had uh a physically based Shader right uh the closest you could probably
1:28:48: get is just plugging the same like textures and parameters into the corresponding engines also physically
1:28:54: based Shader um in ter in terms of like it's basically like asking how do I
1:29:01: run my Unity shaders in on real engine like it's the same level of difficulty yeah it's also like because like some sh
1:29:09: some shaders can be like you know build like for specific way the rendering pipeline works and Achieve certain
1:29:15: effects you know based on that but if the pipeline Works different then you need to do the effect like a different
1:29:20: way or maybe it's not even possible with a different pipeline you know to get the same effect um so it kind of becomes you know
1:29:29: one of those things just it doesn't translate automatically usually have to like have
1:29:34: somebody figure out how to do similar effect you know with a different one it kind of depends on the Shader but like
1:29:40: it it is one of those thy things also fuz bipolar bear is asking are you feeling any better getting over concr
1:29:46: yeah it's like it's better than was like a Friday like Friday I pretty much like was bad um I like a fever and and such
1:29:54: so I'm just kind of like more kind of blun stuff you know um but is it is getting better everyone
1:30:03: seems to have gotten hit by the freaking by an annoying head cold got the
1:30:11: plag kind like to say but yeah uh so that's pretty much
1:30:18: all the questions right now um we have about 30 minutes left so there still
1:30:24: plenty of time for questions like if we could some more make sure like you know to like get them in before before the
1:30:29: time runs out um just so we don't get like you know like any last time questions like um like last time um but
1:30:39: uh we can like about some other stuff I can actually show you some other things so the other thing I wanted to show you
1:30:44: with the spherical harmonics uh let me change this back to SM
1:30:50: POV so all the liting in the world um if I make like a make like a sphere
1:30:59: how do I already have a sphere um so there's like you know multiple things that kind of contribute to the
1:31:05: liting of the sphere actually lies in the scene but there's also like you know unbe Li thing
1:31:12: um there's also unbe Li thing that's you know uh computed you typically from the
1:31:19: sky box and right know like you don't in normal bills you can't control de but this is actually a new component uh this
1:31:26: is not really polish so this might actually look different uh but I kind of like you know
1:31:31: give you a little bit of peak um if I go into rendering there's uh where
1:31:39: is uh ambient light sh2 and this actually uses second order
1:31:47: spherical harmonic you know to determine ambient lighting and if I know plug red
1:31:52: here for example now there's like a red ambient light everywhere and you see it
1:31:57: immediately immediately you know I fix the scene and everything's red red is all red uh so like the the
1:32:06: first the first band you know is literally just like ambient is like covers everything but if I do the second
1:32:12: band is going to be like one side so now it's also red B is red you know only
1:32:20: from Top you see and the bottom is like dark and seem like everything there is from the top could also change it you
1:32:26: know to be like uh I'm going to do blue blue now it's blue from the
1:32:32: top and if I actually reverse it if I like you know plug minus one if I go
1:32:38: minus now it's blue from the bottom um if I plug it you know if I
1:32:46: plug it here this is going to be from one axis so now you know it's blue from
1:32:51: this one so it's almost like you know there's like blue light kind of coming from here and you can kind of see it on
1:32:57: everything you know over there um and then if I go into the even higher
1:33:03: ones you're actually going to see this kind of creates this kind of like band
1:33:10: effect see like like um let me actually use the different axis one so it's a little bit clear I'm going to use this
1:33:18: one here we go there should be there's a Bend here and there's Bend here here um
1:33:25: so this is the way uh to De like you know the lighting is like the amp liting is kind of encoded and I can actually
1:33:31: just plug a bunch of random stuff here um so see know we kind of created like
1:33:37: this kind of funky ument light thing um and I can actually control it and usually like you know it's it's m so
1:33:43: actually kind of matches the environment I'm just kind of plugging random colors in here but actually looks kind of neat
1:33:49: with this world um It's kind neat
1:33:56: looking uh but essentially it is using you know the spherical harmonics to
1:34:01: control how the ambiant lighting works so it's one of the things that's probably going to be exposed and uh what
1:34:08: probably I'd like to so you can kind of compute it you know from a texture and then you can also blend it so like you
1:34:15: know like if you if you like you know move around environment you could like what is that that is not me that is
1:34:23: the that something in the world that's so weird
1:34:29: um anyways uh we want to make it like you know so it's um um so you can kind of blend it so
1:34:37: like you know for example if you move around the world light thing will also change so it's going to expose like you know some new options but the basic
1:34:44: building block that is used for that you know is the spheral harmonics which originally you know I them because of
1:34:50: the gassian plus because they use them to encode color but it's also useful you know for this other thing uh it kind of
1:34:57: gives you like more control over stuff um I'll actually keep this on because I kind of like how it looks um I can also
1:35:04: show you uh if I create another empty object I've
1:35:11: added uh I'm actually I'm going to no this works um I'm going to add a
1:35:16: procedural asset uh procedural misses uh there's I
1:35:23: spere icosphere measures sh2 so what this does um this is like the icosphere
1:35:31: procedual MH except the uh both colors like vertex colors and
1:35:39: the radius um you can modulate them using spherical
1:35:45: harmonics um so if I set up a render is actually using the uman
1:35:52: lighting as well of the world so it's going to [Music] um I'm not going to show the colors
1:35:57: because we can kind of see that with that but it's kind of funky if you play with the radius because right now it's
1:36:03: only using you know the first band which is like it's the same everywhere um I will switch it to
1:36:10: different material so we can kind of see uh positive and negative better though so go verx
1:36:19: color metallic there we go
1:36:24: uh don't want this one plug in the vertx color metallic
1:36:30: there we go uh so I think it's like inverted by default um no it's dark by default mind
1:36:38: so if I actually remove this one and I only give it like the positive band you see like it kind of creates
1:36:45: this kind of shape um and I'm actually going to increase the number of subdivisions so it's kind of smoother
1:36:50: there we go um and what you're seeing is like you know it's still you know the
1:36:57: base is the sphere but what is happening the uh function the radius is being
1:37:03: modulated based on the direction and this band you know it it goes like vertical like this which means over here
1:37:10: the radius is zero and as it keep increasing it actually goes negative here that's why this part is actually no
1:37:16: this is positive because the color are inverted um it goes like you know it
1:37:22: goes positive here and negative here and it's visualized you know by the color
1:37:29: um and if I change this one to be on this
1:37:34: one you see it kind of like rotates it to I'm going to plug zero here it's
1:37:39: essentially the same thing except it's you know um on this axis and the way you
1:37:45: can think about spherical harmonics uh I'm going to grab my brush
1:37:51: here brush give me R I'm not used to these controllers
1:37:58: there we go the we can thing about spherical harmonics it's like um it's like having
1:38:05: like a standing wave except it's on a sphere so you know over here imagine we
1:38:11: have like you know like you have like start and end and you have like a wave here you have like a wave that goes
1:38:18: positive and it goes negative it's just like you know a sine wave it's essentially this except is wrapped
1:38:24: around the sphere so you know on one side it goes
1:38:30: you know it goes positive and on the other side it goes negative and this one it just kind of
1:38:36: like you know it it flips it so like it's kind of when it goes negative it still like flips it so it goes like this
1:38:43: but essentially it's like you know it's this except flowing a
1:38:49: sphere if it kind of makes sense um and this is you know this is the
1:38:55: first order that's you know why it's just like one like this um if I go and
1:39:02: it's you know and you have like one band like you know in each kind of Direction uh if I go here like you know
1:39:08: it also does it like in this direction but then when I go into the higher order bands so let's do this one two you see
1:39:18: now it does this kind of shape and the reason for this is because for the second order the sine
1:39:26: wave it goes like this one two and as you know how it wraps around
1:39:32: this is like you literally have it like you know is literally this peak you know much just this peak or
1:39:40: maybe like it's reverse but doesn't matter this big you know is going to match this one this one is going to match this
1:39:48: one this one's going to match this one so it's kind of you know it's almost
1:39:54: like it took the sine wave and like you kind of wrapped it
1:39:59: around and it's also like in three dimensions makes them little bit more complicated but essentially it's this kind of
1:40:05: principle um and what this does it's like it's a way of like sort of encoding information
1:40:11: where you know it can be sample directional because you can kind of combine these different bands you know
1:40:18: to create all kinds of different you know functions on the surface of the sphere so I can like you know plug one
1:40:23: here or maybe I'm to put a negative one here do like this thing and do a thing
1:40:31: here I'm going to I'm going to do a thing
1:40:37: here I don't want it to be that big you can create like you know all kinds of like weird
1:40:43: shapes um and like you don't need you know that many par me for this like uh
1:40:49: the sh2 which is like in know second order which has nine uh coefficients is enough you know to
1:40:56: encode like the ambient like ambient liing information like this um in the world and doesn't need
1:41:02: much data to do it these uh Gan SP they use third order which is 16 uh channels
1:41:10: and that can encode even more information um so it's essentially like a mechanism
1:41:17: you know to like encode information because if you think about a sphere each point of the sphere it's a Direction
1:41:23: from the center of the sphere you know to that point so if you if you want to like you
1:41:29: know sample point on the sphere you're essentially know taking something that's inside like that's go from the center
1:41:35: and you have a directional Vector um and if the sphere is like unit sphere
1:41:41: this is you know just unit Direction You sample the spherical harmonic and it gives you it gives you color or it gives
1:41:46: you you know radius or it gives you you know audio amplitude for the ambisonics
1:41:52: so like if if you have like you know an audio Source here uh the spherical harmonics it
1:41:58: encodes the audio channels the audio channels are the individual
1:42:03: coefficients and then you know you know you sample it based on where the listener is relative to the audio
1:42:13: Source um and this is kind of cool waying you know to going of be able to visualize it there's actually one thing
1:42:18: I also wanted to bring in um let me also not getting any more questions like if
1:42:24: if you got any more questions uh make sure like you know to post them before the stream minut uh I'll try to at them
1:42:31: before like any the sense but if there's no more questions we probably not like rambling about things for a bit
1:42:48: um I did want to showcase so there's like this kind of diagram
1:42:53: um that you can kind of see a lot like in different displ because like um the spical harmonies they they're one of the
1:43:00: like building blocks you know like vectors and other stuff that kind of comes up with a lot of other stuff if you look at this diagram this is this
1:43:07: should be very familiar this this is literally like you know what I've showed you with
1:43:13: this because I've always been like what does this mean like you know what what is this visualizing and this is
1:43:21: literally just the value you where the spherical harmonics modulates the
1:43:28: radius so if I if I change know back uh go say zero if it's just the first band you
1:43:35: know then it's just it's literally just a sphere you know as the first one uh if
1:43:42: we go for the second
1:43:48: band the first order you see it makes that kind of shape that's exactly that shape
1:43:53: uh there and there's like you know three of them then the second second order you
1:43:59: know there's a bunch so like if we uh have one two three four so this
1:44:05: one this one makes this shape you know and this matches our shape
1:44:11: here and then we also if we plug this one to this
1:44:17: band you know this makes shape on a different axis um
1:44:23: and plug this one here this makes this kind of
1:44:30: shape does it look familiar is that
1:44:36: shape and then this one this is going to now to kind I get them
1:44:43: combined and this mix this shape so like is literally these shapes and all this
1:44:49: all this indicates is the radius modulated by the you know value of the spherical harmonics function you know at
1:44:58: each point of the sphere with like the colors indicating like where it goes positive and it goes
1:45:04: negative because you know if if you got a sine wave it goes you know it goes positive and it goes negative positive
1:45:10: negative same same here you know it goes positive negative positive negative and
1:45:16: if you add like more bends you know this one you can see like it now it goes like you know three times so the third order
1:45:22: is you know positive negative positive negative positive negative like you know it goes
1:45:28: through three three waves essentially and the more orders you add like you
1:45:34: know more the more complex the shape because the more complex the information you can encode
1:45:39: is um I guess so you can almost think about it like um
1:45:47: you can almost think about it like the the different frequencies that like make up a sound wave for example because the
1:45:52: the that they represent is constructed out of their interference with each
1:45:58: other kind of I mean it's kind of waves in general is like um like these are
1:46:04: sort of like you know the basis ones and then like you know you combine them to like create whatever function you want
1:46:09: usually like this is like an extreme variation you know so like this is going to be like this kind of like you know
1:46:15: kind of showcase um but you can use these kind of you know to constructively destructively
1:46:21: like create like some kind of like Smooth function it's actually the same kind of it's similar approach that's
1:46:26: like used for encoding images with like jpeg because with jpeg um with jpeg what
1:46:32: you do um you essentially uh vary like
1:46:37: you know kind of frequency information in a block of 8 by8 pixels I for my brush there we go you know so like in a
1:46:44: JPEG you're essentially overlying you know like bunch of things and it can have like a very high frequency detail
1:46:50: you know that goes with like low frequency detail and like it ends up like you know making you know like if if you want like
1:46:58: very big detail you can have a function that's you know that's going to be bright here and low here and what this
1:47:04: is you know is this is going to be like this you know and then you're going to have like information that you know varat a lot
1:47:11: more so it's going to create like you know kind of bands and if you have enough of these you can actually
1:47:16: constract pretty much any um you can constract any image you want what it is
1:47:22: you're essentially using you know the for transform where you translate you know the spal the information from spal
1:47:29: domain into frequency domain um you comp you can compose any
1:47:35: wave out of like you know any frequencies as long as you have like any of them and it's kind of the similar thing if if you had enough of these you
1:47:41: could you know have arbitrarily precise information on the surface of a sphere
1:47:47: and the spherical harmonics is what makes the encoding kind of like you know efficient especially like if the if you
1:47:53: don't have too many bands it makes it very efficient to encode low frequency detail so like you cannot like you know
1:48:00: have like something that's like um something that's like you know a color
1:48:05: that would be like you know in very small area very different the information is very low frequency which
1:48:11: means it kind of varies more slowly for but for stuff like ambient lighting that's perfect because that doesn't have
1:48:16: very high frequency details yeah would you get couple more questions
1:48:23: actually yeah there's more questions um so Lucas is also asking so far these
1:48:31: examples use second sh will there be higher order ones so um cin SP actually
1:48:36: has third order ones so like uh if I switch to this um you see for the gasan
1:48:42: split it goes all the way to sh15 uh there's also data structure for
1:48:49: sh4 uh so if I switch to this
1:48:55: um and I'll create just name the
1:49:02: object uh I just go data I'm just going to make Vol
1:49:09: field just another cool thing about these like I've implemented
1:49:15: these very generically so I can be replace FAL
1:49:21: harmonic um and then you know I'm just going to use float oh actually I forgot to the order
1:49:29: uh so right now the hard ones all all the way up to fourth order so I can add
1:49:35: you know fourth orders for harmonic and these have like you know 25
1:49:40: coefficients um there's nothing that like uses these right now I do want to add like so it
1:49:47: actually generates the icosphere for all of these all the way off The L4 um if
1:49:52: there's interest I can Implement like you know arbitrary level of support but that like
1:49:58: that's going to make it a little more complicated for the data model um and I don't think like you see like high order
1:50:04: ones like super often anyway so I'll probably done one for later but right now they go all the way to L4 uh within
1:50:11: the data model is another question you know how much can to change with the prox can you drive it for example yes
1:50:18: it's literally like it's not that different you know from other data tabs like these you know
1:50:24: vectors so if I grab this um uh let me
1:50:29: actually do I'm going to show it with this one because this going to look look fun okay um so if we go back you know to
1:50:39: this kind of thing um this is just a value like you know like any other so I
1:50:45: can drive it and there's a new set of nodes there's only few basic ones probably
1:50:52: going to need some more in the future uh wait what is happening sorry I was selecting
1:50:58: something oh um so if I go math there spherical
1:51:04: harmonics you can pack them and there like evaluation so we can like you know you can plug any spherical harmonic and
1:51:10: evaluate value at like vector do you want uh this one is second order so I'm going to pick this
1:51:16: one and I pick the type float so I can you know essentially just
1:51:24: pack it and I go like one like I'm going to move this a little bit
1:51:31: closer and say like I just want to you know animate like do Funk stuff with this just kind of for visualization
1:51:38: purposes um I'll go up time for
1:51:45: time and I'll I do like Simplex noise I think
1:51:54: yeah it works nice uh actually works because it go s negative too could also just use
1:52:01: um some size there we go plug this
1:52:07: in go one there we go PL this in I just PL this into this and now I have this
1:52:15: blob thing whatever this is actually looks kind of neat that is
1:52:22: it yeah hold on let me um let me change the color so it's also
1:52:31: like oh that's what I want that's too much
1:52:42: um there we go
1:52:48: oo but I can like you know mess with it and do like whatever stuff you want
1:52:54: um I'm going to do some more I'm just going to
1:53:02: multiply uh where's multiply my dumy operators
1:53:10: uh there we go I'm just going to plug this in so I can just V it a little bit so they're not like all the
1:53:16: same um so I'm going to like
1:53:23: 3 something random value Rister Simplex to just go here oh that's too much
1:53:31: actually that's too fast there we got look know this funky
1:53:38: shape so like you can drive them you know you can like mess around with them um you can sample them I've kind of
1:53:45: hinted at this like little bit earlier so if I go
1:53:50: moth uh sphere harmonics there's evaluate someone kind of helpers because
1:53:57: like you need to specify the spherical harmonic you want so one
1:54:03: spiral har harmonics oh two of type
1:54:14: float and the type is float I get out of it
1:54:19: float did I do it wrong I did something wrong
1:54:24: um spherical harmonics what did I do
1:54:31: wrong this is spherical harmonics L2 float oh it misspelled spherical
1:54:43: me there we go so I got the evaluate
1:54:48: node so I can actually you know be like I want I want to know what is in dispersion vector and right now there's
1:54:55: not one so I'm going to do it here oh it's not updated oh need M
1:55:00: continues up but that's weird why is not updating by
1:55:07: default something's clogged
1:55:13: why what is wrong with this did I BK something I still have
1:55:20: bugs yeah it's kind of like we you know work in progress and stuff so like I think there's OD why is this not
1:55:31: evaluating yeah should be able to evalate them but there's something wrong right now
1:55:38: um yeah working progress build wow wow oh oh there we go are you animating
1:55:46: there we go you're animating the Ambiance to color x with offset a couple
1:55:52: of them yes there we go that's really need so I can create like all kinds of
1:55:58: like effects with that wow it's kind of like almost like
1:56:04: rotating that's crazy it pretty much is I think like they're like offset like it
1:56:09: it's like when you combine them like it essentially creates this kind of like rotating effect you might have like seen
1:56:15: it like earlier too is like whenever was combining some of them it's kind of funky like when you
1:56:21: look at this too this just going oh wow
1:56:27: it's like a hole in the middle sometimes one thing I'm like you know
1:56:33: just hoping somebody will make with this is like like a audio
1:56:38: visualizer yes but yeah hope hopefully that also answers you know those extra questions
1:56:44: is like you know you can you can work with them in with prolex my goal was to essentially add you know a new data
1:56:51: primitive you know the way vectors are the way matrices are this is you know just another one because it's useful for
1:56:57: so many things so the cin FL there were sort of like the initial impulse to do
1:57:02: it but then I was like oh wait these are useful for this and they're useful for that and you know and for this other
1:57:07: thing and I made sure like you know the implantation is very generic so like with any data type that does support um
1:57:16: that supports you know um multiplication or specifically scaling you can use it with spherical
1:57:23: harmonics like you could make spherical harmonics like you know that like has matrices each point which like I don't
1:57:28: know if there's like any practical use for it but somebody will make something crazy with it um we also have like about
1:57:37: two and a half minutes left so like that's pretty much um like about time to like wrap it up
1:57:44: um yeah so uh yeah I think this kind of covers
1:57:50: everything and like anything yourself um no I think that this is uh
1:57:56: magical and now I want to play with it for 10 hours straight yeah yeah people will have like lots of fun with
1:58:04: it okay um I think we're good to like um we're going to be good to like end it
1:58:10: here so okay uh thank you very much everyone you know like who has like joined uh I hope
1:58:17: like you enjoyed like this episode of resonance uh I hope like you enjoy like the everyone had like you know good New
1:58:24: Year um like you know enjo like the holidays and everything and thank you everyone for like you know asking
1:58:30: questions um you know people kind of staying with us and uh you know for just
1:58:36: kind of in general supporting our as night um thank you also know for being
1:58:42: here like for you know joining on the stream helping answer questions and showcase
1:58:47: things I'm glad to have made this one uh
1:58:53: hopefully next time we'll be less diseased yes yeah was kind of like just
1:58:59: I still kind of like blah today so it's a little bit but play was like the
1:59:05: stream was like enj regardless um yeah so thank you very
1:59:10: much uh thank you for like being here thank you for supporting guys night um
1:59:15: and we'll see with the next one uh actually what's the next one it's going to be okay that's on the 12th so I'll
1:59:22: still be around uh the one on the 19 probably
1:59:28: will have to skip because I'll be like at FC um so I don't think I'll be able to
1:59:34: do one then so the next one is going to be around but uh the next one the one
1:59:39: after is going to be skipped um okay so
1:59:44: anyways uh thank you very much you know uh thank you for watching and we'll see you with the next one
1:59:54: i w where's the OBS uh
2:00:00: [Music] blind okay there we go w