Cloud updates

Cloud updates are, as the name suggest, updates specifically to the cloud, often to add optimizations or fixes. Sometimes, such updates can have user-facing changes with the addition of commands for the Resonite Bot.

2023/09/27

The following notes were sent by Frooxius:

I've been looking into an issue with the cloud and migrations, looks like there are some leftovers from the half-way registered accounts, which were missing some necessary entities to track storage usage.

I've pushed an update which will correct those accounts. It might take a bit to process. I'll keep monitoring the situation as it goes.

Bugfixes:

• Added mechanism to fix half-way registered accounts without storage tracking.
  • This should fix the accounts not being able to sync/migrate anything

2023/10/01

First update

The following notes were sent by Frooxius:

I've been looking into how to help the asset variant queue and did some tweaks and fixes that should help make the queue go a fair bit faster.

We're also adding more computation workload when we can. Things will probably still take a bit, but this should help it a bit.

We appreciate everyone's patience and understanding with this!

Tweaks:

• Mobile texture variant computations are now deferred, since they're not necessary right now

Bugfixes:

• Fixed asset variant computations failing when temporary file fails to be removed on the computation worker
• Fixed completed asset variants not being accepted by the server in some cases

Second update

The following notes were sent by Frooxius:

I've looked into things some more. Looks like there's an issue, where lots of variants have been re-queued multiple times (potentially dozens of times).

This means that a lot of the current queue are duplicates, which should go down pretty quick once the actual bulk of them gets processed (the system will not compute them multiple times if they're already computed, it'll just skip).

The issue is that we don't know how big the "real" queue is, but it's smaller than the actual number of assets.

This change should prevent the queue from growing with lots of duplicates from this point on, but it will just take a bit to process through things for it to go down.

Tweaks:

• Fix asset variant computations getting re-enqueued too frequently, resulting in a large number of duplicate queue entries

2023/10/02

First update

The following notes were sent by Frooxius:

Some more work on the cloud to make sure things will run more smoothly. The tracking of asset variants has been improved to be more accurate and exclude duplicate jobs - you'll see the number jump down quite a bit as a result.

Some improvements to the cloud as well to decrease DB load of some things and make things more responsive.

I'm still investigating some issues that are affecting some accounts, so more to come in a bit.

Tweaks:

• Improved asset metadata & variants job count statistics
  • These now include actual distinct job, rather than number of items in queue, which includes potentially large number of duplicates
  • There are now separate statistics for metadata and variant jobs
  • Added statistics for completed metadata and variant jobs to help track progress over time
• Optimized count queries on the database backend
  • This significantly lowers database throughout load for certain queries, resulting in better responsiveness and lower usage
• Optimized cloud statistics endpoint, to sigificantly lower database load
  • The statistics are now computed on background worker and then cached, rather than computed on the fly

Bugfixes:

• Fix account deletion emails not being sent
• Fixed record storage usage statistics email not being sent
• Fix storage entitlement failing to process when storage sources are not setup for user's account

Second update

The following notes were sent by Frooxius:

Sorted out some more issues with cloud and registrations.

If you registered and never got an email to verify, please open a ticket at https://support.resonite.com/ Please include your username in the ticket, otherwise we won't be able to help you.

Bugfixes:

• Remove = symbol from email verification token URL's
  • This fixes the verification link not working with some email systems (reported by @raithsphere)
• Fixed errors causing registration to not complete

2023/10/05

The following notes were sent by Frooxius:

Pushed a fix to the cloud that should fixup the issues with contacts that are stuck as not migrated.

Bugfixes:

• Add cloud job that corrects all contacts migrated status
  • This should gradually fix any instances of "Contact hasn't migrated" that already occured over next few hours (might be quicker) - you will need to restart for the change to be reflected
• Fix contacts being stuck as "Contact hasn't migrated" when changing the other contact

2023/10/06

The following notes were sent by Frooxius:

Another update, this one is just cloud-side, so no need to update anything on your end. We changed handling of exit messages and added command that lets you submit them!

New features:

• Exit message benefit from Patreon is now lifetime and accumulative
  • If you get exit message quota as a benefit, each month you get additional messages, depending on your tier. In other words, if your tier gives you 1 exit message and you support for 3 months, you get to submit 3! If you support for another month, you get to submit 4th when the month rolls over!
• You can submit exit messages using /addExitMessage <message> command sent to the Resonite bot
• You can list exit messages you have submitted by sending /listExitMessages command to the Resonite bot

2023/10/07

The following notes were sent by Frooxius:

Found some more issues, pushing fix.

Bugfixes:

• Add limit for maximum share bytes for storage quota sharing
  • This fixes deserialization errors when processing storage updates, resulting in server errors

2023/10/10

The following notes were sent by Frooxius:

Did some more investigation into the issues and some improvements on how some of the things are handled. The user statistics are now back up!

Optimizations:

• Implement more efficient iteration over keys in Redis database

Bugfixes:

• Fix null reference exception when determining valid content types for metadata
• Fix exception when connecting and a bridge network node is not found

2023/10/11

The following notes were sent by Frooxius:

I've added some new tooling to the cloud for tackling Patreon benefits. We had a number of people who linked their Patreon to wrong account, which caused this account to claim the benefits.

We now have tooling which lets us remove those benefits from the account, so they can be re-claimed on another one in these cases.

If you're affected by this and you'd like your benefits to move to another account, please make a ticket at https://support.resonite.com/

New features:

• Implemented admin commands for listing patreon fund events and removing them from their account
  • This allows the fund events and associated benefits to be re-claimed on another account, which can help when the user's benefits were originally claimed on the wrong account
  • Users cannot remove the benefits themselves, these commands are only for the staff. If your benefits got claimed on the wrong account, make a support ticket at support.resonite.com and we'll sort this out for you

2023/10/14

The following notes were sent by Frooxius:

Sorting out through some more account issues that people are having.

Bugfixes:

• Fix email system not sending attachments
  • This fixes the record usage .json report not being included in the email (based on support ticket by @jatsy)
• Fix large record usage .jsons not sending by compressing them into a ZIP file
• Fixed emails not being sent in some cases, due to service unavailable (503) error from the email provider
  • The fallback service will now be used

2023/10/17

The following notes were sent by Frooxius:

Fixing up some issues with storage calculations and assigning storage to groups.

New features:

• Added internal command for assisting with account/group storage issues

Bugfixes:

• Improve logic for calculating which storage sources of the same group take priority
  • This fixes issue where users updating their tiers would get wrong version of the storage (based on support ticket by @wolf_seisenbacher)

2023/10/18

The following notes were sent by Frooxius:

Adding some more self-serve functions to the cloud! We get a number of requests asking to change usernames in tickets, so this should help clear up those up and scale better.

New features:

• Added ability to change your own username
  • You can do so by sending /changeUsername <old username> <new username> command to the Resonite bot in the contacts list
  • You will need to wait 7 days before you can change your username again. This delay is subject to change. We generally recommend you do not change your username often.

2023/10/22

The following notes were sent by Frooxius:

Just a small update, adding some extra diagnostics to the cloud.

Tweaks:

• Added additional diagnostics to the cloud worker to help with diagnosing the recent downtime

2023/12/12

The following notes were sent by Frooxius:

We've received a number of reports of users having issues giving their groups storage space. There has been one issue that's been likely culprit - the updates of the storage were getting backed up and would take longer to actually update, which this update should resolve.

Another additional issue is that some groups have members with 0 quota assigned to them. This will prevent them from syncing things to the group, even when the group itself has storage. I've added new command that lets you update the quota. You can use /getGroupInfo <group> command to check how much each member has.

Hope this helps!

New features:

• Add /setMemberQuota <group> <user> <quota in GB> command, which allows changing the assigned personal quota for given group member
  • Use negative number (-1) to do not limit the member's group quota (this means the they are limited by the quota of the whole group)
• Make /getGroupSpace <group> command work for members of the group, instead of just platform admins
  • You can use this to check how much storage is being used by given group

Tweaks:

• Move storage update processing to user queue processor, to prevent asset cleanups and processing from causing the storages to get backed up (based on a number of support tickets and internal report by @dantetucker)

2024/01/13

The following notes were sent by Frooxius:

Pushing an important cloud update, which should fix users being locked out of their accounts due to changing their password to one that does not conform to the password rules (which was mistakenly not validated on password change).

No need to update, this is purely cloud change.

Bugfixes:

• Fixed password change not validating password properly, allowing potentially very insecure password to be set when resetting password on an account
  • Also fixed not being able to login with changed password that doesn't conform to the password rules, resulting in some users being locked out of their accounts after changing the password (reported by @GhoulHound, @kurotori and @Dante, based on a number of support tickets and issue #1039)
• Fixed server error when trying to login with session token that has been already expired

2024/03/01

The following notes were sent by Frooxius:

Just a small cloud update to help with some support/moderation tickets.

Security:

• Added internal tools to help diagnose cases of false ban evasion detections and assist in unblocking those accounts (requested by @Dante)

2024/07/12

The following notes were sent by Frooxius:

This is cloud update to address a security vulnerability in a library. You do not need to do anything.

Security:

• Updated System.Text.Json to 8.0.4 for cloud services due to a high security vulnerability in older version (reported by @Stiefel, security ticket #471041)
  • CVE details: https://github.com/advisories/GHSA-hh2w-p6rv-4g7w
  • We're investigating if client version of library needs to be updated as well. The CVE states this affects .NET 8.0 applications (which client doesn't use right now, but the new headless does) and affects untrusted input (which the client should only interact with our trusted cloud right now), so we might not update it at the point. This is due to compatibility issues between .NET Framework with non-major versions of the library

2024/07/18

The following notes were sent by Frooxius:

This should help resolve some of the issues that appeared with the cloud yesterday and today, things seem to be a lot more stable now.

Fixes:

• Removed mechanism that clears user status on disconnect (based on report by @Skywind Kitsune, @Exo / Cybrix, @bredo, issue #2607)
  • This has been resulting in high usage on Redis, causing parts of the cloud service to be slow to respond
  • This is not a critical mechanism - it only makes users appear offline a few minutes sooner if their SignalR connection gets interrupted or if they crash - but in its absence, their status will auto-expire within 5 minutes

2024/07/20

The following notes were sent by Frooxius:

Ok I found out what the issue was that was causing contacts to nore receive updates in some cases.

I made an oopsie in the new contact initialization method and instead on awaiting for all the contact update registration to be completed... it awaits on a completed task instead, meaning that users don't subscribe to all their contacts for updates.

Things should be all good now, I think with this everything should be resolved now.

I'll keep monitoring things for a bit, but I think we're good.

You don't need to do anything, I've restarted the cloud.

Fixes:

• Fixed contact initialization not waiting to be fully finished, resulting in users not receiving status updates for all their contacts

2024/07/21

The following notes were sent by Frooxius:

Ok one more thing! Bridges & Relays should now be registered back up.

Bugfixes:

• Fixed SkyFrost relays & bridges not being registered with the cloud, breaking LNL connections (reported by @bredo, @T.O.A.S.T.E.R., issue #2640)

2024/09/14

The following notes were sent by ProbablePrime:

USWest is now running the .NET 8 Relay and Bridge.

Its also exporting Prometheus formatted metrics which are a little hard to read but still cool. We'll be able to tie this into metrics.yellowdogman.com soon.

No change log was provided for this update. The following is unofficial.

Tweaks:

• US West relay and bridge switched to the .NET 8 rewrite of the bridge.

2024/09/14

The following notes were sent by ProbablePrime:

ALL LNL Bridges and Relays are now running .NET 8 Goodness.

They're also all reporting metrics over on: https://metrics.yellowdogman.com/d/cd82198a-7768-4371-9b85-88af1c19de90/network-nodes?orgId=1

I just spent 30 mins editing the raw json of the dashboard so that they all have the same colors across all graphs too!

It'll take awhile for EU to "make sense" but you can see it starting to appear on the graphs:

No change log was provided for this update. The following is unofficial.

Features:

• Added metrics for the relays and bridges
  • Metrics can be accessed via https://metrics.yellowdogman.com/d/cd82198a-7768-4371-9b85-88af1c19de90/network-nodes?orgId=1

Tweaks:

• All bridges and relays are now switched to the .NET 8 rewrite of the bridge.

2024/09/27

The following notes were sent by Frooxius:

I pushed update to the cloud, switching our email provider back to Azure. They seem to have fixed the issue causing emails to not be delivered and they are generally more reliable than SendGrid, so this should help.

Tweaks:

• Switch primary email provider back to Azure Communications Service from SendGrid
  • This should fix email delivery issues (reported by @Dante, issue #2987)

2024/10/22

Update 1

The following notes were sent by Frooxius:

Pushed a fix to the cloud which fixes the deleted record recovery not working in some cases. Doesn't require a client update and it's pretty high importance.

If you are affected by this, just re-run the command and it should work now.

Also note I'm still waiting for some people to test latest prerelease , particularly with Vive controllers. See ⁠🎥devlog for more info! Once this is tested, I'll push out a new build with fixes and improvements.

Fixes:

• Fixed deleted record recovery failing to recover records in some cases (reported by @Delta, ghost and @989onan, issue #1357)
  • This affected records that were moved to the trash bucket server-side
  • This will fix recovering items that failed to recover, so if you ran into this and the items were deleted less than 30 days ago, just re-run the command

Update 2

The following notes were sent by Frooxius:

Another cloud update, some of the changes broke the cloud registration process, this fixes it.

Fixes:

• Fix regression in cloud breaking new user registration (reported by @Jae “awa” J4, @⭐🌟 the a͙x͙o͙l͙o͙t͙l͙ 🌟⭐, @꧁༺ 𝕋𝕠𝕜𝕚_𝕡𝕠𝕦𝕔𝕖 ༻꧂, issue #3110)