tl;dr: server performance has been extremely poor in my Agones instance (NA region) since wiping it a week ago. I think it may be caused by a bug in rotation math and/or physics interactions, and after a bit of testing, I was able to reproduce the issue in a couple instances.
I think one of the most recent server builds introduced a positional or rotational calculation bug (local rotations maybe?) in PCVR regarding both teleports and collision properties, as well as a possibly related occlusion culling and LOD bug visible in the forest. Symptoms include, but are not limited to:
Logging out and logging back in again resulting in being transported to locations multiple meters away from the original source (that were not previously visited), or being teleported back to spawn unexpectedly (presumably due to the previous position being invalid).
Excessively high server tick latency after loading certain overworld chunks, impacting the whole server. A confirmed source is objects out of position interacting with the physics engine, which I was able to reproduce with Prefabulator (see below).
Forest chunks levels of detail layering incorrectly, in incorrect positions and rotations. (See below.)
Certain attachments acting in undefined ways (typically, jettisoning themselves away at high speeds) after a weapon breaks or grab is released through another object, in one instance crashing the entire server.
I’ve played ATT for awhile and this all seems new (even compared to previous Mines and Forest lag).
I was able to verify this in a toy example, through the following steps:
Crafting a one-sided craft piece (Craft Piece Side Flat 1Way prefab), attach it to a medium handle (Handle Medium Standard).
Using Prefabulator, select the craft piece, then move it a small ways (~0.1 meter)
After doing this, the server lagged excessively in a similar way to the latency we were experiencing until I deleted the craft piece. (Which is to say, we narrowed it down to physics being the apparent culprit.)
I only tried this after noticing other, related physics anomalies that appeared on the server naturally after a save wipe, which I’m pretty sure aren’t related to my own tinkering:
Teleports home and logout-login cycles having undefined effects. In particular, players being teleported home unexpectedly or into unexpected locations when returning online.
Certain attachments jettisoning themselves in unexpected directions at incredibly high speeds, especially when a handle is broken.
Certain collidable objects rendering correctly, but having unpredictable physics that incredibly lag the server.
LOD layers and occlusion in the forest acting in unpredictable, spastic, and flickering ways, with rotations and orientations of objects at lower levels of detail being visibly incorrect.
If I had to guess (and if these bugs are related at all), I think a recent patch may have introduced a rotation bug somehow, impacting each object’s collision mesh or position. It doesn’t seem to affect prefab part locations or rendering outside of occasional ghosting (those seem fine, as tested with Prefabulator), and it doesn’t seem to be causing prefab placement issues (like earlier bugs in the Mines), but does seem to be impacting physics interactions and player teleportation.
Unfortunately, the result is server performance tanking to unusable levels even after I force a restart (eg, by logging everyone off and letting the instance spin back up), which is then exacerbated by the login bug and the collidable object bugs described above.
Server: Goldkin’s World (my personal private server)
Time: Persistent over the past week from the timestamp of this post
Discord Username: Goldkin