Rexo client freeze

[Dethdeath]

I came across something that made the client freeze on the scorpion test server, but it turns out it can be reproduced on the regular server too.
The whole reason this even started is because I noticed my armor was 50/200 after relogging in rexo on the test server.
I took some fairly random actions after this which ended up freezing my client on map load.
The characters this happens to are essentially broken, because they are unable to load any map afterwards.

I didn't know what actually caused the client to freeze, so I replicated everything I did on the 2 initial characters it happened to.
This included their battle rank, certs, what actions they took after zoning, their inventory layout and what they did when they were back on Sanctuary.
After this I started redoing the whole process but removing certain things from the equation, like having a bank or med app on the character.
Since it first happened on the scorpion test server, I thought the scorpion or its ammo were related in some way, but it turned out it wasn't those either.
After that I reduced the battle rank from 25 to 24 on the next run and this also lead to a freeze.
I tried to replicate it on a BR15 and a BR23 after that and had no luck.

So after hours of testing, I've managed to reduce the variables to a more acceptable amount.
What I know so far is that you probably need to be BR24 or 25, have a large weapon in slot 4 or your inventory, medkits, rexo, zone somewhere, come back to sanc and switch to PJs, back to rexo and zone again.
Some of these things are probably not needed to reproduce it, so there's more testing to be done.

Here's some videos from the main server and test server:

Main server:

FreezeMain.mp4

Test server:

FreezeTest.mp4

My current theory is that it's the way the inventory gets sorted when going from Rexo to PJs. I haven't been able to reproduce it with agile.

[ScrawnyRonnie]

I did some testing and was able to reproduce the issue. I was also able to make it not happen and with clue of BR 24 or higher and the suffering I went through a few weeks ago I thought to try something different. When you do the /setbr command, right after if you do /helmet or go into customize appearance and change to one of the other options (leaving it on helmet doesn't help) the problem doesn't happen.

Curious then, if someone get BR 24 the "normal way", if they happened to do this armor change with the necessary items in their inv (it seems to require a larger weapon in size like HA or Sniper) would it break their character? Is every character at or above that BR a potential "bug causer" if they don't change their cosmetic settings manually at some point?

Also tried on 61007 and had the same results.

[Fate-JH]

In an effort to refine the testing process, you don't need to zone twice or shoot anything in between.

• Kit yourself out properly
• Switch between rexo and standard to cycle items out of your inventory and to the ground
• Switch between standard and rexo to make the inventory high capacity again
• Zone

Anyway, I seem to have fixed this issue locally somehow but I created a potential new one. The default state for the cosmetics menu is both "helmet" and "bare head" radio buttons on at the same time and that makes no sense either as radio buttons or as cosmetics state. Visually, nothing changes and the helmet is still on and the cosmetics object on the converted packet is still empty. Still, I'm concerned and would like to see if I can correct this.

[ScrawnyRonnie]

The client freeze still happens after latest changes to #1040, but only with BR 25; 24 doesn't do it anymore. There are a few "requirements" to make it happen as well:

• Larger size weapon, like an HA one
• Multiple medkits
• Something other than a REK in slot 1 (weapon or med app crashed)
  For example, it won't happen with this:
  

But will with these:

So the standard loadout looks like this:

I also tested using /helmet after the /setbr 25 command, but it now caused the crash:

Very specific BR and inventory requirements to happen...
Don't remember if it was mentioned already, but if this happens you can no longer log into that character.

[Fate-JH]

There's actually a bit more to it than just the set up. I performed the BR24 test, it passed, the I progressed to BR25 for the next test, cycled loadouts then zoned, but that test also passed. I started completely from scratch by setting battle rank to 1 and then committing suicide to reset my inventory and uniform. BR25, grab Heavy Assault weapon, grab medkits, Standard, Reinforced, zone; and, crash. It's not the items themselves; it's not where they are held or stored; it's the process they go through.

I do have a thought about what's going on but it doesn't account for why BR24 worked when I only adjusted cosmetics.

[Fate-JH]

What I planned to change didn't actually work. I did discover that, with access to the database, I can regain control of that character that crashed by deleting from the savedplayer table. Specifically, all I have to do is delete the loadout column for that character. The matter definitely has something to do with the state of the holsters and inventory but it's hard to say what.

[Dethdeath]

So that loadout looks completely normal in the database?
I was thinking that maybe it would be possible to simply not store whatever the loadout is that causes the freeze, but if it looks like normal loadout data, there would be no way to detect it.

[Fate-JH]

Okay, we're at this stage of the nonsense.

"Tool,0,140,_0-0-272-16 /SimpleItem,1,728,/Tool,2,845,_0-0-28-25/Tool,3,429,_0-0-272-35/Tool,4,324,_0-0-540-1/Kit,6,536,/Kit,10,536,"

This encoded inventory loads the character correctly. It involves a Beamer, REK, Suppressor, Lasher, Forceblade, two Medkits. The part for the REK is in bold.

"Tool,0,140,_0-0-272-16/Tool,2,845,_0-0-28-25/Tool,3,429,_0-0-272-35/Tool,4,324,_0-0-540-1/Kit,6,536,/Kit,10,536,"

This one does not load. The REK has been lost. To be honest, this is what a loadout would normally look like after doing the exo-suit swap and entering an offending state. You can even set up the former inventory, log out, log back in correctly, but remove the REK, log out, then try to log back in but fail. It's not even possible to put the former encoding back into the database column and regain the ability to log back in for some reason.

[Dethdeath]

What happens if the BR of the character that is failing to log in is changed (via the DB) to like 23? Or their BR25 cosmetic is changed?

[Fate-JH]

Changing the battle rank from 25 to 24 allows the character to load. Setting it back to BR25 just revists the issue.
As you noted, BR24 isn't an issue and anything beneath that would be visible by the player base as the requirements are a long rifle in the second rexo holster, a weapon-type item in the other hand, and medkits. None of that seems unlikely for a common loadout.
Changing the cosmetics otpions doesn't do anything for the loading process.

[ScrawnyRonnie]

I played around with this for too long... and while I already knew it only happened with certain items in certain slots, there are many variations (more than I bothered to test I'm sure) that don't cause it to happen. The note about BR 24 then going to BR 25 successfully zoning, I did that multiple times and crashed. Here are some examples that will crash or not crash and some are a matter of switching just one item with another.

Put a REK in slot 1 on any of the ones that cause a crash and you won't crash any more... weird, but everything about this is.

[ScrawnyRonnie]

Also, if before zoning and after doing the armor swap back to rexo, you add or remove an item from your inventory you will not crash. I guess that fixes whatever corruption happens to the loadout. Do we need to weigh up the cost benefit of tracking this down vs taking a note and moving on? If someone were to be weird enough to actually do this you could recover their character by deleting that field in the table. Or is this a symptom of another larger problem we've just scratched the surface of with higher battle ranks? Who knows...

[Fate-JH]

I have been thinking about mitigation options in case further analysis accomplishes nothing. Taking into account the examples posted above, the mechanically easiest solution might be to remove the long rifle from the latter holster before zoning and put it back in after the character creation packet has ben dispatched. The latter is simple by my estimates (equip) but I don't know of good places in the code where it would be would be easy to isolate for the former (removal).

[ScrawnyRonnie]

Is the juice worth the squeeze? Yes, an issue has been identified in test, but this has yet to happen live and for what it takes to cause it I can't see anyone ever doing this for real. Want to swap armor types? That's what favorites are for 😆

[Fate-JH]

I'm disconnecting this issue from the BR24 cosmetics fix PR so that I can merge that one at my leisure.

[Fate-JH]

I removed the mutuality of the PR so this should have stayed open.

OH. Is that was the consequences of being double linked like that?

[Dethdeath]

The latter is simple by my estimates (equip) but I don't know of good places in the code where it would be would be easy to isolate for the former (removal).

The only place that makes sense to me for the removal would be right as the zoning process starts.

Since we do have some knowledge about the cause, there are some parameters it can check for and flag the player if 2/3 or 3/3 match. The parameters being something like:
BR25 or higher.
Did this person switch to PJs and then back to rexo (not via a favorite).
The rough inventory layout (large weapon in slot 4, 2 med/stamina kits).

Another solution would be to place an extra (invisible?) item in the empty space of the inventory when a loadout like this is detected at the equipment terminal along with the PJ/rexo swap.
That should also prevent it.
The extra item would be dropped after zoning successfully, or when the player replaces it with something else.

Or we just ignore the issue and hope it never happens to anyone.

[ScrawnyRonnie]

Tested this one again locally after merging #1172 but it still happens at BR 25+. May never actually see this issue happen, but if it does (and a resolution is not found) the easiest way I found to recover that broken character is to delete the row for that avatar_id on the savedplayer table so they can log in and be in default PJs hopefully never to repeat their mistake again.