How should we handle synchronization over different factories at once

[hubastard]

Here is an ongoing discussion that I had with @Mushroom on Discord about the game synching in general since he is working on the remote save game for initial state right now. We will keep this discussion here, so more people are aware

CodIsAFishToday at 5:00 PM
Okay, how would you propose to handle the scenario I proposed with the different systems?
Right now the factories kind of run in the background unloaded, but having multiple large factories loaded would mince the host CPU
Consider 4 different players in 4 different systems - in a single player game this would not be an issue, as only 1 system would be loaded at a time
hubastard Today at 5:01 PM
Yeah I thought about that it will be tricky
CodIsAFish Today at 5:01 PM
For 4 players, that's now the host doing 4x the work, as well as keeping the other clients in sync with the goings on in those 4 systems
hubastard Today at 5:02 PM
I wonder if we could elect 1 master player per system.
CodIsAFish Today at 5:02 PM
Ideally, clients get sent a "state" for the system
hubastard Today at 5:02 PM
yep
But yeah, maybe we could have the "Master player" which is the one that will deal with sending state update to the clients. Then once another player move to another system like to the "system A", he become the "system A master" and sends its state changes to the server
CodIsAFish Today at 5:05 PM
Possibly, but that introduces more chance for desync
Hmmn
Maybe we just see how badly it screws with the host player by having a lot of stuff loaded
I suppose it's not rendering it so it just has to do the computwe as opposed to render cycles
hubastard Today at 5:06 PM
In fact, I think simulation of other system might not be too much
But, we shouldn't load meshes
CodIsAFish Today at 5:07 PM
Yes and no, it still has to load entity positions in etc, which it doesn't do for remote systems right now
hubastard Today at 5:09 PM
ok, I wonder what kind of desync would be possible. In my opinion only the drones that moves between systems would possibly need more attention for desyncs
CodIsAFish Today at 5:09 PM
That's it's own system so that's manageable
It's the individual sytems which are the ones that will go wrong
Right now, the game only has the concept of having one "factory" loaded at a time, and it has a lot fo code dependency on that
hubastard Today at 5:11 PM
yeah ok I get you now
We could try a quick hack at loading a save game that as lots of factories and loaded them all at one. But I feel like it will be really hard on the host pc
CodIsAFish Today at 5:13 PM
Yes, I think it would possibly be
I'm digging to see if it's possible to decouple things
hubastard Today at 5:16 PM
But my gut feeling is really that we will probably need to elect a "master factory client" and make extra sure that things that move between factories are sync properly and conflict resolve by the true "host" of the game
CodIsAFish Today at 5:16 PM
Yes, I guess just tell them they need a beefy CPU for late-game
hubastard Today at 5:17 PM
But in truth we only need to elect a "master factory client" if there is a player physically in this factory.
CodIsAFish Today at 5:18 PM
Yes, and they get the same "blind updates" as any other person would if they were in the same system as the host
Just means the host is simulating x2
hubastard Today at 5:18 PM
Yes so even, if you have 64 factories total, if you are only 4 players you don't need to load the 64 factories at the same time
CodIsAFish Today at 5:18 PM
As otherwise entity positions etc aren't tracked I think
Yes but 4 factories in late-game is a lot
hubastard Today at 5:19 PM
Yep, but it would be potentially simulated by 4 different players
The host would have the overhead of fixing any synching problem
between the updates it received from the "master factory clients"
CodIsAFish Today at 5:20 PM
How does a player know if they have desynced?
Even differeny framerates could cause a desync, ie something having enough for a drone to launch and not
Even a 1 frame/tick difference starts cascading failures
They key function we need to be investigation in-depth imo is PlanetFactory.GameTick(long time)
hubastard Today at 5:24 PM
Yes
And see how does the simulation works while a factory is not loaded
CodIsAFish Today at 5:25 PM
Well it has a flag it passe,s so we just need to map that out
hubastard Today at 5:25 PM
Because, an unloaded factory doesn't prevent drone from moving in and out of your planet
CodIsAFish Today at 5:25 PM
But for example positions of things on the cargo belts are not updates
Look at CargoTraffic.GameTick()

[sp00ktober]

Maybe a desync could be spotted by syncing GameMain.gameTick? This would rely on the game being deterministic.

Im not sure how deterministic it really is but wouldnt this mean that the server "only" needs to transmit the update messages from clients?
(and check for desync ofc)

We also could reduce networking by only syncing players on the same planet and partially sync players in the same solar system (except for logistics). If players leave/enter another planet/system they could get the current state and get synced with the others.

One question would then be which of oll those states should get used when saving the game..

[hubastard]

Now I'm wondering, instead of having one master client that sent the game state to everyone. Maybe we should keep the game state on the server itself, but keep the simulation client side.

Let's say we have 4 players.

The server will know which player is where in the galaxy, so when a player loads a factory / planet and the server notice that it's the only player on that planet, it promotes this player as the source of truth for this planet. The promoted "leader" of this planet will be in charges of sending the planet state to the server and the server will keep the list of all the loaded factories states and take care of sending the state updates to clients that are relevant.

Now since we have all the loaded factories loaded at once on the server we should be able to detect and fix desync directly on the server. When the server fix a desync, it should send an update to all clients including the planet leaders.

This being said I think that one of the clients should be the real "master client" but only for some game state update like "game tick", stars system / galaxy states maybe.

Also, the players should not be part of the game state and updated independently and there last position should be saved in a server save file to be able to have the same game state when you restart the server.

If we do this, we should also be able to host the server in the cloud.

[sp00ktober]

I think this might be a good idea, but we would need to patch a lot of methods (every method that updates the game state basically) and also need to send alot of network packets.
Maybe we should not fully rely on the server sending the gamestate because of that, but try to include some of the determinism of the game to minimize the network load?

[Mushroom]

I think the problem with that is if someone lags, for example, that would cause a lot of issues, especially on a high throughput factory.

For example, how would we handle fractionation? That's RNG based, which would differ from client to client.

Ideally, I think the best way to handle it would be to have an authoritative host, let the clients render what they want client side, but the host would handle the actual counts etc.

Yes it would require a lot of patching, but we have a start with the "isActive" flag in the game tick.

One problem this could introduce is bandwidth saturation, but everything will have trade-offs. (This is similar to how Cities in Motion 2 did its multiplayer)

[sp00ktober]

We should implement something to separate lagging players from the rest of the system to not let the lag spread to the others.
And i also think that we either need an authoritative host that determines the output of any RNG or just let the server decide it.

basically bool isActive = GameMain.localPlanet == this.planet in PlanetFactory so it just checks if the factory thats about to be processed is on the localPlanet the player is on.

GameData.factories is a PlanetFactory[] array which gets filled by GameData::GetOrCreateFactory(PlanetData planet) which in turn gets called when a planet is loaded, see attached image.

It gets also filled when a savegame is loaded in GameData::Import(BinaryReader r)

GameData::GetOrCreateFactory(PlanetData planet) also calls PlanetFactory.Init() which then creates a FactorySystem(PlanetData planet) which then gets updated in the game update loop.

So to me it looks like the GameData.factories array gets slowly filled as the player moves around in the universe and when a savegame is loaded.

---

PlanetModelingManager::LoadingPlanetFactoryMain() is also responsible for loading all the assets and vegetation on a planet. It is only working for the localPlanet, calling it for a remote planet will cause much problems..

GameData::ArrivePlanet(PlanetData) requests a call to GameData::OnActivePlanetLoaded(PlanetData) which calls PlanetData::LoadFactory() which enqueues over PlanetModellingManager::RequestLoadPlanetFactory(PlanetData) a call to PlanetModellingManager::LoadingPlanetFactoryCoroutine() which then calls PlanetModelingManager::LoadingPlanetFactoryMain()

---

Just saw the isActive flag is passed to FactorySystem::GameTick(long time, bool isActive) but its not used in there. Does that mean the game updates every factory on every planet at any time?

[sp00ktober]

do you mean the CargoPath::Update() function? as i dont see one in CargoTraffic

[Mushroom]

That's the one, too many things named cargo!

[hubastard]

Yeah I agree with @Mushroom that having the host being authoritative and sending the states would be the best. As of the belts I don't think that we will be able to sync them properly, especially since clients can start to take item directly from the belts. Maybe we should start by making the host send updates of its game state, let the clients simulate the rest in between the update, then we will be able to see what breaks.

[sp00ktober]

isnt there a RNG involved in vein mining aswell? I remember that someone said that..
What i was thinking about was this: what if we init all PRNG with the same seed so they produce deterministic output.

This would maybe result in the belts beeing sort of synced by default and we only need to check a few times (a second?) if they still are truly in sync.

[Mushroom]

That could work as an option, but the desync is likely to come from lag once they have a shared initial state (ie, save file). A potential test could be to run the game on something that can absolutely not hack it whatsoever and runs like 10FPS and see how bad it gets. (VM with 1 core assigned and a software renderer maybe?)

[sp00ktober]

We also need some sort of storage where we store updates that are planet specific.
For example when a player mines something on a remote planet we must tell that everyone that arrives to that planet at a later point in time.
This is also solved when the game is saved and then reloaded as the changes should then be available to everyone that joins the game.

maybe there is a way to store data of remote planets without setting them as localPlanet, i need to look into this more.

[hubastard]

I think the work that @Mushroom is doing will solve this. Since when you join the game later the entire game state will be sent to you.

[sp00ktober]

Yes but consider this:
You are on your home planet an another player is on a moon. You mine some trees, then the other player comes back to the home planet.
He will still see the trees as he was on a different planet when they were mined.

[hubastard]

But shouldn't he receives a packet from the other player saying he had mined the tree X on planet Y. If so the change to the planet Y data for the other player should now be changed.

Does the planet needs to be loaded to be able to change its data ?

[sp00ktober]

Thats what i need to find out now

[hubastard]

@Mushroom and @sp00ktober

Ok, I've been thinking about our approach for the game synching and I'm starting to have some doubts that we will be able to scale this to sync the remote players, especially joining a big game, it will take a lot of time. I got and idea that I want to share and have your thoughts. What I want to do is a on demand system. Instead of sending the entire save file to all the players, we would only send the data that is relevant to each player.

For example, in this implementation the host would periodically broadcast ONLY "globals states" (states that are not related to an actual planet). For example:

1. Game tick for synchronization
2. Drone positions (since they move between planets)
3. Game Statistics
4. Dyson spheres data
5. Researches
6. History Data (Maybe could be only on demand when a player opens the stats panel)

So, by doing this, we essentially synchronize the "big picture" of the game, but not the micro details of each planet / factory.

Then when a player connects to the server the following thing will happen:

1. The server will lookup if we have some last connection info about this player (lastPosition, localPlanet, inventory, etc).
2. Then he will send the player the GameDesc of the game (from this the player will generate a fresh new game / world)
3. Then the server will send ONLY the planetData of the closest planet for this player (usually it will be its previous localPlanet)
4. Since the player will have created the same galaxy and he is receiving state updated for his localPlanet, for him the game will looks "synchronized".

When the player fly to space, the server will track them and when they move closer to another planet, then the server will start sending them the current state of this planet instead. That way once the player lands on the other planet everything will be loaded and will look "normal"

Also, when a player interacts with the game world (spawn a building, cut a tree, etc...) the server will ONLY send this information to other players that are on the same planet. Other players that are not on the planet or even in the star system don't need to know about this micro detail. They will receive the appropriate state only if they physically come close to the planet and since the host will always send the "global state of the game" to everyone, they won't even notice.

After all of this is done, I think that we will have a pretty good coverage of all the synchronization that will need to be done.
The main issue that I can think of right now is synchronizing belts. But we could tackle that once all the rest is working.

What do you think this new technique?

[sp00ktober]

sounds good, i can also have a look at how to load the visual of the model manually if you want

[Flapperkewiet]

The AddEntityData() function is only called from AddEntityDataWithComponent() and CreateMonster() function, while imported data its placed manually.
When the base game loads a savefile/imports data they call GameData.ArrivePlanet() which i assume calculates what things need to be drawn.

[sp00ktober]

ArrivePlanet() basically does this #9 (comment)

[Mushroom]

Just a note that I am looking at the remote request of PlanetModelingManager.LoadingPlanetFactoryMain(), which actually takes care of some of this. Mostly it is done by overriding GameData.GetOrCreateFactory() and PlanetModelingManager.RequestLoadPlanetFactory() - in my opinion breaking the planet laod into smaller sections like this will be more performant.

[hubastard]

Looking forward to it :D