Performance test | Mirai seriously outperforms Suki

[REALMWTH]

Miral timings: https://timin.gs/?id=283d611ae22d48c79ecbb6b23e1d243e

Suki timings: https://timin.gs/?id=88ed0483bf094ffe9bb96164d484775b

[REALMWTH]

For testing I've used updated bot stresser: https://github.com/PureGero/minecraft-stress-test

World size is 2000x2000, fully preloaded with Chunky

Server CPU is Core I7 9700k

[MartijnMuijsers]

Thank you for testing!

Could you upload the suki-global.yml and suki-world-defaults.yml? I have not added them to Timings yet.

Edit: and also the values of the Suki environment variables if you used them

[REALMWTH]

global

# This is the global configuration file for Suki.
# As you can see, there's a lot to configure. Some options may impact gameplay, so use
# with caution, and make sure you know what each option does before configuring.
# 
# If you need help with the configuration or have any questions related to Suki,
# join us in our Discord for SuCraft, or check the GitHub Wiki pages.
# 
# The world configuration options are inside
# their respective world folder. The files are named suki-world.yml
# 
# Wiki: https://github.com/SuCraft/Suki/wiki
# Discord: https://discord.com/invite/pbsPkpUjG4

_version: 17
console-logs:
  change-invalid-pool-element-level-to-info: true
  chat-messages:
    empty: false
    not-secure: false
  invalid-client-view-distance: true
  invalid-statistics: false
  login-protocol:
    packets:
      handshake: false
      login-start: false
      set-compression: false
  move-checks:
    too-quickly:
      self: true
      vehicle: true
    wrongly:
      self: true
      vehicle: true
enchantability:
  bane-of-arthropods:
    hoe: false
    pickaxe: false
    shovel: false
  efficiency:
    sword: false
  fire-aspect:
    axe: false
    hoe: false
    pickaxe: false
    shovel: false
  fortune:
    sword: false
  knockback:
    axe: false
    hoe: false
    pickaxe: false
    shovel: false
  looting:
    axe: false
    bow: false
    crossbow: false
    hoe: false
    pickaxe: false
    shovel: false
    trident: false
  protection:
    elytra: false
  sharpness:
    hoe: false
    pickaxe: false
    shovel: false
  silk-touch:
    sword: false
  smite:
    hoe: false
    pickaxe: false
    shovel: false
  thorns:
    elytra: false
hide-game-modes:
  replacement: survival
keepalive:
  kick-player-for:
    late-response: false
    unexpected-response: false
    wrong-response-id: false
  send-interval: 3000
login-protocol:
  slow-login-timeout: 600
network-constrained-chunk-sending:
  assume-packets-in-transit-arrived-after-time: 600
  ignore-network-conditions-for-chunks-within-chessboard-distance: 1
plugins:
  hide-if-no-permission: true
  show-legacy-asterisk: false
signed-chat:
  enabled: false
  if-disabled:
    hide-insecure-popup: true
    use-chat-preview: false
survival-debug-stick:
  switch-property-timeout: 300
tps-catchup:
  max: 50
view-distance:
  allow-invalid-client-view-distance: false
  see:
    max-total-chunks: 100000
  track:
    max-total-chunks: 100000

[REALMWTH]

world-defaults

# This is the world defaults configuration file for Suki.
# As you can see, there's a lot to configure. Some options may impact gameplay, so use
# with caution, and make sure you know what each option does before configuring.
# 
# If you need help with the configuration or have any questions related to Suki,
# join us in our Discord for SuCraft, or check the GitHub Wiki pages.
# 
# Configuration options here apply to all worlds, unless you specify overrides inside
# the world-specific config file inside each world folder.
# 
# Wiki: https://github.com/SuCraft/Suki/wiki
# Discord: https://discord.com/invite/pbsPkpUjG4

_version: 19
bone-meal-regrows-bark:
  bone-meal-is-used-up-chance: 1.0
  enabled: false
break-blocks-in-survival:
  barrier-if-holding-barrier: false
  light-if-holding-light: false
difficulty-changes:
  cave-spiders-have-poison-on-easy-difficulty: false
  lightning-spawns-fire: true
  minimum-health-after-starving: -1.0
  players-share-villager-gossips:
    positive:
      always: false
  raids-are-always-hard: false
  spiders-with-effects-on-all-difficulties: false
  villager-reputation-changes:
    cured:
      major-positive: 20
      minor-positive: 25
    trade:
      trading: 2
    villager-hurt:
      major-negative: 0
      minor-negative: 25
    villager-killed:
      major-negative: 25
      minor-negative: 0
  wither-effect-from-wither-skulls-on-easy-difficulty: false
  zombie-reinforcements-on-all-difficulties: false
entity-ticking:
  immune-from-inactive:
    non-sitting-pets: true
    sheared-sheep: true
    ticks:
      baby: 5
      mob-targeting-mob:
        axolotl-targeting-fish: 0
        axolotl-targeting-squid: 0
        cat-targeting-rabbit: 0
        default: 0
        enderman-targeting-endermite: 0
        fox-targeting-chicken: 0
        guardian-targeting-squid: 0
        llama-targeting-wolf: 0
        mob-targeting-turtle: 0
        ocelot-targeting-chicken: 0
        piglin-targeting-hoglin: 0
        piglin-targeting-wither-skeleton: 0
        polar-bear-targeting-fox: 0
        snow-golem-targeting-non-player: 0
        witch-targeting-witch: 0
        wither-skeleton-targeting-iron-golem: 0
        wither-skeleton-targeting-piglin: 0
        wither-skeleton-targeting-snow-golem: 0
        wolf-targeting-fox: 0
        wolf-targeting-sheep: 0
        zoglin-targeting-non-player: 0
  skipped-active-entity-ratio:
    ratio: 0.75
  wake-up-duration-standard-deviation-ratio: 0.2
interaction-distance:
  max: 6.0
item-pickup:
  hopper:
    minecart-immunity:
      detection:
        horizontal-distance: 32.0
        max-age: 1200
        tick-rate: 20
        vertical-distance: 4.0
        while-inactive: true
      duration: 200
    tick-rate: 1
  villager:
    delay-after-drop: 100
lootables:
  only-refresh-for-players: true
mob-spawning:
  all-nether-bricks-count-as-fortress: false
move-checks:
  too-quickly:
    self: true
    vehicle: true
  wrongly:
    self: true
    vehicle: true
network-constrained-chunk-sending:
  max-chunk-packets-in-transit:
    target-player-chunk-send-rate-margin-factor: 1.15
never-despawn-entities-due-to-distance:
  wither: false
prevent-block-update-client-lag:
  enabled: false
relocate-corrupted-chunks:
  enabled: true
view-distance:
  if-can-see-but-not-tracking:
    send-chunk-refresh: false
    send-end-portal-fill-sound: true
    send-ender-dragon-spawn-sound: true
    send-wither-spawn-sound: true
  load:
    limit-by-client-view-distance: true
    limit-by-see-view-distance: true
    max: -1
    per-player: true
  see:
    limit-by-client-view-distance: true
    max: -1
    per-player: true
  track:
    limit-by-client-view-distance: true
    max: 10
    per-player: true

[REALMWTH]

Forgot to mention, that all bots and me are using view-distance 10

updated 26 minutes Mirai timings: https://timin.gs/?id=9feac017e14f4df19ed1e3be623f9393

I guess this is something related to Paper chunk rewrite and not related to Suki at all

[MartijnMuijsers]

The client bots are using view distance 10?

[MartijnMuijsers]

(Suki does not have the Paper chunk rewrite yet, but the latest version of Mirai does)

[MartijnMuijsers]

Small note: your environment variables include
-Dsuki.systemcpus.asyncexecutor=8 -Dsuki.systemcpus.tracker=8 -Dsuki.systemcpus.upgradeworld=3 -Dsuki.systemcpus.levelexecutor=-1
but it should be
-Dsuki.threads.asyncexecutor=8 -Dsuki.threads.tracker=8 -Dsuki.threads.upgradeworld=3 -Dsuki.threads.levelexecutor=-1

(The values are correct 🙂)

[MartijnMuijsers]

How much RAM do you have?

[REALMWTH]

Fixed, gonna have another stress test run

I have 32gb of RAM

[MartijnMuijsers]

And what is your simulation-distance?

[REALMWTH]

simulation-distance: 4

[MartijnMuijsers]

Your settings

  see:
    max-total-chunks: 100000
  track:
    max-total-chunks: 100000

see should be based on RAM. I think for your RAM, 50000 would be better, but maybe 100000 is okay, hard to say.
track should be based on CPU. I think for your CPU, somewhere around 25000 would be better.
(I have track on 2500 for 4x VPS core @ 2.0 GHz, and see on 12500 for 8GB RAM)

[REALMWTH]

https://timin.gs/?id=0c2755ecc26045e8aa230795d2c9489a

Will apply changes to configuration and have another stress test

As soon as bots are spreading and loading lots of chunks, TPS goes down a lot. I'll look at paper's chunk rewrite so I can understand how it works better. Really thought that async-chunks config entry was introduced with the patch.

[MartijnMuijsers]

All your other settings should be okay! I use item-pickup.hopper.tick-rate: 20 but 1 is also okay.

[MartijnMuijsers]

Also you have max-concurrent-sends: 128 which is just really weird, where did you get that? Should not be higher than the number of threads you have. I recommend 6 or so.

[MartijnMuijsers]

I'm not sure why Server Oversleep is so high for you. For me it's like 7% max (with full server tick @ 150%).

[REALMWTH]

Also you have max-concurrent-sends: 128 which is just really weird, where did you get that? Should not be higher than the number of threads you have. I recommend 6 or so.

That's just me who made some weird changes, fixed. Testing it now.

Update: it seems that lowering this value makes chunks send very slow at high players numbers.
I've reconnected to server and loading time of one chunk below me took around 10 seconds

[REALMWTH]

[REALMWTH]

That helped TPS a lot at the cost of very slow chunks loading
https://timin.gs/?id=000ab2ba3c414655b833cc97df778646

[MartijnMuijsers]

Okay, that setting is usually not supposed to do that, originally. Maybe something changed a while back. I know servers that run fine and have the setting on 1. Strange.

Thank you! This will hopefully allow to find the problem.

[REALMWTH]

I hope so, glad to help. Rising the value to 32 helped getting chunks way faster at the cost of TPS
https://timin.gs/?id=fc3ea153fbb944eb82c0e688e42615d1

[MartijnMuijsers]

Thank you for the information. I'm going to hunt down Minecraft::Server Oversleep (SELF) and see what it could be!

[MartijnMuijsers]

Hi, this is not a fix, but purely a test. You have a much better testing environment than I do, so if that's okay, could you run
another test with Suki using the latest build https://github.com/SuCraft/Suki/actions/runs/3276027161 - with the following in suki-global.yml:

tps-catchup:
  max: 50

replaced by:

tps:
  max-oversleep: 0

and your original setting of max-concurrent-sends: 128 (since it works fine in Mirai)?

Once again, this is not a fix, but I am very interesting to see what the TPS is like, and also how the chunk loading performance feels.

[MartijnMuijsers]

Also just as a small note I am assuming you are running the bots on a different machine than the server. If they are the same machine, the environment variables need to be set lower than 8 of course.

[REALMWTH]

Also just as a small note I am assuming you are running the bots on a different machine than the server. If they are the same machine, the environment variables need to be set lower than 8 of course.

Lowered number of threads to 6 and made changes requested below, testing now

Update: https://timin.gs/?id=0f064a5024d14e0faca20ce65cba1b81

It seems that chunk rewrite patch makes all difference. Yesterday I've tested latest build of Purpur and it seems to run slightly better even than Miral

[MartijnMuijsers]

Thank you very much for testing! That may be true, but this issue still should not be happening, and I wish to find out what is taking time. I have also set up the Minecraft stress server but I only have a laptop of which a lot of capacity is already taken up by IDE and related applications, so it's hard for me to test with that many players and world size. I'm currently rewriting some code so it's easier to inspect what exactly the CPU use is for.

Could you try it with the max-oversleep: 0 setting suggested above ^? I saw in the Timings it was still set at 50.

[REALMWTH]

Thank you very much for testing! That may be true, but this issue still should not be happening, and I wish to find out what is taking time. I have also set up the Minecraft stress server but I only have a laptop of which a lot of capacity is already taken up by IDE and related applications, so it's hard for me to test with that many players and world size. I'm currently rewriting some code so it's easier to inspect what exactly the CPU use is for.

Could you try it with the max-oversleep: 0 setting suggested above ^? I saw in the Timings it was still set at 50.

Sorry, my fault. I've changed max-oversleep in wrong config line

tps-catchup:
  max-oversleep: 0

When the correct one is tps
Will provide test results in ~10 minutes

[MartijnMuijsers]

Yes this is partially my fault for suddenly renaming the configuration option 😅

[REALMWTH]

https://timin.gs/?id=b3cacbfec57f4e05bc60726d1f23adb7

The server feels much more stable, TPS is much higher, however, chunk loading is slow
As soon as I disconnect bots, chunks loading fast

[MartijnMuijsers]

Ah yes, that was mostly what was expected. It was definitely not a fix, but a good test. The timings will help me a lot. Thank you!

[REALMWTH]

Great! Glad to help you ;)

[REALMWTH]

After teleportation (and sometimes, randomly) server completely freezes for a few seconds
I've managed to grab timings report: https://timin.gs/?id=c4a6c51f481a465888a3e8a38c2b5cd3

P.S. I've tweaked some configs

[MartijnMuijsers]

Thank you so much! This is very interesting. The chunk sending and view distance patches have been specifically optimized in several ways to make teleporting blazingly fast. I've never seen this happen on my own server, and I get random unprovoked compliments from players often about how suspiciously fast teleporting is. Clearly I have overlooked some issues, I will look into it carefully!

[REALMWTH]

Thank you so much! This is very interesting. The chunk sending and view distance patches have been specifically optimized in several ways to make teleporting blazingly fast. I've never seen this happen on my own server, and I get random unprovoked compliments from players often about how suspiciously fast teleporting is. Clearly I have overlooked some issues, I will look into it carefully!

Issue can be tied to misconfiguration for players amount. I've tested with 100 and 200 bots, server is freezing in both tests on teleport.

[MartijnMuijsers]

You have max-total-chunks: -1 for both track and see, and max in world settings also to -1, this means their track distance will always be equal to their client view distance, regardless of your server.properties view-distance. But yet I am not understanding the result from just that. One note to make: you use -Dsuki.systemcpus.forexecutors=8, which is main thread + async chunk threads. It will not be used however, since async-chunks.threads set to 8 specifically in paper.yml.

[REALMWTH]

You have max-total-chunks: -1 for both track and see, and max in world settings also to -1, this means their track distance will always be equal to their client view distance, regardless of your server.properties view-distance. But yet I am not understanding the result from just that. One note to make: you use -Dsuki.systemcpus.forexecutors=8, which is main thread + async chunk threads. It will not be used however, since async-chunks.threads set to 8 specifically in paper.yml.

Adjusted configs and made a new test: https://timin.gs/?id=e88c0f9105404c42a91f4415a5045627

[MartijnMuijsers]

Hey! You set track.max-total-chunks: 4 in the global configuration, this means the server can track at most 4 chunks in total across all players. Of course, this doesn't actually happen because there are some other minimums, but this is not the intended way to use the setting haha. If you set it to -1, but set track.max to 4 in the world configuration (as you have now) it should work as you probably intended to test.

PS I wrote an extra introductory paragraph in https://github.com/SuCraft/Suki/wiki/Features#view-distance explaining the view distance feature.

[REALMWTH]

Hey! You set track.max-total-chunks: 4 in the global configuration, this means the server can track at most 4 chunks in total across all players. Of course, this doesn't actually happen because there are some other minimums, but this is not the intended way to use the setting haha. If you set it to -1, but set track.max to 4 in the world configuration (as you have now) it should work as you probably intended to test.

PS I wrote an extra introductory paragraph in https://github.com/SuCraft/Suki/wiki/Features#view-distance explaining the view distance feature.

🤦
Thanks a lot for explanation :)

https://timin.gs/?id=bacc9dbc9aa145aabb462755d444ab80

TPS Drop is on teleport
At the same time chunks load extremely slow (not a single chunk loads in minutes)

[MartijnMuijsers]

I'm getting closer to the core of the problem. I modified some routines, made network-constrained chunk sending optional and added more timings.

If you could, would you run the following tests?

• A simple test with mostly Suki recommended settings, using the newest build, and network-constrained-chunk-sending.enabled: false (which is default)
• The same test but with Mirai without the new chunk system (https://github.com/etil2jz/Mirai/actions/runs/3060393927)

[REALMWTH]

I'm getting closer to the core of the problem. I modified some routines, made network-constrained chunk sending optional and added more timings.

If you could, would you run the following tests?

* A simple test with mostly Suki recommended settings, using the newest build, and `network-constrained-chunk-sending.enabled: false` (which is default)

* The same test but with Mirai without the new chunk system (https://github.com/etil2jz/Mirai/actions/runs/3060393927)

Suki test: https://timin.gs/?id=e550e8f9612a495da6eb0ceb26c3df68
Mirai test: https://timin.gs/?id=c7c35d52bd03452e92c17fb4c0f601fe
I've teleported multiple times during test and now I don't see server freeze, but chunks sending is still not very good
Test environment is good enough, as I've copied the whole server folder before tests, so I guess the results can be compared (not 100% accurate for sure). The only change in Mirai server folder is the server software.

Made a new test running latest Miral, the results are interesting: chunks loading way faster, and I can see higher number of entities, but TPS is very bad: https://timin.gs/?id=ff0fe24672284addaf38b4b866ce1688

[MartijnMuijsers]

Thanks, I've used your figures to make a few adaptations to the most general patches. If you'd want, you could try another test against the in-progress version that is up-to-date with upstream: https://github.com/SuCraft/Suki/actions/runs/3300905327

[REALMWTH]

Thanks, I've used your figures to make a few adaptations to the most general patches. If you'd want, you could try another test against the in-progress version that is up-to-date with upstream: https://github.com/SuCraft/Suki/actions/runs/3300905327

https://timin.gs/?id=54b2178b183e4de0a1960a81ec43d27e