Desync when changing owner

[M3rgi1]

Describe the bug
2 Spheres with NetworkRigidbody with Sync Mode Client to Server and Spawned with Autority to Player 1.
Player 1 applys force to one of the spheres and waits until all spheres are still.(They can collide among themselves).
Then on the Server the Authority of all Spheres is removed and assigend to the 2 Player. While doing this sometimes(not always aproximatly every 3-5 time) the positions of the sphere got out of sync. And on the Client that not owns the sphere the spheres are at exactly twice the position they should be.

[IMPORTANT] How can we reproduce the issue, step by step:
Create a base Multiplayer Project with Network Manager. Create a sphere with network Idendity, NetworkRigidbody, Rigidbody, sphere collider. And make it a Spawnable Prefab. Have 2 Clients with a Network PlayerController that can apply force to one of the Spheres if its his turn. Spawn the Spheres with authority of Client 1. Than Client one should ad force to his sphere. Then Remove The authority from both Spheres and assign Client to to have Authority about the Spheres. That leads to the spheres be out of sync.
In my use case its a turn based minigolf game where i want to calculate each turn on the actieve Player.

Expected behavior
Expected would be to change the owner of the spheres without the position get out of sync and beiing double the original on one of the clients

Screenshots

Desktop (please complete the following information):

• OS: Windows
• Build target: standalone
• Unity version: 6000.0.30f1
• Mirror Version: 89.8.0

Additional context