Revise SerializationTest to make it more robust with CUDA 10.x and 11.x

[hcho3]

#5808 added support for CUDA 11, and @canonizer reported that the SerializationTest test was succeeding with CUDA 10 but failing for CUDA 11. For now, we worked around it by increasing the data size, but we should make the test more robust.

[trivialfis]

It's a tough problem, like any issue related to floating error. On the one hand we don't want to relax the test too much, the serialization tests catch a lots of bugs during development. On the other hand floating error can cause structural changes in trees so in that case we don't actually have anything to assert. If you have suggestions feel free to share, I would be happy to look into them.

[trivialfis]

Fixed by #7446