p2p: grpc: received message larger than max

[overvenus]

Checkpoint can not advance when replicating 72k tables in v6.5.0.

[2022/12/30 17:01:00.000 +08:00] [INFO] [server.go:535] ["peer connection received"] [senderID=379a741e-154d-4863-a15c-da459813c5d1] [senderAdvertiseAddr=172.16.5.37:48112] [addr=172.16.5.37:57070] [epoch=372]
[2022/12/30 17:01:00.006 +08:00] [WARN] [server.go:630] ["peer-to-peer message handler error"] [error="rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4207687 vs. 4194304)"] [errorVerbose="rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4207687 vs. 4194304)\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageServer).receive\n\tgithub.com/pingcap/tiflow/pkg/p2p/server.go:722\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageServer).SendMessage.func1\n\tgithub.com/pingcap/tiflow/pkg/p2p/server.go:629\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594"]
[2022/12/30 17:01:00.006 +08:00] [WARN] [server.go:633] ["error receiving from peer"] [error="context canceled"]
[2022/12/30 17:01:00.025 +08:00] [WARN] [server.go:334] ["sending response to peer failed"] [error="[CDC:ErrPeerMessageInternalSenderClosed]peer-to-peer message server tries to send to a closed stream. Internal only."] [errorVerbose="[CDC:ErrPeerMessageInternalSenderClosed]peer-to-peer message server tries to send to a closed stream. Internal only.\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/p2p.(*streamHandle).Send\n\tgithub.com/pingcap/tiflow/pkg/p2p/server_stream_handle.go:66\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageServer).tick\n\tgithub.com/pingcap/tiflow/pkg/p2p/server.go:329\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageServer).run\n\tgithub.com/pingcap/tiflow/pkg/p2p/server.go:228\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageServer).Run.func2\n\tgithub.com/pingcap/tiflow/pkg/p2p/server.go:206\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594"]
[2022/12/30 17:01:00.025 +08:00] [INFO] [server.go:343] ["Deregistering peer"] [sender=379a741e-154d-4863-a15c-da459813c5d1] [epoch=372] []
[2022/12/30 17:01:00.391 +08:00] [INFO] [server.go:535] ["peer connection received"] [senderID=0dc8a568-2e7c-4f35-b6fa-a5d3991cf1c6] [senderAdvertiseAddr=172.16.5.37:48114] [addr=172.16.5.37:57074] [epoch=406]

[overvenus]

This issue causes changefeed stuck, checkpoint can no longer advance.