Complete test_server_task.

[janiversen]

We need to test:

• async

  • server start

  • client start

  • client stop

  • server stop

  • server start

  • client start

  • server stop

  • server start

  • client reconnect!!

  • client stop

  • server stop

And the same for sync.

This should be done for all transport types:
- tcp
- tls
- udp
- serial
- unix

For async we should also test multiple servers.

We do not support multiple sync servers (although it seems to work).

This should work on all platforms (unix is not supported for windows and should cause an exception).

Versions

• Python:
• OS:
• Pymodbus:
• Modbus Hardware (if used):

Pymodbus Specific

• Server: tcp/rtu/ascii - sync/async
• Client: tcp/rtu/ascii - sync/async

Description

What were you trying, what has happened, what went wrong, and what did you expect?

Code and Logs

# code and logs here.

# please use the following to format logs when posting them here
import pymodbus
pymodbus.pymodbus_apply_logging_config()

[alexrudd2]

• client reconnect!!

For some empirical data...

I have about 30 modbus TCP servers (and correspondingly 30 async pymodbus TCP clients) that are connected 24/7/365. pymodbus 2.5.3 would handle re-connections fine for months/years.
pymodbus 3.x handles 99% of reconnects, but will eventually fail in hours/days. By "fail" I mean remain disconnected permanently.

In fact, this problem is why I'm trying to understand the source code and submit patches to solve it. It's very difficult to diagnose such issues because of the long time frame. Also, there are different types of situations where a reconnect is needed - here's a non-exhaustive list:

• Modbus protocol errors when e.g. part of the frame is dropped (Unable to decode response Modbus Error: Unknown response 65)
• pymodbus.exceptions.ModbusIOException: Modbus Error: [Input/Output] Unable to decode request
• Timeout errors (in the underlying protocol) (Fatal error: protocol.data_received() call failed.)
• Timeout errors (in the client code)
• Socket errors
• pymodbus.exceptions.ConnectionException Modbus Error: [Connection] Connection lost during request
• Probably many others :)

I mean to say these tests are both very difficult, and very welcome. I have high hopes for #1298 improving things. I have been running that code for 4 hours now, and it hasn't failed yet. (It takes anywhere between 1 and 72 hours to fail).

[janiversen]

Thanks for your insight. Regarding the client, there is a major cleanup pending (avoiding creating a new object in create_protocol), and that should hopefully improve stability.

One of the reasons 3.x is at 99% while 2.5.3 was at 100%, is that performance have approved with something like a factor 4 (throughput) and factor 2.5 (cpu load).

Could you please start storing the logs, since you have so many installations we might be able to get a good idea of where the problems are by looking at multiple logs. Store and at least mail to me (if you do not want to make them public).

Any help on enhancing test_server_task.py is highly welcomed, we could setup a branch to work together ?? A smal help would be just do define the tests you would like to see as empty test cases with a comment explaining what should happen.

[janiversen]

Most of your list is something we easily can simulate with the new simulator (a bit enhanced 😬 ).

[janiversen]

Test suite completed. The sync test waits for the reworked client.

[alexrudd2]

Could you please start storing the logs, since you have so many installations we might be able to get a good idea of where the problems are by looking at multiple logs. Store and at least mail to me (if you do not want to make them public).

Any help on enhancing test_server_task.py is highly welcomed, we could setup a branch to work together ?? A smal help would be just do define the tests you would like to see as empty test cases with a comment explaining what should happen.

I have tried many times, and was not able to catch anything useful. It takes hours/days for the problem to appear.

But! I wonder if you have fixed it problem here (the reconnection task is being garbage collected?). It seems we both read the same HackerNews article. :)

https://github.com/pymodbus-dev/pymodbus/pull/1318/files#diff-6365ec826302e6b3d7fbe311cdb1a1c0b75e5b270c77dca73bc13fa62ddfb25cR162