Storage write ends with exception if image is compressed with xz or gz

[gylstorffq]

If i write a image to a usb drive with mtda-0.17-1 on a raspi4 with mtda-cli storage write <image> the process ends with the following exception:

  File "/usr/bin/mtda-cli", line 825, in <module>
    app.main()
  File "/usr/bin/mtda-cli", line 814, in main
    status = cmds[cmd](stuff)
  File "/usr/bin/mtda-cli", line 473, in storage_cmd
    return cmds[cmd](args)
  File "/usr/bin/mtda-cli", line 563, in storage_write
    status = self.agent.storage_write_image(
  File "/usr/lib/python3/dist-packages/mtda/client.py", line 269, in storage_write_image
    return self._storage_write(image, imgname, imgsize, callback)
  File "/usr/lib/python3/dist-packages/mtda/client.py", line 210, in _storage_write
    status = self.storage_close()
  File "/usr/lib/python3/dist-packages/mtda/client.py", line 148, in storage_close
    return self._impl.storage_close(self._session)
  File "/usr/lib/python3/dist-packages/zerorpc/core.py", line 278, in <lambda>
    return lambda *args, **kargs: self(method, *args, **kargs)
  File "/usr/lib/python3/dist-packages/zerorpc/core.py", line 270, in __call__
    return self._process_response(request_event, bufchan, timeout)
  File "/usr/lib/python3/dist-packages/zerorpc/core.py", line 227, in _process_response
    reply_event = bufchan.recv(timeout=timeout)
  File "/usr/lib/python3/dist-packages/zerorpc/channel.py", line 255, in recv
    event = self._input_queue.get(timeout=timeout)
  File "src/gevent/queue.py", line 334, in gevent._gevent_cqueue.Queue.get
  File "src/gevent/queue.py", line 349, in gevent._gevent_cqueue.Queue.get
  File "src/gevent/queue.py", line 326, in gevent._gevent_cqueue.Queue.__get_or_peek
    SimpleQueue = _PySimpleQueue
  File "src/gevent/_waiter.py", line 151, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
zerorpc.exceptions.LostRemote: Lost remote after 40s heartbeat

[chombourger]

This suggests that the mtda daemon on the RPI4 did not respond to the ZeroRPC heartbeat.
Can we increase the debug level to 3 on the RPI side and check what the logs say (if anything)?

[chombourger]

It may also be good to let us know:

• the compression format of your image
• the size of your image
• what would mtda be writing to on the RPI side? (did you configure the storage "driver" to write to a stick connected to the RPi which would then be exposed to the Device Under Test using the USB mass storage gadget?

it may also be interesting to check the kernel logs on the RPI

[gylstorffq]

I tested it with no compression and xz in both cases the error occurs. The image is uncompressed ~8GB and compressed ~300MB. MTDA writes directly to a USB drive. Here is the config for storage:

[storage]
variant = usbf
file = /dev/sda

The image is written successfully to the disk.
As I want to use mtda together with LAVA this will fail the LAVA pipeline.

Also we tested it on the USB3 and USB2 slot and is in both cases the same behavior.

[gylstorffq]

mtda_deamon_2023-01-09.log.gz
Here is a log from the daemon.

[chombourger]

Thank you so much for the additional details and for the logs.
I took only a quick glance and did not spot anything abnormal.
Will probably have to reproduce your setup (my primary device is the NanoPI for developing/testing MTDA, I however have a rpi4 that I use for some other projects but can use for spot tests)

[chombourger]

Hmm looks like it worked ok for me on the RPi using a rather large image:

$ mtda-cli -r 192.168.1.69 storage write service-stick-image.wic.zst 
service-stick-image.wic.zst: [####################] 100% (13.12 GiB read, 14.01 GiB written, 8.96 MiB/s)

I also checked that further interactions with the rpi4 using mtda-cli

[chombourger]

For completeness, here's the config file used on my end:

[main]
debug=2

[power]
variant = shellcmd
on-cmd = /bin/true
off-cmd = /bin/true
check-on = /bin/false

[storage]
variant = usbf
file = /dev/sda

[console]
variant  = serial
port     = /dev/ttyUSB0
rate     = 115200

obviously the power commands are dummy ones (not really using this rpi4 with MTDA or with a DUT attached)

[gylstorffq]

@chombourger I tested it with zst compression - this works. So it only leads to the error with gz,xz and raw images.