Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nrf52 targets stall reading from pseduo serial device sometimes #838

Closed
geeksville opened this issue Aug 13, 2021 · 8 comments
Closed

nrf52 targets stall reading from pseduo serial device sometimes #838

geeksville opened this issue Aug 13, 2021 · 8 comments
Assignees
@geeksville
Labels
bug Something isn't working

Comments

@geeksville
Copy link
Member

while merging #837 I noticed the following:

Sometimes nrf52 targets will stall reading from the pseduo serial port. This is only apparent if you have the port open via a terminal program or "meshtastic --noproto".

Steps to repro:

  1. Run "screen /dev/ttyACM0 921600"
  2. See a bunch of debug output
  3. After about an hour (on average), you'll see the status LED on the tbeam stop blinking.
  4. Blinking stops until you send the device a character on the serial port. Which implies the polling on the pseduo device is blocking when it should not.
@geeksville geeksville added the bug Something isn't working label Aug 13, 2021
@geeksville geeksville self-assigned this Aug 13, 2021
@geeksville
Copy link
Member Author

hmm the every 20 second battery messages are not getting printed promptly. they seem to sit in a queue until a char is sent from the pc

geeksville added a commit to geeksville/Meshtastic-esp32 that referenced this issue Aug 13, 2021
@geeksville
cope with race on available() vs read() found while looking at meshta…
a9f8080 
…stic#838
@geeksville
Copy link
Member Author

geeksville commented Aug 13, 2021 •
edited
Loading

the problem is occurring below the level of Stream.flush(), it is in the nrf52 usb code? cdc_device.c?

@HookdomPonix
Copy link
Contributor

Tiny USB 1.4.2 loaded in your enviro? Or 1.4.0?

1.4.0 had a race condition on the NRF52.

@geeksville
Copy link
Member Author

1.4.3 alas - I think they still have a problem. I'll work on a fix soon but my back is bugging me so it will probably be another couple of days...

@geeksville
Copy link
Member Author

This issue has been mentioned on Meshtastic. There might be relevant details there:

https://meshtastic.discourse.group/t/geeksvilles-current-task-list-now-working-on-nrf52-fixes/514/66

@geeksville
Copy link
Member Author

OH! I think I just found the bug. Testing now - looks good! USB client code on linux hangs because it seems to expect to always receive a response to reads from the device (even if no characters are pending). Without that response the cdc driver on linux hangs eventually (until a new character is sent from linux to the device)


    if ( 0 == tud_cdc_n_write_flush(itf) )
    {
      // If there is no data left, a ZLP should be sent if
      // xferred_bytes is multiple of EP Packet size and not zero
      if ( !tu_fifo_count(&p_cdc->tx_ff) && xferred_bytes /* && (0 == (xferred_bytes & (BULK_PACKET_SIZE-1))) */ )
      {
        if ( usbd_edpt_claim(rhport, p_cdc->ep_in) )
        {
          usbd_edpt_xfer(rhport, p_cdc->ep_in, NULL, 0);
        }
      }
    }

geeksville added a commit to geeksville/Adafruit_TinyUSB_Arduino that referenced this issue Aug 15, 2021
@geeksville
rough fix for linux serial clients hanging while talking to nrf52 cdc…
a282d74 
… serial

Problem statement:
Sometimes characters written from the nrf52 pseudo serial device to a linux
host would stop showing up on the linux app side.  This occurred randomly
but typically within a few minutes after starting a session.

The reading from linux
would remain 'hung' until you type a character on the linux side (thus
doing a new write TOWARDS the nrf52).

Theory: it seemed like something was getting hung on the linux side waiting
on a response to a read.  This led me to this code.  If I changed the code
to always send a response to any read from ep_in (rather than opting to not
send a response at all in some circumstances), "screen" apps stopped hanging.

Notes:
* problem reproducable with 'screen /dev/ttyACM0 921600' or other serial
clients (I initially used the meshtastic-python app)
* It is a _long_ time since I've messed with USB code and my analysis is
probably pretty sloppy here.  But I wanted it report it upstream in case
it is useful for others.  If there is a better fix for this problem (likely)
I'd love to change to it.

'fixes' meshtastic/firmware#838
@geeksville
Copy link
Member Author

fixed in Adafruit_TinyUSB_Arduino. The next clean build (for anyone who does "pio update") should get this fix automatically.

geeksville added a commit to geeksville/Adafruit_TinyUSB_Arduino that referenced this issue Aug 15, 2021
@geeksville
rough fix for linux serial clients hanging while talking to nrf52 cdc…
43cced0 
… serial

Problem statement:
Sometimes characters written from the nrf52 pseudo serial device to a linux
host would stop showing up on the linux app side.  This occurred randomly
but typically within a few minutes after starting a session.

The reading from linux
would remain 'hung' until you type a character on the linux side (thus
doing a new write TOWARDS the nrf52).

Theory: it seemed like something was getting hung on the linux side waiting
on a response to a read.  This led me to this code.  If I changed the code
to always send a response to any read from ep_in (rather than opting to not
send a response at all in some circumstances), "screen" apps stopped hanging.

Notes:
* problem reproducable with 'screen /dev/ttyACM0 921600' or other serial
clients (I initially used the meshtastic-python app)
* It is a _long_ time since I've messed with USB code and my analysis is
probably pretty sloppy here.  But I wanted it report it upstream in case
it is useful for others.  If there is a better fix for this problem (likely)
I'd love to change to it.

'fixes' meshtastic/firmware#838
@geeksville geeksville mentioned this issue Aug 15, 2021
Closed
@geeksville
Copy link
Member Author

fixed in lib above

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@geeksville geeksville

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@geeksville @HookdomPonix