Adds a watch flag to watch the elf file and reloads the file if it changed

[JomerDev]

This PR adds a --watch_elf/-w flag that watches the elf file for changes and reloads defmt-print on file modify or create.

To make this work I had to add tokio as dependency, since stdin.read otherwise blocked until some more serial data is received at which point it is too late to make the reload (if you do you end up with broken frames for the first few messages).

I also had to add alterable_logger to defmt-decoder to allow for the global logger to be overwritten after it was already set.

I'm open to making the flag and/or the logger changes depending on a feature.

This PR would fix issue #794

In verbose mode the logger also catches some logs from the notify crate used for the file watcher. This can probably be worked around if need be

[Urhengulas]

Could this not be achieved with cargo watch?

[JomerDev]

In theory yes, however afaik you'd usually run defmt-print by piping data from a serial port into it and I doubt cargo watch sends any through stdin received data to the inner command

[jonathanpallant]

This generally looks OK to me, modulo a few comments.

I am a little bit worried about adding tokio to the defat-print dependency tree, but maybe someone can persuade me that this is better than running the TCP stream read in a thread with a 500ms timeout that checks some shared shutdown flag.

[jonathanpallant]

I tested this on macOS, and I the file watching didn't appear to work.

I also wonder if some extra logging is useful, to let people know the ELF has been changed and reloaded. The --verbose flag does that.

The problem here appears to be the notifier gave the full path, but I passed a relative path to defmt-print, and they don't match. So it never reloaded the file.

[Urhengulas]

Maybe it actually does make sense to use async. In the end defmt-print does a lot of waiting either on stdin or tcp. But I'd like if we first port the current behaviour of defmt-print to async in a separate PR and see if that makes sense. And then we add the watch flag after.

@JomerDev Would you be willing to do that work?

[JomerDev]

Maybe it actually does make sense to use async. In the end defmt-print does a lot of waiting either on stdin or tcp. But I'd like if we first port the current behaviour of defmt-print to async in a separate PR and see if that makes sense. And then we add the watch flag after.

@JomerDev Would you be willing to do that work?

That would pretty much just mean that I'll split this PR apart, keeping the file watch stuff in here and move the async stuff into a separate PR, right? I can do that, sure

[JomerDev]

Done, see #855

[Urhengulas]

Please rebase on main. I will review next week.

[Urhengulas]

@JomerDev ping

[JomerDev]

Sorry, I didn't have much time last week,. I'll try and get it done tomorrow

[Urhengulas]

First round of review, still need to actually test it

[JomerDev]

So, I don't know why the lint is complaining, I tried to apply it's suggestion but the rust-analyzer complains. I also don't get the lint when I run clippy locally

[Urhengulas]

So, I don't know why the lint is complaining, I tried to apply it's suggestion but the rust-analyzer complains. I also don't get the lint when I run clippy locally

I do see the suggestion locally and also my RA is happy with it. I am using clippy 0.1.80 (0514789 2024-07-21).

The diff is:

diff --git a/print/src/main.rs b/print/src/main.rs
index 88b0cfd..0125a27 100644
--- a/print/src/main.rs
+++ b/print/src/main.rs
@@ -134,7 +134,7 @@ async fn has_file_changed(rx: &mut Receiver<Result<Event, notify::Error>>, path:
     true
 }
 
-async fn run_and_watch(opts: Opts, mut source: &mut Source) -> anyhow::Result<()> {
+async fn run_and_watch(opts: Opts, source: &mut Source) -> anyhow::Result<()> {
     let (tx, mut rx) = tokio::sync::mpsc::channel(1);
 
     let mut watcher = RecommendedWatcher::new(
@@ -151,7 +151,7 @@ async fn run_and_watch(opts: Opts, mut source: &mut Source) -> anyhow::Result<()
 
     loop {
         select! {
-            r = run(opts.clone(), &mut source) => r?,
+            r = run(opts.clone(), source) => r?,
             _ = has_file_changed(&mut rx, &path) => ()
         }
     }

[Urhengulas]

I just tried it locally, but unfortunately could not get it to work.

My assumption

I think the path filter is not correct.

What I did

First, I patched qemu-run to not decode the defmt frames but just dump it raw.¹

I take dbg.rs as an example

• the source is at firmware/qemu/src/bin/dbg.rs
• the binary is at firmware/target/thumbv7m-none-eabi/debug/dbg
• the defmt frames are at firmware/qemu/src/bin/dbg.out

Then from print/ I execute defmt-print, which prints the frames once:

$ bat ../firmware/qemu/src/bin/dbg.out | cargo run -q -- -e ../firmware/target/thumbv7m-none-eabi/debug/dbg --watch-elf
TRACE x + 1 = 43
TRACE x - 1 = 41
INFO  the answer is 41
TRACE x - 2 = 40
TRACE x + 2 = 44
TRACE 
TRACE x = 42

But when I re-compile with (executed in repository root) cargo xtask test-snapshot dbg --overwrite, it does not seem to restart

I added some debug statements to has_file_changed (see below) and the path filter blocks all the events. Probably it is because of absolute and relative path, or because of the ...

[print/src/main.rs:124:13] &path = "../firmware/target/thumbv7m-none-eabi/debug/dbg"
[print/src/main.rs:124:13] &event = Event {
    kind: Remove(
        File,
    ),
    paths: [
        "/home/urhengulas/Documents/github.com/knurling-rs/defmt/print/../firmware/target/thumbv7m-none-eabi/debug/dbg",
    ],
    attr:tracker: None,
    attr:flag: None,
    attr:info: None,
    attr:source: None,
}
[print/src/main.rs:125:16] event.paths.contains(path) = false
[print/src/main.rs:124:13] &path = "../firmware/target/thumbv7m-none-eabi/debug/dbg"
[print/src/main.rs:124:13] &event = Event {
    kind: Create(
        File,
    ),
    paths: [
        "/home/urhengulas/Documents/github.com/knurling-rs/defmt/print/../firmware/target/thumbv7m-none-eabi/debug/dbg",
    ],
    attr:tracker: None,
    attr:flag: None,
    attr:info: None,
    attr:source: None,
}
[print/src/main.rs:125:16] event.paths.contains(path) = false
[print/src/main.rs:124:13] &path = "../firmware/target/thumbv7m-none-eabi/debug/dbg"
[print/src/main.rs:124:13] &event = Event {
    kind: Access(
        Close(
            Write,
        ),
    ),
    paths: [
        "/home/urhengulas/Documents/github.com/knurling-rs/defmt/print/../firmware/target/thumbv7m-none-eabi/debug/.cargo-lock",
    ],
    attr:tracker: None,
    attr:flag: None,
    attr:info: None,
    attr:source: None,
}
[print/src/main.rs:125:16] event.paths.contains(path) = false

async fn has_file_changed(rx: &mut Receiver<Result<Event, notify::Error>>, path: &PathBuf) -> bool {
    loop {
        if let Some(Ok(event)) = rx.recv().await {
            dbg!(&path, &event); // 124
            if dbg!(event.paths.contains(path)) {
                dbg!(&event); // 126
                match event.kind {
                    notify::EventKind::Create(_) | notify::EventKind::Modify(_) => {
                        dbg!(event); // 129
                        break;
                    }
                    _ => (),
                }
            }
        }
    }
    true
}

Footnotes

1. I can send you a branch if you want it for testing. ↩

[JomerDev]

Hmm. I definitely agree that it has to do with the relative path. A possible fix could be to just remove every instance of ../ or ./ we find in the given path (since we only check that the event paths contains the given path), though that could potentially cause issues. A better solution might be to use std::fs::canonicalize on the given path, I would assume that event.paths only contains canonicalized paths (but I do not actually know if that is true)

[Urhengulas]

@JomerDev I played a bit around with it and for it seems it is enough if we canonicalize the ELF path the user provides. Can you please test if that works for you?

[Urhengulas]

@Urhengulas said:

@JomerDev I played a bit around with it and for it seems it is enough if we canonicalize the ELF path the user provides. Can you please test if that works for you?

@jonathanpallant Can you please test if that also works on MacOS?

[jonathanpallant]

On macOS:

Console 1:

$ nc -k -l 8000 | cargo run --bin defmt-print --release -- -e /Users/jonathan/Documents/ferrous-systems/rust-exercises/nrf52-code/radio-app/target/thumbv7em-none-eabihf/release/hello --show-skipped-frames -w
    Finished `release` profile [optimized + debuginfo] target(s) in 0.05s
     Running `target/release/defmt-print -e /Users/jonathan/Documents/ferrous-systems/rust-exercises/nrf52-code/radio-app/target/thumbv7em-none-eabihf/release/hello --show-skipped-frames -w`
Hello, world
Hello, test

Console 2:

$ echo -n -e "\x02\x00" | nc localhost 8000
$ $EDITOR src/bin/hello.rs
$ cargo build --release
$ echo -n -e "\x02\x00" | nc localhost 8000

Seems to work as expected.

[jonathanpallant]

I also tested it with ../../ferrous-systems/rust-exercises/nrf52-code/radio-app/target/thumbv7em-none-eabihf/release/hello and ~/Documents/ferrous-systems/rust-exercises/nrf52-code/radio-app/target/thumbv7em-none-eabihf/release/hello and both were fine.

[Urhengulas]

@jonathanpallant thank you for testing

@JomerDev thank you for the work