Experiments: Add breakpoint to an experiment

[mattseddon]

A nice to have for experiments would be the ability to add a breakpoint and debug the workings of the experiment.

Original point can be found here in the review ticket.

[rogermparent]

@iterative/dvc Is there any existing protocol for running a debugger against pipeline executions? I can't find anything in the DVC docs.

From when I explored using the debugger API to launch exp run, the way pipeline scripts are run from a static string on the shell (e.g. python train.py) prevents us from both basic Python debugging or debugpy attaching even if the script is also Python like our demo's train.py. We can see dvc in the debugger, and that it opens another process, but we can't inspect the child process. I suppose we could run the train script directly in the debugger, but I have a hunch that would be too far from dvc exp run to be useful.

[pmrowla]

Is there any existing protocol for running a debugger against pipeline executions? I can't find anything in the DVC docs.

There is not. For one thing, this would all have to be language specific, and there is no guarantee that pipeline stages are going to be python scripts. I think the normal workflow here would be that users debug their stages individually outside of DVC.

If you really want to debug something that is running inside DVC, the way to do it would be to use remote debugging. So you would configure your pipeline stage command to run your stage inside the appropriate debugger session, and then connect to it remotely from a separate debugger process.

So for debugpy something you would configure dvc.yaml like:

stages:
  train:
    cmd: python -m debugpy --listen 1234 --wait-for-client train.py ...

and to debug it, you would start dvc exp run and then connect to the debugger session with

$ python -m debugpy --connect 1234

For non-python stages it would work the same way. For a compiled executable and GDB example:

stages:
  train:
    cmd: gdbserver :1234 ./train ...

$ gdb -q train
(gdb) target remote 172.0.0.1:1234

[mattseddon]

Closing for now as this is a lower priority and we cannot progress without changes in DVC.