Document the standard way to support prod and dev deployments

[DominicOram]

As a support engineer I would like to be able to deploy my services (IOCs, BlueAPI plans etc.) in a way so that I can:

1. Quickly iterate and fix production code (dev mode)
2. Be certain what is in production and not have changes that are live that are not in version control (production mode)

These are obviously somewhat in conflict but a balance should be struck between the two.

Specifically in use dev mode I need to be able to:

• Put a debugger on live code
• Edit live code and see the change live within <1min

I need to switch between the codebase being in dev and production mode in a reasonable time frame e.g 5 mins and as simply as possible e.g. a 3 step process. External clients to my service should not care if I'm in dev mode or production mode and the service should behave exactly the same in either mode other than allowing live editing.

I would like it to be clear when I am using a "dev mode" service in production, ideally with alerts to many developers so that I can foster a culture of productionising code ASAP.

[callumforrester]

@gilesknap, @keithralphs and @DiamondJoseph have done some looking into live debugging of containers. The main consideration is how we deal with ephemeral changes inside containers.

[DiamondJoseph]

From discussion with Tom just now, proposed loop is as such

How to enable hotswapping in remote debugging microsoft/debugpy#212 (comment)

This means we can have tight loop very fast e.g. insertion of prints, trying out potential fixes, then when the issue has been fixed only there do we go through the build container, release, wait for Argo to pick up new version loop. But we also never lose the potential fixes: they live inside the developer machine until they are ready to be pushed to the remote. The code is always being run with the infrastructure (devices, other services).

I haven't yet tried out the hotswapping in debugpy, only the remote debugging (for blueapi, tiled)