Refactor Docker setup, follow best-practices in containerization and make caldera easier to deploy

[daw1012345]

Description

This project is currently quite difficult to get working. This is one of (at least) three pull requests intended to make caldera simpler to deploy or get running locally. Docker is a great tool for this, and should be the easiest and quickest deployment option.

Unfortunately, the current Dockerfile doesn't even build (due to an outdated version of Ubuntu).
This PR:

• Updates the Dockerfile to use a staged build to compile the UI and copies it over to the final container. This avoids the dance of installing node and npm, building the UI, then removing node and npm.
• Ensures that previous builds of caldera do not pollute the build within the container. For example, if caldera was ever compiled outside of Docker (especially the UI), it will not be recompiled in the container. This can cause a large headache, for example when the Docker build does not re-build the UI when the VITE_BASE_URL was changed.
• Containers should contain all basic requirements for all of the functionality to work. For example - if a container is compiled without emu support and then published to DockerHub, a user will not be able to easily get all the dependencies into the container (without modifying the Dockerfile and building it themselves). This PR ensures this is the case.
• The emu and atomic plugins depend on large submodules to work, but are capable of fetching them when they are enabled. This PR introduces two flavours of images - slim and full. The full (default) flavor contains said submodules to allow the container to work in environments without an internet connection, while the slim flavour disables the plugins and only fetches the submodules when they are enabled.
• Modify the workflow to build and push both slim and full images to GHCR (tagged appropriately).
• Adds a dependency on setuptools to work with recent versions of Python3
• Makes Docker the recommended deployment option in the README
• Every deployment of caldera using any of the currently provided Docker containers has the same password as it is generated during build-time.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How Has This Been Tested?

Repeated build tests of both flavors of caldera (full + slim). The resulting containers appear to work when interacted with.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have made corresponding changes to the documentation
• [NOT RELEVANT] I have added tests that prove my fix is effective or that my feature works

[rfulwell]

These changes are super interesting and I look forward to going through them in detail! 🚀

Unfortunately, the current Dockerfile doesn't even build (due to an outdated version of Ubuntu).

In the meantime, here's the minimal fix for the Docker build which was approved and just waiting on a merge:
#3099 @daw1012345

[rfulwell]

Docker is a great tool for [deployment], and should be the easiest and quickest deployment option.

💯

Thank you for tackling these changes! I fully agree that this project will be much more useful when Docker deployment is trivial. 👍

[daw1012345]

Thanks @rfulwell. I made a few additional change to the Docker setup, I was wondering if you could test it yourself as well to make sure everything works? I am by no means a caldera power user and would like to make sure these changes don't break anything.

[daw1012345]

And thank you for the original PR fixing the outdated Ubuntu version in the Dockerfile, hope that gets merged quickly as well.

[elegantmoose]

On our queue.

[endiz]

While I don't use docker for this project, I think quite a lot of people do to quickly get started. This PR would be a great service to the community.

[elegantmoose]

@endiz Yep, @uruwhy is gonna look at this one once he has a minute.

[uruwhy]

some minor merge conflicts and a couple suggestions to get rid of the warning messages, and should be good to go

[uruwhy]

One final piece since it looks like there's a conflict between the setuptools in requirements.txt and in tox.ini. Can you remove the setuptools from tox.ini since we're now specifying it in requirements.txt? That should allow our workflows to run properly, and once those pass we can move this forward

[elegantmoose]

@uruwhy approved and this is now on queue. Will look at asap.

[elegantmoose]

@daw1012345 @endiz @rfulwell - want to get your thoughts:

UPDATE: Belay below comments. Github error, was not running right dockerfile.

1. Any issues with using Ubuntu 24.04 instead of 23.04? (since LTS)
2. By default, this creates the user credentials and puts them in the container at caldera/conf/local.yaml; i.e. a user will need to do a docker exec -it <container> /bin/bash to get in the container and then view the user credentials to use. Is this okay (i.e. not too annoying)? Or would we want to like print out to terminal the user credentials so easy to get/use.

[daw1012345]

@elegantmoose I had no issues using a newer version of Debian>

Regarding your second point: it prints out the credentials on first run already (at the very top).

[elegantmoose]

I dont follow. You mean like where the config file is?

[daw1012345]

I dont follow. You mean like where the config file is?

No, it prints out the generated credentials.

For example:

(This is a sample container, the random credentials are not sensitive)

[elegantmoose]

weird, not seeing that at all. Let me investigate.

--> UPDATE github had a hiccup. (was telling me was on your branch, but none of the new code was there)

[elegantmoose]

@daw1012345 Alright cool, working clean now.

One request, could be we put like ENV COLUMNS=120 in the Dockerfile so that the Caldera log output is not too narrow?

Impetus: When I went to copy the credential password from the log output, it breaks over multiple lines and has large gap spaces, so I had to paste into notepad, remove newlines and return then re copy, then paste into Caldera.

[daw1012345]

@elegantmoose The better approach would be to encourage the users to run it in proper TTY mode (-it flag).

[elegantmoose]

TY @daw1012345 . 👏