Move database initialization into a separate Init Container

[dervoeti]

Improvement of #396.
We want to move the database initialization into a separate Init Container to have separate logs for this process.

[dervoeti]

The current problem with this is that Vector currently runs as a normal container besides the Superset container, which means it only starts after the init containers. Which means it won't start at all, if the init container fails. The recently announced native sidecar containers would be a good solution for this (= make Vector a native sidecar container, see https://kubernetes.io/blog/2023/08/25/native-sidecar-containers/). But it's in 1.28 only and still behind a feature gate that's disabled by default.

So, I see three solutions for this:

• Wait for native sidecar containers to be generally available and add a minimum Kubernetes version requirement to our platform
• Accept that logs of the init container are not captured by Vector (they can still be read via kubectl logs)
• Don't move the initialization of the database to an init container and just keep it in the command chain of the Superset container (then the logs are captured by Vector)

[fhennig]

Hmm yes interesting. Even if we wait, we need an intermediate solution. With my gut feeling I'd say we just leave it in the main container and it's done. It might be worth looking at other operators. maybe there are other init containers that are also not logged? I feel like this might be a broader issue.

[dervoeti]

I briefly talked to @siegfriedweber about this yesterday, the logs of the init container are captured by Vector if it succeeds, so the problem arises only if at least one of the init containers fails. It's probably a good idea to check our init containers, even though they will mostly do simple operations (cp, chmod, chown, ...) that are usually very unlikely to fail in practice.
I'd also say we just leave it in the main container for now. That's what's already implemented.