Migration path from stable/airflow to bitnami/airflow

[scottrigby]

Which chart:
bitnami/airflow

Is your feature request related to a problem? Please describe.

There isn't urgently a clear migration path from stable/airflow to another well maintained airflow chart.

Describe the solution you'd like

• find a migration path to move from stable/airflow to bitnami/airflow (maybe involving a major version)
• support in the bitnami/airflow the features included in the stable one that we don’t have at this moment
• deprecate the stable one following the right process, etc

@carrodher said it would be great if the current maintainers/contributors would be willing to help, and would be happy to work together on creating this migration path and moving the missing features.

Describe alternatives you've considered

A chart hosted in the airflow org. See apache/airflow#10523

Additional context

As I understand it, on the Bitnami side the only requirement for the charts is that the default images need to be bitnami images in the main chart and subcharts, for example in this case, redis & postgresql. However, there are some charts like PostgreSQL where although the default images are the bitnami ones, it is easily replaced by another postgresql images just using --set image.repository=whatever.

[carrodher]

Thanks for creating this issue @scottrigby, we are happy to investigate a migration path from the current stable to the bitnami one, adding the features/examples/docs whatever is missing to make a smooth transition.

Regarding the 2nd day maintenance, we already have some charts in our repository contributed from community users like #2072, that means we are happy to work together on adding more charts to our catalog (this time it is not an addition but a major refactor) and also providing support, updates, etc. Everything related to the chart support is handled publicly in GitHub via issues and PRs, just the tests and release process is done outside GitHub (in our internal CI/CD pipeline in this case).

Having said that, we look forward to hearing how we can collaborate with current maintainers to carry out this migration successfully (if you prefer this path, of course).

[scottrigby]

CC @gsemet @maver1ck @thesuperzapper

[thesuperzapper]

To be frank, the current stable/airflow chart is the current authoritative airflow helm chart, we aren't looking to get rid of it.

What we need is a place to host it after November (when helm 2 goes away). After November, we would likely maintain the existing chart until the offical airflow helm chart is feature complete and stable enough for widespread use.

Given you guys have release infrastructure already setup, it would be awesome if we could simply lift and shift into your repository (seperate from your existing chart of course). This saves lots of hassle on both user/dev side, especially since users will have to migrate to the official chart at some point anyway.

If this is not possible, no worries, I expect we will be able to find another place to host it.

[scottrigby]

@thesuperzapper charts within a helm repo must have unique names, so to make what you're recommending work the chart would have to be renamed before being added. Wouldn't it be better to merge any missing features from stable/airflow into bitnami/airflow? If you'd like help splicing in history from stable I could give it a shot, though I may need your help correctly reconciling conflicts, since you're intimately familiar with this chart as a maintainer.

Like you said you could alternatively host it anywhere else - under your own GitHub user for example. But it seems there's willingness to make a fruitful collaboration here. To me it seems that might ultimately be the best case scenario for end users.

[gsemet]

Given the among of history behind the stable/airflow chart, I am not sure the git history approach is realistic. I (maybe @thesuperzapper as well) simply cannot tell you right away what are the features difference with the bitnami chart.

I am not even sure the difference is that there would be only "missing" features on the bitnami chart, from what I see it looks like more complex, maybe more complete, take a slightly different approach on things like DAG propagation, etc, and aligning both may not even make any sense.

"stable/airflow" is like it is, a big baby that works in production. Our top priority is that we would like to see it alive after November.
Helping grow the official airflow chart is second, and of course we will help if possible.

But it it really important to decouple both problematics. Personally, because life is life and Airflow stuff does not feed the family, sadly, I do not have time and even the will to take the risk of aligning our chart with the bitnami chart + write and verify the migration guide + handle all use cases that will arise from this (big) change.
By november. Can work. Can require huge among of refacto and even test. Don't know.

I stick to our proposal:

• move as it is "stable/airflow" somewhere so it is november-deadline proof
• points to the official chart (Bitnami chart?) so the community knows the future is here. New users can evaluate both, and choose the one they want, old users have the minimun change to do (just update an url?) and contributors can help bridging both chart without time constraint

[thesuperzapper]

@scottrigby @gsemet I really don't mind if it needs to be renamed, but I am much more comfortable having the chat move as-is given how many people are using it in production.

So that leaves us with 3 options:

1. Move stable/airflow to bitnami/charts (replacing bitnami/airflow, and implementing any missing features only found bitnami/airflow)
2. Move stable/airflow to bitnami/charts (renaming stable/airflow so both can live concurrently in this repo)
3. Move stable/airflow to some other repo

[thesuperzapper]

@carrodher are any of these options acceptable to you?

[carrodher]

From the Bitnami POV, there are only three requirements to add a new chart to the catalog:

• The chart should use Bitnami based container images.
• Follow the same structure/patterns that the rest of the Bitnami charts and the Best Practices for Creating Production-Ready Helm charts guide.
• Use an OSI approved license for all the software.

IMHO the 2nd option is not easy to follow as you will need to meet the above requirements in order to add the chart to the bitnami catalog, although named bitnami/stable-airflow. Regarding the 3rd option is not something to be decided by us (as Bitnami) but for the stable/airflow maintainers.

About the 1st option, the already existing bitnami/airflow chart meets the above requirements, probably the stable/airflow one, as it is a mature community chart, also meets the requirements (except the one about Bitnami images). So I think the easy path here is to fuse both charts in a single one meeting the above Bitnami requirements and also the stable/airflow requirements. In this way, there is a chart hosted in the Bitnami repository following the best practices and standardization from Bitnami, integrated into the test & release pipeline, etc with all the existing features already present in stable/airflow that are missing in the current bitnami/airflow.
Of course, if that is the desired option, we are happy to help to modify the existing chart and Docker image in order to incorporate whatever is needed to support the current features, as well as preparing the transition guide from users to upgrade from stable/airflow to bitnami/airflow when needed.

[thesuperzapper]

@scottrigby my thoughts

1. I believe that using the official Airflow Docker images is important
   • In terms of redis and postgres I don't mind, and believe we are already using bitnami ones
2. I am not looking to force users into a complex 'migration' but rather be able to just point to a new repository if possible:
   • this is because users will have to migrate to the official Apache chart in a few months anyway (and at that time, we should probably deprecate the non-official charts)
3. have you compared the features of stable/airflow and bitnami/airflow recently?
   • There is a significant number of differences, and missing features from bitnami/airflow, so it may honestly be easier to do a major version bump of bitnami/airflow and just drop stable/airflow in its place (with a migration guide, obviously)

[gsemet]

I think it will be better to let the different charts live their life independently, I see too many blockers to have a smooth migration about users of current stable/airflow chart (I understand both Bitnami and airflow has their own process and quality assurance). We are begining to look at the alternative "create our own dedicated "community" github group where we will put stable/airflow as it is now without changing it.

We realy want to ease migration and so we will clearly point to the other charts, and even try to make the community create migration guides or something like this. But we don't know the bitnami chart as of today, how hard or easy it would be to change the docker image, adapt the process etc. And the among of test required to ensure it will not break somewhere is not feasible for us.

For me, from the outside, it looks like the bitnami chart is a bit too far from what we have now in stable/airflow, so let's keep different charts and let the community use whatever they want.

[carrodher]

Yes, it makes sense. From our point of view, we are happy to integrate the chart here if needed, but in the end, this is not a decision on our side, if you find it easy following a different path is fine. If finally, you consider integrating the existing chart into the Bitnami catalog following the mentioned process, we can help with that.

[stale]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.