Airflow integration in kubeflow pipeline #160
Labels
enhancement
New feature or request
Comments
|
@goswamig . Did something similar recently may not be exactly , I can share steps but I think we need to have a more strong rationale and justification to have Airflow integrated within Kubeflow, since end of the day orchestration is already taken care |
|
@Harikantipudi in AWS distro of kubeflow, we are trying to help customers who wants to use some of the managed services like RDS (meta data store), S3 (artifact storage), Cognito (user managements) etc. I believe mapping out Argo to managed services like aws managed airflow does frees up resources usage of argo side car that runs in the data plane of clusters. |
judyheflin
pushed a commit
to judyheflin/kubeflow-manifests
that referenced
this issue
Apr 20, 2022
judyheflin
pushed a commit
that referenced
this issue
Apr 22, 2022
* Update _index.md * Rename _index.md to .._index.md * Rename content/en/docs/How-tos/.._index.md to content/en/docs/Tasks/_index.md * Rename content/en/docs/How-tos/knitting-paragraphs.md to content/en/docs/Tasks/knitting-paragraphs.md * Rename content/en/docs/How-tos/weaving-words.md to content/en/docs/Tasks/weaving-words.md * Rename content/en/docs/How-tos/Ponycopters/_index.md to content/en/docs/Tasks/Ponycopters/_index.md * Rename content/en/docs/How-tos/Ponycopters/configuring-ponycopters.md to content/en/docs/Tasks/Ponycopters/configuring-ponycopters.md * Rename content/en/docs/How-tos/Ponycopters/launching-ponycopters.md to content/en/docs/Tasks/Ponycopters/launching-ponycopters.md * improved favicon instructions * update theme submodule * Fixed a typo and some code formatting. * More detail on landing pages and templates * Info on updating the theme * Update _index.md * Update _index.md * Fixing a badly used shortcode... * Update _index.md * Update README.md * update config to demonstrate feedback widget * s/ocassionally/occasionally/ * s/ocasionally/occasionally/ * Updating theme submodule * Linking to the staged website from the README. * Fixing broken link to example repository. * Clarified where to find the user guide. * Updating theme submodule * Adding resources * Adding instructions * Goldydocs rebranding part one... * More content updates/simplification/porridgification * Updated tasks page * Improved overview instructions * Contribution guidelines * Contribution guidelines * Updating theme submodule * Making the templatey nature more explicit * Updated example getting started page * Fix default baseURL or things go weird with absolute URLs in menus.... * Update config.toml * Update config.toml * Bump js-yaml from 3.12.1 to 3.13.1 Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.12.1 to 3.13.1. - [Release notes](https://github.com/nodeca/js-yaml/releases) - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@3.12.1...3.13.1) Signed-off-by: dependabot[bot] <support@github.com> * Update config.toml Co-Authored-By: RichieEscarez <rescarez@google.com> * Update config.toml Co-Authored-By: RichieEscarez <rescarez@google.com> * Updating theme submodule * tiny change for github subdir * small change to github subdir * remove redundant resources folder * Update README.md * Update README.md Removed reference to nonexistent theme docs in this repo * removing redundant files * Update config.toml * Added links to user guide repo * 404 page * Update _index.md * Changing placekitten links to be https * Update _index.html * Adding missing title to README.md * updating theme * Update README.md * Bump lodash from 4.17.11 to 4.17.14 Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.14. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.11...4.17.14) Signed-off-by: dependabot[bot] <support@github.com> * Add config option for about link in footer I needed a way to disable the About link in the footer and discovered this option by looking in the code. It's working for my site using the Docsy theme so I thought other people might benefit from knowing about it. * Change the value to false Changed the value to false so the Docsy example site is not affected * Move images to params.images * Bump mixin-deep from 1.3.1 to 1.3.2 Bumps [mixin-deep](https://github.com/jonschlinkert/mixin-deep) from 1.3.1 to 1.3.2. - [Release notes](https://github.com/jonschlinkert/mixin-deep/releases) - [Commits](jonschlinkert/mixin-deep@1.3.1...1.3.2) Signed-off-by: dependabot[bot] <support@github.com> * Updating theme submodule * Update config.toml Add a param for Algolia docsearch, related to PR on Docsy theme google/docsy#127 * Adding offlineSearch param As requested in https://github.com/google/docsy/pull/142/files#r338549380 * Bump packages with CVEs * Create search-index.md * Updating theme submodule * Update _index.md * Let Goldmark render raw HTML * Remove div that's confusing Goldmark Fixes google/docsy#174 * Added parameters for version banner. * Updating theme submodule * Updating theme submodule * Remove a unnecessary file (content/en/search-index.md) * Add reading time parameter * Updating theme submodule * Fix typo descripton -> description * point to google/docsy-example instead * update package.json and package-lock.json + Update dev dependencies to latest versions. (I confirmed that running `hugo server` and viewing the site works as expected.) + Fix URLs in package file. Fixes #45. * Create dependabot.yml Help keep the dev dependencies up to date. https://dependabot.com/#how-it-works * fixing typo * Bump lodash from 4.17.15 to 4.17.19 Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.19. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.15...4.17.19) Signed-off-by: dependabot[bot] <support@github.com> * Update config.toml Update to work with latest Chroma style support * Updating theme submodule * Updating theme submodule * Bump autoprefixer from 9.8.0 to 9.8.6 Bumps [autoprefixer](https://github.com/postcss/autoprefixer) from 9.8.0 to 9.8.6. - [Release notes](https://github.com/postcss/autoprefixer/releases) - [Changelog](https://github.com/postcss/autoprefixer/blob/master/CHANGELOG.md) - [Commits](postcss/autoprefixer@9.8.0...9.8.6) Signed-off-by: dependabot[bot] <support@github.com> * add dockerfile and change readme with instructions to build and run the docker image * Bump postcss-cli from 7.1.1 to 7.1.2 Bumps [postcss-cli](https://github.com/postcss/postcss-cli) from 7.1.1 to 7.1.2. - [Release notes](https://github.com/postcss/postcss-cli/releases) - [Changelog](https://github.com/postcss/postcss-cli/blob/master/CHANGELOG.md) - [Commits](postcss/postcss-cli@7.1.1...7.1.2) Signed-off-by: dependabot[bot] <support@github.com> * Updating theme submodule * Add config param for prism relates to google/docsy#325 * Set the prism option to false by default * Fixing broken links * Updating Docsy submodule * Updating theme submodule * Update docsy theme & enable 'print entire section' * Add persian translations. * Update README.md Add template instructions * Updating theme submodule * 🏗️ Dockerfile + docker-compose.yaml build/running Add Dockerfile for building with the latest and the docker-compose, building the site and running. $ docker-compose up Starting docsy-example_site_1 ... done Attaching to docsy-example_site_1 site_1 | Start building sites … site_1 | site_1 | | FA | NO | EN site_1 | -------------------+----+----+----- site_1 | Pages | 18 | 76 | 46 site_1 | Paginator pages | 0 | 0 | 0 site_1 | Non-page files | 3 | 1 | 3 site_1 | Static files | 38 | 38 | 38 site_1 | Processed images | 8 | 2 | 7 site_1 | Aliases | 3 | 0 | 3 site_1 | Sitemaps | 2 | 1 | 1 site_1 | Cleaned | 0 | 0 | 0 site_1 | site_1 | Built in 837 ms site_1 | Watching for changes in /src/{assets,content,layouts,package.json,themes} site_1 | Watching for config changes in /src/config.toml, /src/themes/docsy/config.toml site_1 | Environment: "DEV" site_1 | Serving pages from memory site_1 | Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender site_1 | Web Server is available at //localhost:1313/ (bind address 0.0.0.0) site_1 | Press Ctrl+C to stop Open a new terminal and test it $ curl -X HEAD -i localhost:1313 Warning: Setting custom HTTP method to HEAD with -X/--request may not work the Warning: way you want. Consider using -I/--head instead. HTTP/1.1 200 OK Accept-Ranges: bytes Content-Length: 27590 Content-Type: text/html; charset=utf-8 Last-Modified: Thu, 17 Dec 2020 08:47:13 GMT Date: Thu, 17 Dec 2020 08:47:25 GMT * Update to latest Docsy * Add a troubleshooting section. Error building site: TOCSS: failed to transform "scss/main.scss" (text/x-scss): resource "scss/scss/main. Solution: install hugo-extended Signed-off-by: Gaurav Gahlot <gauravgahlot0107@gmail.com> * Update README.md * 🐳 Dockerfile: Use the repo name as image tag * 📝 README: add steps with docker-compose Running with docker-compose is simpler for this case * Update Docker Compose instructions. * Fix indentation * Make cleanup steps a subsection * Add github_branch * Dockerfile: use base image with extended version of Hugo * Update search engine Fixes google/docsy#479 * Bump y18n from 4.0.0 to 4.0.1 Bumps [y18n](https://github.com/yargs/y18n) from 4.0.0 to 4.0.1. - [Release notes](https://github.com/yargs/y18n/releases) - [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md) - [Commits](https://github.com/yargs/y18n/commits) Signed-off-by: dependabot[bot] <support@github.com> * Dockerfile: use klakegg/hugo:ext-alpine * Bump lodash from 4.17.20 to 4.17.21 Bumps [lodash](https://github.com/lodash/lodash) from 4.17.20 to 4.17.21. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.20...4.17.21) Signed-off-by: dependabot[bot] <support@github.com> * Bump browserslist from 4.14.0 to 4.16.6 Bumps [browserslist](https://github.com/browserslist/browserslist) from 4.14.0 to 4.16.6. - [Release notes](https://github.com/browserslist/browserslist/releases) - [Changelog](https://github.com/browserslist/browserslist/blob/main/CHANGELOG.md) - [Commits](browserslist/browserslist@4.14.0...4.16.6) Signed-off-by: dependabot[bot] <support@github.com> * Bump glob-parent from 5.1.1 to 5.1.2 Bumps [glob-parent](https://github.com/gulpjs/glob-parent) from 5.1.1 to 5.1.2. - [Release notes](https://github.com/gulpjs/glob-parent/releases) - [Changelog](https://github.com/gulpjs/glob-parent/blob/main/CHANGELOG.md) - [Commits](gulpjs/glob-parent@v5.1.1...v5.1.2) --- updated-dependencies: - dependency-name: glob-parent dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Bump postcss from 7.0.32 to 7.0.36 Bumps [postcss](https://github.com/postcss/postcss) from 7.0.32 to 7.0.36. - [Release notes](https://github.com/postcss/postcss/releases) - [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md) - [Commits](postcss/postcss@7.0.32...7.0.36) --- updated-dependencies: - dependency-name: postcss dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Updating theme submodule * Update config.toml * Updating theme submodule * Add taxonomy sample code Enable taxonomoy in `config.toml` and add tags and categories to the FrontMatter in the "Getting Started" docs. * Add navbar_translucent_over_cover_disable and sort ui elements * Delete package-lock.json and ignore it * Add a netlify.toml and update HUGO_VERSION * Update package.json: postcss* * Flip incorrect boolean in comment Fixes #141 * README: add link to deploy-logs * Min version of Node is latest LTS version * Add link to official Netlify dashboard * Bump autoprefixer from 9.8.8 to 10.4.0 Bumps [autoprefixer](https://github.com/postcss/autoprefixer) from 9.8.8 to 10.4.0. - [Release notes](https://github.com/postcss/autoprefixer/releases) - [Changelog](https://github.com/postcss/autoprefixer/blob/main/CHANGELOG.md) - [Commits](postcss/autoprefixer@9.8.8...10.4.0) --- updated-dependencies: - dependency-name: autoprefixer dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Bump postcss-cli from 8.3.1 to 9.0.2 Bumps [postcss-cli](https://github.com/postcss/postcss-cli) from 8.3.1 to 9.0.2. - [Release notes](https://github.com/postcss/postcss-cli/releases) - [Changelog](https://github.com/postcss/postcss-cli/blob/master/CHANGELOG.md) - [Commits](postcss/postcss-cli@8.3.1...9.0.2) --- updated-dependencies: - dependency-name: postcss-cli dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * config.toml: replace tabs with whitespace * Bumped docsy theme to latest version * README.md: correct dead link (installing/using themes) (#160) * config.toml: change order of language settings and theme/module declaration (#162) * Updating docsy theme submodule (#164) * config.toml: comment typo: s/lang/length I don't think taxonomyCloud has a language, and I think they need to have the same length, so I think this is what was intended. * Leave param name unqualified in comment * Make docsy example site use docsy theme as hugo module (#156) * docsy as module: refer to current HEAD of repo (#169) * Remove outdated deployment script (#170) * Update example site to docsy theme v0.2.0 (#171) * README.md: reflect changes due to use of theme as hugo module (#172) * Checkpoint for Kubeflow on AWS 1.4 docs staging 4/12 7:47pm * Finished About section organization * Fixed AWS logos and updated docs structure * AWS-specific doc restructuring * About page notes update * Added prereq page and deployment drafts * Added Profiles component guide * Restructured Deployment docs for KF on AWS website * Syncing About, Deployment, and Components section * Added AWS Labs kubeflow-manifests doc issue template to feedback prompt * Staged Uninstall deployment doc * Fixed Uninstall relative links * Move all files into src/website directory * Cleaned up extra files * Added website working directory to kubeflow-manifests repo * Added Hugo Docsy theme submodule * Testing GitHub actions build * Modified Config file for testing * Removed GH actions test * Added PostCSS dependency * GH actions test #2 * GH actions test #3 * Restructured deployment pages * Updated RDS&S3, EFS, and FSx installation guides based on recent PRs * Reverted file structure and refreshed EFS/FSx/RDS&S3 docs * GH actions test * Removed GH workflows script * Minimized docs pages for website staging * Load balancer installation guide refresh * Updated Usage Tracking command for KF on AWS * Finished uninstall and RDS S3 pages * Finished vanilla guide edits * Deployment PR feedback and link fixes Co-authored-by: LisaFC <lcarey@google.com> Co-authored-by: Sarah Maddox <sarahmaddox@google.com> Co-authored-by: Kayce Basques <kayce@google.com> Co-authored-by: Kayce Basques <kaycebasques@gmail.com> Co-authored-by: Aijamal NK <aizhamal.nk@gmail.com> Co-authored-by: Andrew Chen <chenopis@users.noreply.github.com> Co-authored-by: Sharif Salah <sharifsalah@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: RichieEscarez <rescarez@google.com> Co-authored-by: Erin McKean <emckean@google.com> Co-authored-by: Kshithij Iyer <kshithij.ki@gmail.com> Co-authored-by: Alex Earnshaw <aearnshaw@axway.com> Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh> Co-authored-by: Fabrizio Ferri-Benedetti <algernon@gmail.com> Co-authored-by: Austin Macdonald <austin@redhat.com> Co-authored-by: Sarah Maddox <sarahmaddox@users.noreply.github.com> Co-authored-by: Takayama Fumihiko <tekezo@pqrs.org> Co-authored-by: Lisa Carey <lcarey@lcarey-macbookpro.roam.corp.google.com> Co-authored-by: Ulf Adams <ulfjack@users.noreply.github.com> Co-authored-by: Daniel Holbach <daniel@weave.works> Co-authored-by: Jeff Williams <jefesaurus@google.com> Co-authored-by: Christian Oliff <christianoliff@pm.me> Co-authored-by: Christoph Raaflaub <raaflaub@puzzle.ch> Co-authored-by: Geri Ochoa <gerino@protonmail.com> Co-authored-by: Alex Earnshaw <alex.earnshaw@gmail.com> Co-authored-by: Andreas Deininger <adeininger@gmail.com> Co-authored-by: Gareth Watts <gareth@omnipotent.net> Co-authored-by: theSinner <amirajorloo@gmail.com> Co-authored-by: Marcello de Sales <marcello.desales@gmail.com> Co-authored-by: Gaurav Gahlot <gauravgahlot0107@gmail.com> Co-authored-by: Daan Seynaeve <daan.seynaeve@openanalytics.eu> Co-authored-by: Patrice Chalin <pchalin@gmail.com> Co-authored-by: Mark Hansen <markhansen@google.com> Co-authored-by: Andreas Deininger <adeininger@urbanonline.de> Co-authored-by: Andreas Deininger <andreas@deininger.net> Co-authored-by: Patrice Chalin <chalin@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
Currently Kubeflow pipeline uses Argo in backend and frontend, in order to integrate it with AWS airflow, we need to enable the support of airflow.
Describe the solution you'd like
Swap out the need of argo in kubeflow to airflow.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: