Skip to content

Latest commit

 

History

History
220 lines (129 loc) · 7.36 KB

CHANGELOG.md

File metadata and controls

220 lines (129 loc) · 7.36 KB

Changelog

Unreleased

0.8.2 - 2023-08-29

  • Fixed incompatibility with airflow's dumb-init entrypoint

0.8.1 - 2023-03-17

  • Submit_operator supports node and pipeline parameters
  • Pass node and pipeline parameters to submit_operator in airflow_dag_template.j2 jinja template
  • Extra option to skip grouping Spark nodes

0.8.0 - 2022-03-23

  • Support spark projects on K8S
  • FIX: handle KEDRO_ENV environment variable
  • FIX: Removed hardcoded project path from spark job

0.7.3 - 2021-11-16

  • Take DAG status from the final task
  • Fix finding pyspark DataFrames
  • MLFLOW_RUN_ID passed as environment variable to dataproc oriented pipelines

0.7.2 - 2021-10-25

  • Support annotations with quotes
  • For pyspark projects, allows to configure post script for dataproc initialization
  • FIX: Broken support for kedro<0.17

0.7.1 - 2021-10-21

  • Support for failure notifications via slack
  • FIX: Missing jinja template for dataproc init script

[0.7.0] - 2021-10-19

  • Schedule supports dag-name parameter
  • Support for kedro with pyspark, using Google Dataproc
  • Support for custom pod templates
  • FIX: adding missing dependency package: tabulate
  • FIX: Fix default config template
  • Support populating k8s node env_vars from Airflow variables
  • Generalize auth handler
  • Added VarsAuthHandler for MLflow authentication which gets credentials from Airflow variables
  • Changed logging level for pod creation request to debug

0.6.7 - 2021-09-01

  • Support for generation of authentication header for secured MLflow API endpoint (via GOOGLE_APPLICATION_CREDENTIALS)

0.6.6 - 2021-08-16

  • Support for passing Authorization header for secured Airflow API endpoint (via env variable: AIRFLOW_API_TOKEN)
  • Logging dag_id and execution_date in mlflow run params

0.6.5 - 2021-08-05

  • FIX: Adjust service account setup for image based tasks

0.6.4 - 2021-08-05

  • FIX: Adjusted operators to make them compatible with Airflow >= 2.1.1
  • FIX: Restore dependency versions that release process bumped unintentionally

0.6.3 - 2021-08-04

  • FIX: Avoid file based Jinja template for data-volume-init

0.6.2 - 2021-08-04

  • FIX: Add missing Jinja template for data-volume-init to module manifest

0.6.1 - 2021-08-04

  • FIX: service_account_name or image_pull_secrets should be passed to data-volume-init step as well

0.6.0 - 2021-07-29

  • Added option to specify service_account_name or image_pull_secrets to executed dag.

0.5.4 - 2021-07-21

  • Run config contains optional start_date in format YYYYMMDD, if not specified default is left to days_ago(2)

0.5.3 - 2021-07-12

  • Support for airflow macro parameters and variables

0.5.2 - 2021-07-05

  • Support for Secrets in k8s
  • FIX: DeletePipelineStorageOperator was missing trigger rule 'all_done'

0.5.1 - 2021-05-17

  • Docker image added as a Mlflow run parameter (to support kedro inference pipeline)

0.5.0 - 2021-04-30

  • External dependencies can be added as optionals in configuration
  • Support for labels, tolerations and annotations in k8s
  • Logging added to operators

0.4.0 - 2021-04-20

  • Support of S3 as DAG destination
  • Operators extracted from DAG template as reusable components
  • Selection of specific pipeline by name from CLI

0.3.0 - 2021-04-15

  • Resources configuration added that allows describing cpu and memory resources required in k8s by pods
  • Shared persistent volume can be made optional
  • Pod startup timeout is configurable, with default to 600 seconds

0.2.0 - 2021-04-01

  • Command list-pipelines allows display of pipelines which were generated with the plugin
  • Command ui opens webbrowser with Apache Airflow
  • Move options to config file and restructure
  • MLflow enabled only if kedro-mlflow present in dependencies and configuration is in place
  • Command init to initialize configuration for the plugin in kedro project

0.1.2 - 2021-03-24

  • Allow override image parameter from CLI for upload_pipeline, run_once and compile
  • Conditional SSL verification for Apache Airflow client, via AIRFLOWCLIENTSSL_VERIFY environment variable
  • run-once with wait-for-completion checks for task instance failure as well

0.1.1 - 2021-03-19

  • Inits temporary volume with data from image kedro data directory (/home/kedro/data)
  • Increased startup time for pods to 10 minutes

0.1.0 - 2021-03-17

  • Creates mlflow experiment on pipeline start if it does not exist
  • Temporary persistent volumes generation and removal for pipelines
  • upload-pipeline command that generates DAG to Airflow directory
  • schedule command that generates DAG to Airflow directory with the given schedule
  • run-once command that uploads pipeline to Airflow and creates the DAG run

0.0.4 - 2021-03-10

Fixed

  • Added DAG template to package manifest

0.0.3 - 2021-03-09

Fixed

  • Drop dependency on airflow

0.0.2 - 2021-03-09

Added

  • Initial implementation of kedro_airflow_k8s plugin stub.