Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Making the role Apache Airflow 2.0 compatible #92

Merged
merged 24 commits into from
Apr 13, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
*.yml linguist-detectable=true
*.yaml linguist-detectable=true
*.html linguist-detectable=false
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,6 @@ tests/playbook.retry
tests/__pycache__
.molecule
.cache
.vagrant
tests/roles
.vscode/*
*.vt100
12 changes: 7 additions & 5 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
---
dist: xenial
language: python
python: "3.8.6"
sudo: required

python: "3.7"
os: linux
services:
- docker

install:
- pip3 install pipenv
- pip install pipenv
- pipenv sync
env:
jobs:
- MOLECULE_DISTRO=debian:buster-slim
script:
- pipenv run molecule test --all

Expand Down
18 changes: 0 additions & 18 deletions .yamlint

This file was deleted.

38 changes: 38 additions & 0 deletions .yamllint
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
# Based on ansible-lint config
extends: default

ignore: |
.travis.yml
molecule/
tests/

rules:
braces:
max-spaces-inside: 1
level: error
brackets:
max-spaces-inside: 1
level: error
colons:
max-spaces-after: -1
level: error
commas:
max-spaces-after: -1
level: error
comments: disable
comments-indentation: disable
document-start: disable
empty-lines:
max: 3
level: error
hyphens:
level: error
indentation: enable
key-duplicates: enable
line-length: disable
new-line-at-end-of-file: disable
new-lines:
type: unix
trailing-spaces: disable
truthy: disable
91 changes: 81 additions & 10 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,124 +1,195 @@
# Change Log

All notable changes to this project will be documented in this file.
This project adheres to [Semantic Versioning](http://semver.org/) and [Keep a changelog](https://github.com/olivierlacan/keep-a-changelog).

## [Unreleased](https://github.com/idealista/airflow-role/tree/develop)

## [2.0.0](https://github.com/idealista/airflow-role/tree/2.0.0)

[Full Changelog](https://github.com/idealista/airflow-role/compare/1.8.4...2.0.0)

### Added

- Support for new Apache Airflow 2.0 and its new configs files
- New templates and new group_vars files to fit better with new Apache Airflow 2.0
- New user related tasks
- [#82](https://github.com/idealista/airflow-role/issues/82) Add support to 2.0 airflow version @lorientedev

### Changed

## [1.8.5](https://github.com/idealista/airflow-role/tree/1.8.5)
- Project cleaning
- Updated ansible and molecule test requirements
- Updated molecule tests
- Updated travis file
- Updated yamllint
- Updated service templates
- Updated config and install tasks
- Updated [`README.md`](./README.md) to give more information about the role and advice some stuff
- [#54](https://github.com/idealista/airflow-role/issues/54) Install tasks fail when run without escalated privileges

- *[#85](https://github.com/idealista/airflow-role/issues/85) Add extra args in pip install to allow different repositories @lorientedev
### Fixed

## [1.8.4](https://github.com/idealista/airflow-role/tree/1.8.4)
- [#61](https://github.com/idealista/airflow-role/issues/61) Better worker restarts
- [#85](https://github.com/idealista/airflow-role/issues/85) Add extra args in pip install to allow different repositories @lorientedev

- *[#82](https://github.com/idealista/airflow-role/issues/82) Add support to airflow 2.0 version. @lorientedev
## [1.8.4](https://github.com/idealista/airflow-role/tree/1.8.4)

## [1.8.3](https://github.com/idealista/airflow-role/tree/1.8.3)
- [#82](https://github.com/idealista/airflow-role/issues/82) Add support to airflow 2.0 version. @lorientedev

- *[#78](https://github.com/idealista/airflow-role/issues/78) Add path for services templates to allow overwrite from playbook @lorientedev
## [1.8.3](https://github.com/idealista/airflow-role/tree/1.8.3)

- [#78](https://github.com/idealista/airflow-role/issues/78) Add path for services templates to allow overwrite from playbook @lorientedev

## [1.8.2](https://github.com/idealista/airflow-role/tree/1.8.2)

- *[#75](https://github.com/idealista/airflow-role/issues/75) Added variable to change private_tmp value in service config @lorientedev
- [#75](https://github.com/idealista/airflow-role/issues/75) Added variable to change private_tmp value in service config @lorientedev

## [1.8.1](https://github.com/idealista/airflow-role/tree/1.8.1)

- *[#68](https://github.com/idealista/airflow-role/issues/68) Update dependencies versions and solve some lint errors @lorientedev
- [#68](https://github.com/idealista/airflow-role/issues/68) Update dependencies versions and solve some lint errors @lorientedev

## [1.8.0](https://github.com/idealista/airflow-role/tree/1.8.0)

[Full Changelog](https://github.com/idealista/airflow-role/compare/1.7.3...1.8.0)

### Added
- *[#61](https://github.com/idealista/airflow-role/issues/61) Add KillSignal=SIGINT to workers service file* @jnogol

- [#61](https://github.com/idealista/airflow-role/issues/61) Add KillSignal=SIGINT to workers service file @jnogol

### Fixed
- *[#50](https://github.com/idealista/airflow-role/issues/50) Fix deprecation warning from jinja templates* @adrimarteau @jnogol

- [#50](https://github.com/idealista/airflow-role/issues/50) Fix deprecation warning from jinja templates @adrimarteau @jnogol

## [1.7.3](https://github.com/idealista/airflow-role/tree/1.7.3)

[Full Changelog](https://github.com/idealista/airflow-role/compare/1.7.2...1.7.3)

### Fixed

- *[#55](https://github.com/idealista/airflow-role/pull/55) Use `{{ airflow_home }}` to set the default `airflow_database_conn` in defaults/main.yml* @davestern

## [1.7.2](https://github.com/idealista/airflow-role/tree/1.7.2)

[Full Changelog](https://github.com/idealista/airflow-role/compare/1.7.1...1.7.2)

### Fixed

- *[#47](https://github.com/idealista/airflow-role/issues/47) Fix web UI when using LDAP and Airflow>=1.10* @jnogol

## [1.7.1](https://github.com/idealista/airflow-role/tree/1.7.1)

[Full Changelog](https://github.com/idealista/airflow-role/compare/1.7.0...1.7.1)

### Changed

- *[#44](https://github.com/idealista/airflow-role/issues/44) Make role compatible with Airflow 1.10.0* @jnogol
- *Update Goss version to 0.3.6* @jnogol

## [1.7.0](https://github.com/idealista/airflow-role/tree/1.7.0)

[Full Changelog](https://github.com/idealista/airflow-role/compare/1.6.0...1.7.0)

### Changed

- *Update default version to 1.9.0* @jnogol
- *Ability to provide `airflow.cfg` template via playbooks* @jnogol
- *[#41](https://github.com/idealista/airflow-role/issues/41) Update `airflow.cfg` template with 1.9.0 features* @jnogol

### Added

- *Add sample DAG in tests to avoid scheduler issues* @jnogol
- *Add `airflow_` tags in `main.yml`* @jnogol

## [1.6.0](https://github.com/idealista/airflow-role/tree/1.6.0)

[Full Changelog](https://github.com/idealista/airflow-role/compare/1.5.0...1.6.0)

### Changed

- *[#38](https://github.com/idealista/airflow-role/pull/38) Parametrized PID files location, and create it on startup if it doesn't already exist.* @fhalim

## [1.5.0](https://github.com/idealista/airflow-role/tree/1.5.0)

[Full Changelog](https://github.com/idealista/airflow-role/compare/1.4.0...1.5.0)

### Added

- *[#36](https://github.com/idealista/airflow-role/issues/32) Add tasks in config to create variables and connections used in Airflow DAGs.* @deytao

## [1.4.0](https://github.com/idealista/airflow-role/tree/1.4.0)

[Full Changelog](https://github.com/idealista/airflow-role/compare/1.3.2...1.4.0)

### Added

- *[#32](https://github.com/idealista/airflow-role/issues/32) Use Goss instead of Testinfra* @jnogol

### Fixed

- *[#33](https://github.com/idealista/airflow-role/pull/33) Travis working* @jnogol

### Changed

- *[#31](https://github.com/idealista/airflow-role/pull/31) Using import_tasks instead of include and avoiding pip cache* @sschaetz
- *[#34](https://github.com/idealista/airflow-role/pull/34) Add config to have customizable pip and airflow executables* @deytao

## [1.3.2](https://github.com/idealista/airflow-role/tree/1.3.2)

[Full Changelog](https://github.com/idealista/airflow-role/compare/1.3.1...1.3.2)

### Fixed

- *[#28](https://github.com/idealista/airflow-role/issues/28) Fix Init DB task hang* @jnogol

## [1.3.1](https://github.com/idealista/airflow-role/tree/1.3.1)

[Full Changelog](https://github.com/idealista/airflow-role/compare/1.3.0...1.3.1)

### Fixed

- *Deleted DAGs automatic managent: better do it with CI tools* @jnogol
- *Deleted port bindings in molecule.yml: they weren't necessary* @jnogol
- *Better and more understandable format for dags_dependencies variable* @jnogol

## [1.3.0](https://github.com/idealista/airflow-role/tree/1.3.0)

[Full Changelog](https://github.com/idealista/airflow-role/compare/1.2.0...1.3.0)

### Added

- *Testinfra tests in Travis CI* @jnogol
- *DAGs and plugins automatic management via repositories and cron job* @jnogol
- *Docker environment in Molecule* @jnogol

### Fixed

- *Test if service is running in test_ansible.py now working* @jnogol

## [1.2.0](https://github.com/idealista/airflow-role/tree/1.2.0)

[Full Changelog](https://github.com/idealista/airflow-role/compare/1.1.0...1.2.0)

### Added

- *Travis CI integration added* @jnogol

### Fixed

- *Authentication via LDAP now working* @jnogol
- *Tiny bugs in tasks/install.yml fixed* @jnogol

## [1.1.0](https://github.com/idealista/airflow-role/tree/1.1.0)

[Full Changelog](https://github.com/idealista/airflow-role/compare/1.0.0...1.1.0)

### Added

- *Added Celery Worker optional installation* @jnogol
- *Added Celery Flower service configuration* @jnogol

## [1.0.0](https://github.com/idealista/airflow-role/tree/1.0.0)

### Added

- *First release*
4 changes: 2 additions & 2 deletions Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ verify_ssl = true
[dev-packages]

[packages]
ansible = "==2.8.8"
molecule = "==3.0.1"
ansible = "==2.9.9"
molecule = "==3.0.6"
docker = "==4.1.0"
ansible-lint = "==4.2.0"

Expand Down
Loading