Skip to content

Conversation

@ashb
Copy link
Member

@ashb ashb commented Oct 12, 2020

This PR documents (from a user point of view) how Scheduler HA #10956 works, what the requirements are, and what config options there are that users might want to configure.

@boring-cyborg boring-cyborg bot added area:docs area:Scheduler including HA (high availability) scheduler labels Oct 12, 2020
@ashb
Copy link
Member Author

ashb commented Oct 12, 2020

This is only a start, I'll add more. What would else would people like to see?

Copy link
Member

@XD-DENG XD-DENG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few typo fixes to suggest.

Copy link
Member

@turbaszek turbaszek Oct 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Consul for instance) we have kept the "operational surface area" to a minimum.
Consul for instance) we have kept the "operational surface area" to a minimum. For more information on architectural choices check `Airflow Improvement Proposal 15 <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651>`_.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, not sure about this one -- I like the idea, but I think I'd rather the docs be the source of truth rather than linking to the AIP.

I might link to the AIP for now though.

Copy link
Member

@turbaszek turbaszek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look good, should we put a stronger emphasis on the fact that serialization is required for HA?

@mik-laj
Copy link
Member

mik-laj commented Oct 12, 2020

Can you also add some documentation on the database setup page? If users know about this requirement in advance, they will be able to select these databases when creating an instance. Otherwise, users will rely on the default values provided by the cloud provider.
https://airflow.readthedocs.io/en/latest/howto/initialize-database.html

@ashb ashb force-pushed the scheduler-ha-docs branch from 20a5f23 to 4fe8e32 Compare October 27, 2020 10:40
@ashb
Copy link
Member Author

ashb commented Oct 27, 2020

Can you also add some documentation on the database setup page? If users know about this requirement in advance, they will be able to select these databases when creating an instance. Otherwise, users will rely on the default values provided by the cloud provider.
https://airflow.readthedocs.io/en/latest/howto/initialize-database.html

I've cross-linked it from there

image

@ashb ashb marked this pull request as ready for review October 27, 2020 11:52
@ryw ryw self-requested a review October 27, 2020 22:56
Copy link
Member

@ryw ryw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashb i think you should merge this as is - will be easier for people to collaborate on improving it w/ small PRs vs review comments.

@ashb ashb force-pushed the scheduler-ha-docs branch from c0d62f5 to 9081d34 Compare October 28, 2020 14:51
@github-actions
Copy link

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

@apache apache deleted a comment from github-actions bot Oct 28, 2020
@apache apache deleted a comment from github-actions bot Oct 28, 2020
@apache apache deleted a comment from github-actions bot Oct 28, 2020
@apache apache deleted a comment from github-actions bot Oct 28, 2020
@github-actions

This comment has been minimized.

@ashb ashb merged commit e802c6c into apache:master Oct 28, 2020
@ashb ashb deleted the scheduler-ha-docs branch October 28, 2020 20:28
michalmisiewicz pushed a commit to michalmisiewicz/airflow that referenced this pull request Oct 30, 2020
szn pushed a commit to szn/airflow that referenced this pull request Nov 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants