diff --git a/docs/apache-airflow/migrations-ref.rst b/docs/apache-airflow/migrations-ref.rst index 4b24e943ccd8c..1f81e8cbb2acc 100644 --- a/docs/apache-airflow/migrations-ref.rst +++ b/docs/apache-airflow/migrations-ref.rst @@ -24,248 +24,250 @@ Here's the list of all the Database Migrations that are executed via when you ru .. All table elements are scraped from migration files .. Beginning of auto-generated table -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``1de7bc13c950`` (head) | ``b1b348e02d07`` | ``2.3.1`` | Add index for ``event`` column in ``log`` table. | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``b1b348e02d07`` | ``75d5ed6c2b43`` | ``2.3.0`` | Update dag.default_view to grid | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``75d5ed6c2b43`` | ``909884dea523`` | ``2.3.0`` | Add map_index to Log. | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``909884dea523`` | ``48925b2719cb`` | ``2.3.0`` | Update migration for FAB tables to add missing constraints | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``48925b2719cb`` | ``4eaab2fe6582`` | ``2.3.0`` | Add map_index to TaskFail | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``4eaab2fe6582`` | ``c97c2ab6aa23`` | ``2.3.0`` | Migrate RTIF to use run_id and map_index | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``c97c2ab6aa23`` | ``c306b5b5ae4a`` | ``2.3.0`` | add callback request table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``c306b5b5ae4a`` | ``a3bcd0914482`` | ``2.3.0`` | Switch XCom table to use ``run_id`` and add ``map_index``. | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``a3bcd0914482`` | ``e655c0453f75`` | ``2.3.0`` | add data_compressed to serialized_dag | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``e655c0453f75`` | ``f9da662e7089`` | ``2.3.0`` | Add ``map_index`` column to TaskInstance to identify task- | -| | | | mapping, and a ``task_map`` table to track mapping values | -| | | | from XCom. | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``f9da662e7089`` | ``786e3737b18f`` | ``2.3.0`` | Add ``LogTemplate`` table to track changes to config values | -| | | | ``log_filename_template`` | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``786e3737b18f`` | ``5e3ec427fdd3`` | ``2.3.0`` | Add ``timetable_description`` column to DagModel for UI. | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``5e3ec427fdd3`` | ``587bdf053233`` | ``2.3.0`` | Increase length of email and username in ``ab_user`` and | -| | | | ``ab_register_user`` table to ``256`` characters | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``587bdf053233`` | ``c381b21cb7e4`` | ``2.2.4`` | Add index for ``dag_id`` column in ``job`` table. | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``c381b21cb7e4`` | ``be2bfac3da23`` | ``2.2.4`` | Create a ``session`` table to store web session data | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``be2bfac3da23`` | ``7b2661a43ba3`` | ``2.2.3`` | Add has_import_errors column to DagModel | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``7b2661a43ba3`` | ``142555e44c17`` | ``2.2.0`` | Change ``TaskInstance`` and ``TaskReschedule`` tables from | -| | | | execution_date to run_id. | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``142555e44c17`` | ``54bebd308c5f`` | ``2.2.0`` | Add data_interval_[start|end] to DagModel and DagRun. | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``54bebd308c5f`` | ``30867afad44a`` | ``2.2.0`` | Adds ``trigger`` table and deferrable operator columns to | -| | | | task instance | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``30867afad44a`` | ``e9304a3141f0`` | ``2.2.0`` | Rename ``concurrency`` column in ``dag`` table to`` | -| | | | max_active_tasks`` | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``e9304a3141f0`` | ``83f031fd9f1c`` | ``2.2.0`` | Make XCom primary key columns non-nullable | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``83f031fd9f1c`` | ``ccde3e26fe78`` | ``2.2.0`` | Improve MSSQL compatibility | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``ccde3e26fe78`` | ``092435bf5d12`` | ``2.1.4`` | Add index on state, dag_id for queued ``dagrun`` | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``092435bf5d12`` | ``97cdd93827b8`` | ``2.1.4`` | Add ``max_active_runs`` column to ``dag_model`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``97cdd93827b8`` | ``a13f7613ad25`` | ``2.1.3`` | Add ``queued_at`` column in ``dag_run`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``a13f7613ad25`` | ``e165e7455d70`` | ``2.1.0`` | Resource based permissions for default ``Flask-AppBuilder`` | -| | | | views | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``e165e7455d70`` | ``90d1635d7b86`` | ``2.1.0`` | Add description field to ``Variable`` model | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``90d1635d7b86`` | ``2e42bb497a22`` | ``2.1.0`` | Increase maximum length of pool name in ``task_instance`` | -| | | | table to ``256`` characters | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``2e42bb497a22`` | ``8646922c8a04`` | ``2.0.2`` | Rename ``last_scheduler_run`` column in ``DAG`` table to | -| | | | ``last_parsed_time`` | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``8646922c8a04`` | ``449b4072c2da`` | ``2.0.2`` | Change default ``pool_slots`` to ``1`` | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``449b4072c2da`` | ``82b7c48c147f`` | ``2.0.2`` | Increase size of ``connection.extra`` field to handle | -| | | | multiple RSA keys | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``82b7c48c147f`` | ``e959f08ac86c`` | ``2.0.1`` | Remove ``can_read`` permission on config resource for | -| | | | ``User`` and ``Viewer`` role | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``e959f08ac86c`` | ``64a7d6477aae`` | ``2.0.0`` | Change field in ``DagCode`` to ``MEDIUMTEXT`` for MySql | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``64a7d6477aae`` | ``61ec73d9401f`` | ``2.0.0`` | Fix description field in ``connection`` to be ``text`` | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``61ec73d9401f`` | ``2c6edca13270`` | ``2.0.0`` | Add description field to ``connection`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``2c6edca13270`` | ``849da589634d`` | ``2.0.0`` | Resource based permissions. | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``849da589634d`` | ``45ba3f1493b9`` | ``2.0.0`` | Prefix DAG permissions. | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``45ba3f1493b9`` | ``364159666cbd`` | ``2.0.0`` | add-k8s-yaml-to-rendered-templates | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``364159666cbd`` | ``52d53670a240`` | ``2.0.0`` | Add ``creating_job_id`` to ``DagRun`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``52d53670a240`` | ``98271e7606e2`` | ``2.0.0`` | fix_mssql_exec_date_rendered_task_instance_fields_for_MSSQL | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``98271e7606e2`` | ``bef4f3d11e8b`` | ``2.0.0`` | Add ``scheduling_decision`` to ``DagRun`` and ``DAG`` | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``bef4f3d11e8b`` | ``e1a11ece99cc`` | ``2.0.0`` | Drop ``KubeResourceVersion`` and ``KubeWorkerId`` | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``e1a11ece99cc`` | ``b247b1e3d1ed`` | ``2.0.0`` | Add external executor ID to TI | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``b247b1e3d1ed`` | ``e38be357a868`` | ``2.0.0`` | Add queued by Job ID to TI | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``e38be357a868`` | ``8d48763f6d53`` | ``2.0.0`` | Add ``sensor_instance`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``8d48763f6d53`` | ``8f966b9c467a`` | ``2.0.0`` | Add unique constraint to ``conn_id`` | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``8f966b9c467a`` | ``3c20cacc0044`` | ``2.0.0`` | Set ``conn_type`` as non-nullable | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``3c20cacc0044`` | ``b25a55525161`` | ``2.0.0`` | Add ``run_type`` column in ``dag_run`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``b25a55525161`` | ``bbf4a7ad0465`` | ``2.0.0`` | Increase length of pool name | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``bbf4a7ad0465`` | ``cf5dc11e79ad`` | ``2.0.0`` | Remove id column from xcom | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``cf5dc11e79ad`` | ``03afc6b6f902`` | ``2.0.0`` | Drop ``user`` and ``chart`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``03afc6b6f902`` | ``92c57b58940d`` | ``1.10.13`` | Increase length of ``Flask-AppBuilder`` | -| | | | ``ab_view_menu.name`` column | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``92c57b58940d`` | ``da3f683c3a5a`` | ``1.10.13`` | Create FAB Tables | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``da3f683c3a5a`` | ``a66efa278eea`` | ``1.10.12`` | Add ``dag_hash`` Column to ``serialized_dag`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``a66efa278eea`` | ``952da73b5eff`` | ``1.10.11`` | Add Precision to ``execution_date`` in | -| | | | ``RenderedTaskInstanceFields`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``952da73b5eff`` | ``852ae6c715af`` | ``1.10.10`` | Add ``dag_code`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``852ae6c715af`` | ``a4c2fd67d16b`` | ``1.10.10`` | Add ``RenderedTaskInstanceFields`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``a4c2fd67d16b`` | ``7939bcff74ba`` | ``1.10.10`` | Add ``pool_slots`` field to ``task_instance`` | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``7939bcff74ba`` | ``fe461863935f`` | ``1.10.8`` | Add ``DagTags`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``fe461863935f`` | ``08364691d074`` | ``1.10.7`` | Increase length for connection password | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``08364691d074`` (merge_point) | ``a56c9515abdc``, | ``1.10.7`` | Straighten out the migrations | -| | ``004c1210f153``, | | | -| | ``74effc47d867``, | | | -| | ``b3b105409875`` | | | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``a56c9515abdc`` | ``c8ffec048a3b`` | ``1.10.3`` | Remove ``dag_stat`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``004c1210f153`` | ``939bb1e647c8`` | ``1.10.4`` | Increase queue name size limit | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``74effc47d867`` | ``6e96a59344a4`` | ``1.10.5`` | change datetime to datetime2(6) on MSSQL tables | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``b3b105409875`` | ``d38e04c12aa2`` | ``1.10.7`` | Add ``root_dag_id`` to ``DAG`` | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``d38e04c12aa2`` | ``6e96a59344a4`` | ``1.10.7`` | Add ``serialized_dag`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``6e96a59344a4`` (branch_point) | ``939bb1e647c8`` | ``1.10.4`` | Make ``TaskInstance.pool`` not nullable | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``939bb1e647c8`` (branch_point) | ``dd4ecb8fbee3`` | ``1.10.3`` | task reschedule foreign key on cascade delete | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``dd4ecb8fbee3`` | ``c8ffec048a3b`` | ``1.10.3`` | Add schedule interval to dag | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``c8ffec048a3b`` (branch_point) | ``41f5f12752f8`` | ``1.10.3`` | Add ``description`` and ``default_view`` column to ``dag`` | -| | | | table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``41f5f12752f8`` | ``03bc53e68815`` | ``1.10.2`` | Add superuser field | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``03bc53e68815`` (merge_point) | ``0a2a5b66e19d``, | ``1.10.2`` | Merge migrations Heads | -| | ``bf00311e1990`` | | | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``0a2a5b66e19d`` | ``9635ae0956e7`` | ``1.10.2`` | Add ``task_reschedule`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``bf00311e1990`` | ``dd25f486b8ea`` | ``1.10.2`` | Add index to ``task_instance`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``dd25f486b8ea`` | ``9635ae0956e7`` | ``1.10.2`` | Add index on ``log`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``9635ae0956e7`` (branch_point) | ``856955da8476`` | ``1.10.0`` | Create index on ``task_fail`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``856955da8476`` | ``f23433877c24`` | ``1.10.0`` | Fix Sqlite foreign key | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``f23433877c24`` | ``05f30312d566`` | ``1.10.0`` | Fix MySQL not null constraint | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``05f30312d566`` (merge_point) | ``86770d1215c0``, | ``1.10.0`` | Merge migrations Heads | -| | ``0e2a74e0fc9f`` | | | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``86770d1215c0`` | ``27c6a30d7c24`` | ``1.10.0`` | Add kubernetes scheduler uniqueness | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``27c6a30d7c24`` | ``33ae817a1ff4`` | ``1.10.0`` | Add ``executor_config`` column to ``task_instance`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``33ae817a1ff4`` | ``d2ae31099d61`` | ``1.10.0`` | Add Kubernetes resource check-pointing | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``0e2a74e0fc9f`` | ``d2ae31099d61`` | ``1.10.0`` | Add time zone awareness | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``d2ae31099d61`` (branch_point) | ``947454bf1dff`` | ``1.8.2`` | Increase text size for MySQL (not relevant for other DBs' | -| | | | text types) | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``947454bf1dff`` | ``bdaa763e6c56`` | ``1.8.2`` | Create index on ``job_id`` column in ``task_instance`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``bdaa763e6c56`` | ``cc1e65623dc7`` | ``1.8.2`` | Make xcom value column a large binary | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``cc1e65623dc7`` | ``127d2bf2dfa7`` | ``1.8.2`` | Add ``max_tries`` column to ``task_instance`` | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``127d2bf2dfa7`` | ``5e7d17757c7a`` | ``1.7.1.3`` | Add ``dag_id``/``state`` index on ``dag_run`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``5e7d17757c7a`` | ``8504051e801b`` | ``1.7.1.3`` | Add ``pid`` field to ``TaskInstance`` | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``8504051e801b`` | ``4addfa1236f1`` | ``1.7.1.3`` | Add indices on ``xcom`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``4addfa1236f1`` | ``f2ca10b85618`` | ``1.7.1.3`` | Add fractional seconds to MySQL tables | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``f2ca10b85618`` | ``64de9cddf6c9`` | ``1.7.1.3`` | Add ``dag_stats`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``64de9cddf6c9`` | ``211e584da130`` | ``1.7.1.3`` | Add ``task_fail`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``211e584da130`` | ``2e82aab8ef20`` | ``1.7.1.3`` | Add TI state index | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``2e82aab8ef20`` | ``1968acfc09e3`` | ``1.7.1`` | Rename user table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``1968acfc09e3`` | ``bba5a7cfc896`` | ``1.7.0`` | Add ``is_encrypted`` column to variable table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``bba5a7cfc896`` | ``bbc73705a13e`` | ``1.7.0`` | Add a column to track the encryption state of the 'Extra' | -| | | | field in connection | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``bbc73705a13e`` | ``4446e08588`` | ``1.7.0`` | Add ``notification_sent`` column to ``sla_miss`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``4446e08588`` | ``561833c1c74b`` | ``1.6.2`` | Add ``start_date`` and ``end_date`` in ``dag_run`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``561833c1c74b`` | ``40e67319e3a9`` | ``1.6.2`` | Add ``password`` column to ``user`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``40e67319e3a9`` | ``2e541a1dcfed`` | ``1.6.0`` | Add ``conf`` column in ``dag_run`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``2e541a1dcfed`` | ``1b38cef5b76e`` | ``1.6.0`` | Change ``task_instance.task_duration`` type to ``FLOAT`` | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``1b38cef5b76e`` | ``502898887f84`` | ``1.6.0`` | Add ``dag_run`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``502898887f84`` | ``52d714495f0`` | ``1.6.0`` | Adding ``extra`` column to ``Log`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``52d714495f0`` | ``338e90f54d61`` | ``1.5.2`` | Add indices in ``job`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``338e90f54d61`` | ``13eb55f81627`` | ``1.5.0`` | Add ``operator`` and ``queued_dttm`` to ``task_instance`` | -| | | | table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``13eb55f81627`` | ``1507a7289a2f`` | ``1.5.0`` | Maintain history for compatibility with earlier migrations | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``1507a7289a2f`` | ``e3a246e0dc1`` | ``1.5.0`` | Add ``is_encrypted`` column in ``connection`` table | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ -| ``e3a246e0dc1`` (base) | ``None`` | ``1.5.0`` | current schema | -+---------------------------------+-------------------+-------------+--------------------------------------------------------------+ ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| Revision ID | Revises ID | Airflow Version | Description | ++=================================+===================+===================+==============================================================+ +| ``1de7bc13c950`` (head) | ``b1b348e02d07`` | ``2.3.1`` | Add index for ``event`` column in ``log`` table. | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``b1b348e02d07`` | ``75d5ed6c2b43`` | ``2.3.0`` | Update dag.default_view to grid | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``75d5ed6c2b43`` | ``909884dea523`` | ``2.3.0`` | Add map_index to Log. | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``909884dea523`` | ``48925b2719cb`` | ``2.3.0`` | Update migration for FAB tables to add missing constraints | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``48925b2719cb`` | ``4eaab2fe6582`` | ``2.3.0`` | Add map_index to TaskFail | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``4eaab2fe6582`` | ``c97c2ab6aa23`` | ``2.3.0`` | Migrate RTIF to use run_id and map_index | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``c97c2ab6aa23`` | ``c306b5b5ae4a`` | ``2.3.0`` | add callback request table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``c306b5b5ae4a`` | ``a3bcd0914482`` | ``2.3.0`` | Switch XCom table to use ``run_id`` and add ``map_index``. | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``a3bcd0914482`` | ``e655c0453f75`` | ``2.3.0`` | add data_compressed to serialized_dag | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``e655c0453f75`` | ``f9da662e7089`` | ``2.3.0`` | Add ``map_index`` column to TaskInstance to identify task- | +| | | | mapping, and a ``task_map`` table to track mapping values | +| | | | from XCom. | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``f9da662e7089`` | ``786e3737b18f`` | ``2.3.0`` | Add ``LogTemplate`` table to track changes to config values | +| | | | ``log_filename_template`` | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``786e3737b18f`` | ``5e3ec427fdd3`` | ``2.3.0`` | Add ``timetable_description`` column to DagModel for UI. | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``5e3ec427fdd3`` | ``587bdf053233`` | ``2.3.0`` | Increase length of email and username in ``ab_user`` and | +| | | | ``ab_register_user`` table to ``256`` characters | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``587bdf053233`` | ``c381b21cb7e4`` | ``2.2.4`` | Add index for ``dag_id`` column in ``job`` table. | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``c381b21cb7e4`` | ``be2bfac3da23`` | ``2.2.4`` | Create a ``session`` table to store web session data | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``be2bfac3da23`` | ``7b2661a43ba3`` | ``2.2.3`` | Add has_import_errors column to DagModel | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``7b2661a43ba3`` | ``142555e44c17`` | ``2.2.0`` | Change ``TaskInstance`` and ``TaskReschedule`` tables from | +| | | | execution_date to run_id. | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``142555e44c17`` | ``54bebd308c5f`` | ``2.2.0`` | Add data_interval_[start|end] to DagModel and DagRun. | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``54bebd308c5f`` | ``30867afad44a`` | ``2.2.0`` | Adds ``trigger`` table and deferrable operator columns to | +| | | | task instance | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``30867afad44a`` | ``e9304a3141f0`` | ``2.2.0`` | Rename ``concurrency`` column in ``dag`` table to`` | +| | | | max_active_tasks`` | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``e9304a3141f0`` | ``83f031fd9f1c`` | ``2.2.0`` | Make XCom primary key columns non-nullable | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``83f031fd9f1c`` | ``ccde3e26fe78`` | ``2.2.0`` | Improve MSSQL compatibility | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``ccde3e26fe78`` | ``092435bf5d12`` | ``2.1.4`` | Add index on state, dag_id for queued ``dagrun`` | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``092435bf5d12`` | ``97cdd93827b8`` | ``2.1.4`` | Add ``max_active_runs`` column to ``dag_model`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``97cdd93827b8`` | ``a13f7613ad25`` | ``2.1.3`` | Add ``queued_at`` column in ``dag_run`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``a13f7613ad25`` | ``e165e7455d70`` | ``2.1.0`` | Resource based permissions for default ``Flask-AppBuilder`` | +| | | | views | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``e165e7455d70`` | ``90d1635d7b86`` | ``2.1.0`` | Add description field to ``Variable`` model | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``90d1635d7b86`` | ``2e42bb497a22`` | ``2.1.0`` | Increase maximum length of pool name in ``task_instance`` | +| | | | table to ``256`` characters | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``2e42bb497a22`` | ``8646922c8a04`` | ``2.0.2`` | Rename ``last_scheduler_run`` column in ``DAG`` table to | +| | | | ``last_parsed_time`` | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``8646922c8a04`` | ``449b4072c2da`` | ``2.0.2`` | Change default ``pool_slots`` to ``1`` | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``449b4072c2da`` | ``82b7c48c147f`` | ``2.0.2`` | Increase size of ``connection.extra`` field to handle | +| | | | multiple RSA keys | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``82b7c48c147f`` | ``e959f08ac86c`` | ``2.0.1`` | Remove ``can_read`` permission on config resource for | +| | | | ``User`` and ``Viewer`` role | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``e959f08ac86c`` | ``64a7d6477aae`` | ``2.0.0`` | Change field in ``DagCode`` to ``MEDIUMTEXT`` for MySql | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``64a7d6477aae`` | ``61ec73d9401f`` | ``2.0.0`` | Fix description field in ``connection`` to be ``text`` | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``61ec73d9401f`` | ``2c6edca13270`` | ``2.0.0`` | Add description field to ``connection`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``2c6edca13270`` | ``849da589634d`` | ``2.0.0`` | Resource based permissions. | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``849da589634d`` | ``45ba3f1493b9`` | ``2.0.0`` | Prefix DAG permissions. | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``45ba3f1493b9`` | ``364159666cbd`` | ``2.0.0`` | add-k8s-yaml-to-rendered-templates | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``364159666cbd`` | ``52d53670a240`` | ``2.0.0`` | Add ``creating_job_id`` to ``DagRun`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``52d53670a240`` | ``98271e7606e2`` | ``2.0.0`` | fix_mssql_exec_date_rendered_task_instance_fields_for_MSSQL | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``98271e7606e2`` | ``bef4f3d11e8b`` | ``2.0.0`` | Add ``scheduling_decision`` to ``DagRun`` and ``DAG`` | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``bef4f3d11e8b`` | ``e1a11ece99cc`` | ``2.0.0`` | Drop ``KubeResourceVersion`` and ``KubeWorkerId`` | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``e1a11ece99cc`` | ``b247b1e3d1ed`` | ``2.0.0`` | Add external executor ID to TI | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``b247b1e3d1ed`` | ``e38be357a868`` | ``2.0.0`` | Add queued by Job ID to TI | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``e38be357a868`` | ``8d48763f6d53`` | ``2.0.0`` | Add ``sensor_instance`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``8d48763f6d53`` | ``8f966b9c467a`` | ``2.0.0`` | Add unique constraint to ``conn_id`` | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``8f966b9c467a`` | ``3c20cacc0044`` | ``2.0.0`` | Set ``conn_type`` as non-nullable | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``3c20cacc0044`` | ``b25a55525161`` | ``2.0.0`` | Add ``run_type`` column in ``dag_run`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``b25a55525161`` | ``bbf4a7ad0465`` | ``2.0.0`` | Increase length of pool name | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``bbf4a7ad0465`` | ``cf5dc11e79ad`` | ``2.0.0`` | Remove id column from xcom | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``cf5dc11e79ad`` | ``03afc6b6f902`` | ``2.0.0`` | Drop ``user`` and ``chart`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``03afc6b6f902`` | ``92c57b58940d`` | ``1.10.13`` | Increase length of ``Flask-AppBuilder`` | +| | | | ``ab_view_menu.name`` column | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``92c57b58940d`` | ``da3f683c3a5a`` | ``1.10.13`` | Create FAB Tables | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``da3f683c3a5a`` | ``a66efa278eea`` | ``1.10.12`` | Add ``dag_hash`` Column to ``serialized_dag`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``a66efa278eea`` | ``952da73b5eff`` | ``1.10.11`` | Add Precision to ``execution_date`` in | +| | | | ``RenderedTaskInstanceFields`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``952da73b5eff`` | ``852ae6c715af`` | ``1.10.10`` | Add ``dag_code`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``852ae6c715af`` | ``a4c2fd67d16b`` | ``1.10.10`` | Add ``RenderedTaskInstanceFields`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``a4c2fd67d16b`` | ``7939bcff74ba`` | ``1.10.10`` | Add ``pool_slots`` field to ``task_instance`` | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``7939bcff74ba`` | ``fe461863935f`` | ``1.10.8`` | Add ``DagTags`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``fe461863935f`` | ``08364691d074`` | ``1.10.7`` | Increase length for connection password | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``08364691d074`` (merge_point) | ``a56c9515abdc``, | ``1.10.7`` | Straighten out the migrations | +| | ``004c1210f153``, | | | +| | ``74effc47d867``, | | | +| | ``b3b105409875`` | | | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``a56c9515abdc`` | ``c8ffec048a3b`` | ``1.10.3`` | Remove ``dag_stat`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``004c1210f153`` | ``939bb1e647c8`` | ``1.10.4`` | Increase queue name size limit | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``74effc47d867`` | ``6e96a59344a4`` | ``1.10.5`` | change datetime to datetime2(6) on MSSQL tables | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``b3b105409875`` | ``d38e04c12aa2`` | ``1.10.7`` | Add ``root_dag_id`` to ``DAG`` | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``d38e04c12aa2`` | ``6e96a59344a4`` | ``1.10.7`` | Add ``serialized_dag`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``6e96a59344a4`` (branch_point) | ``939bb1e647c8`` | ``1.10.4`` | Make ``TaskInstance.pool`` not nullable | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``939bb1e647c8`` (branch_point) | ``dd4ecb8fbee3`` | ``1.10.3`` | task reschedule foreign key on cascade delete | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``dd4ecb8fbee3`` | ``c8ffec048a3b`` | ``1.10.3`` | Add schedule interval to dag | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``c8ffec048a3b`` (branch_point) | ``41f5f12752f8`` | ``1.10.3`` | Add ``description`` and ``default_view`` column to ``dag`` | +| | | | table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``41f5f12752f8`` | ``03bc53e68815`` | ``1.10.2`` | Add superuser field | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``03bc53e68815`` (merge_point) | ``0a2a5b66e19d``, | ``1.10.2`` | Merge migrations Heads | +| | ``bf00311e1990`` | | | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``0a2a5b66e19d`` | ``9635ae0956e7`` | ``1.10.2`` | Add ``task_reschedule`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``bf00311e1990`` | ``dd25f486b8ea`` | ``1.10.2`` | Add index to ``task_instance`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``dd25f486b8ea`` | ``9635ae0956e7`` | ``1.10.2`` | Add index on ``log`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``9635ae0956e7`` (branch_point) | ``856955da8476`` | ``1.10.0`` | Create index on ``task_fail`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``856955da8476`` | ``f23433877c24`` | ``1.10.0`` | Fix Sqlite foreign key | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``f23433877c24`` | ``05f30312d566`` | ``1.10.0`` | Fix MySQL not null constraint | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``05f30312d566`` (merge_point) | ``86770d1215c0``, | ``1.10.0`` | Merge migrations Heads | +| | ``0e2a74e0fc9f`` | | | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``86770d1215c0`` | ``27c6a30d7c24`` | ``1.10.0`` | Add kubernetes scheduler uniqueness | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``27c6a30d7c24`` | ``33ae817a1ff4`` | ``1.10.0`` | Add ``executor_config`` column to ``task_instance`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``33ae817a1ff4`` | ``d2ae31099d61`` | ``1.10.0`` | Add Kubernetes resource check-pointing | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``0e2a74e0fc9f`` | ``d2ae31099d61`` | ``1.10.0`` | Add time zone awareness | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``d2ae31099d61`` (branch_point) | ``947454bf1dff`` | ``1.8.2`` | Increase text size for MySQL (not relevant for other DBs' | +| | | | text types) | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``947454bf1dff`` | ``bdaa763e6c56`` | ``1.8.2`` | Create index on ``job_id`` column in ``task_instance`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``bdaa763e6c56`` | ``cc1e65623dc7`` | ``1.8.2`` | Make xcom value column a large binary | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``cc1e65623dc7`` | ``127d2bf2dfa7`` | ``1.8.2`` | Add ``max_tries`` column to ``task_instance`` | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``127d2bf2dfa7`` | ``5e7d17757c7a`` | ``1.7.1.3`` | Add ``dag_id``/``state`` index on ``dag_run`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``5e7d17757c7a`` | ``8504051e801b`` | ``1.7.1.3`` | Add ``pid`` field to ``TaskInstance`` | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``8504051e801b`` | ``4addfa1236f1`` | ``1.7.1.3`` | Add indices on ``xcom`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``4addfa1236f1`` | ``f2ca10b85618`` | ``1.7.1.3`` | Add fractional seconds to MySQL tables | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``f2ca10b85618`` | ``64de9cddf6c9`` | ``1.7.1.3`` | Add ``dag_stats`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``64de9cddf6c9`` | ``211e584da130`` | ``1.7.1.3`` | Add ``task_fail`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``211e584da130`` | ``2e82aab8ef20`` | ``1.7.1.3`` | Add TI state index | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``2e82aab8ef20`` | ``1968acfc09e3`` | ``1.7.1`` | Rename user table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``1968acfc09e3`` | ``bba5a7cfc896`` | ``1.7.0`` | Add ``is_encrypted`` column to variable table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``bba5a7cfc896`` | ``bbc73705a13e`` | ``1.7.0`` | Add a column to track the encryption state of the 'Extra' | +| | | | field in connection | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``bbc73705a13e`` | ``4446e08588`` | ``1.7.0`` | Add ``notification_sent`` column to ``sla_miss`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``4446e08588`` | ``561833c1c74b`` | ``1.6.2`` | Add ``start_date`` and ``end_date`` in ``dag_run`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``561833c1c74b`` | ``40e67319e3a9`` | ``1.6.2`` | Add ``password`` column to ``user`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``40e67319e3a9`` | ``2e541a1dcfed`` | ``1.6.0`` | Add ``conf`` column in ``dag_run`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``2e541a1dcfed`` | ``1b38cef5b76e`` | ``1.6.0`` | Change ``task_instance.task_duration`` type to ``FLOAT`` | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``1b38cef5b76e`` | ``502898887f84`` | ``1.6.0`` | Add ``dag_run`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``502898887f84`` | ``52d714495f0`` | ``1.6.0`` | Adding ``extra`` column to ``Log`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``52d714495f0`` | ``338e90f54d61`` | ``1.5.2`` | Add indices in ``job`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``338e90f54d61`` | ``13eb55f81627`` | ``1.5.0`` | Add ``operator`` and ``queued_dttm`` to ``task_instance`` | +| | | | table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``13eb55f81627`` | ``1507a7289a2f`` | ``1.5.0`` | Maintain history for compatibility with earlier migrations | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``1507a7289a2f`` | ``e3a246e0dc1`` | ``1.5.0`` | Add ``is_encrypted`` column in ``connection`` table | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ +| ``e3a246e0dc1`` (base) | ``None`` | ``1.5.0`` | current schema | ++---------------------------------+-------------------+-------------------+--------------------------------------------------------------+ .. End of auto-generated table diff --git a/scripts/in_container/run_migration_reference.py b/scripts/in_container/run_migration_reference.py index 3bc656b11db27..cc05408c2aca8 100755 --- a/scripts/in_container/run_migration_reference.py +++ b/scripts/in_container/run_migration_reference.py @@ -59,6 +59,12 @@ def update_doc(file, data): end=" .. End of auto-generated table\n", replacement_text="\n" + tabulate( + headers={ + "revision": "Revision ID", + "down_revision": "Revises ID", + "version": "Airflow Version", + "description": "Description", + }, tabular_data=data, tablefmt='grid', stralign="left",