Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Default value for clear_number column not applied via sqlalchemy #34332

Closed
2 tasks done
abhishekbhakat opened this issue Sep 13, 2023 · 0 comments · Fixed by #34344
Closed
2 tasks done

Default value for clear_number column not applied via sqlalchemy #34332

abhishekbhakat opened this issue Sep 13, 2023 · 0 comments · Fixed by #34344
Labels
affected_version:main_branch Issues Reported for main branch area:MetaDB Meta Database related issues. kind:bug This is a clearly a bug

Comments

@abhishekbhakat
Copy link
Contributor

Apache Airflow version

main (development)

What happened

The SQL query generated does not have a default integer as 0 in the migration causing failure.

INFO  [alembic.runtime.migration] Running upgrade 405de8318b3a -> 375a816bbbf4, add new field 'clear_number' to dagrun
Traceback (most recent call last):
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.NotNullViolation: column "clear_number" of relation "dag_run" contains null values


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/bin/airflow", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/airflow/__main__.py", line 59, in main
    args.func(args)
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/airflow/utils/cli.py", line 114, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/airflow/cli/commands/db_command.py", line 129, in migratedb
    db.upgradedb(
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/airflow/utils/session.py", line 79, in wrapper
    return func(*args, session=session, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/airflow/utils/db.py", line 1627, in upgradedb
    command.upgrade(config, revision=to_revision or "heads")
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/alembic/command.py", line 382, in upgrade
    script.run_env()
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/alembic/script/base.py", line 578, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 109, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/airflow/migrations/env.py", line 117, in <module>
    run_migrations_online()
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/airflow/migrations/env.py", line 111, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/alembic/runtime/environment.py", line 922, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/alembic/runtime/migration.py", line 624, in run_migrations
    step.migration_fn(**kw)
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/airflow/migrations/versions/0129_2_8_0_add_clear_number_to_dag_run.py", line 41, in upgrade
    with op.batch_alter_table("dag_run") as batch_op:
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/alembic/operations/base.py", line 375, in batch_alter_table
    impl.flush()
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/alembic/operations/batch.py", line 113, in flush
    fn(*arg, **kw)
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/alembic/ddl/impl.py", line 322, in add_column
    self._exec(base.AddColumn(table_name, column, schema=schema))
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/alembic/ddl/impl.py", line 193, in _exec
    return conn.execute(  # type: ignore[call-overload]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/sqlalchemy/future/engine.py", line 280, in execute
    return self._execute_20(
           ^^^^^^^^^^^^^^^^^
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
    return connection._execute_ddl(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1477, in _execute_ddl
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
    self._handle_dbapi_exception(
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
    util.raise_(
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/Users/abhishekbhakat/Codes/Turbine/my_local_airflow/airflowenv/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) column "clear_number" of relation "dag_run" contains null values

[SQL: ALTER TABLE dag_run ADD COLUMN clear_number INTEGER NOT NULL]
(Background on this error at: https://sqlalche.me/e/14/gkpj)

SQLAlchemy version was 1.4.49.

What you think should happen instead

The query generated should be ALTER TABLE dag_run ADD COLUMN clear_number INTEGER NOT NULL DEFAULT 0;

How to reproduce

Upgraded from airflow 2.7.1 to main-branch and ran airflow db migrate

Operating System

MacOS 13

Versions of Apache Airflow Providers

Not applicable

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@abhishekbhakat abhishekbhakat added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Sep 13, 2023
@abhishekbhakat abhishekbhakat mentioned this issue Sep 13, 2023
Closed
@nathadfield nathadfield added area:MetaDB Meta Database related issues. and removed needs-triage label for new issues that we didn't triage yet area:core labels Sep 13, 2023
@eladkal eladkal added the affected_version:main_branch Issues Reported for main branch label Sep 13, 2023
@sungwy sungwy mentioned this issue Sep 13, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affected_version:main_branch Issues Reported for main branch area:MetaDB Meta Database related issues. kind:bug This is a clearly a bug
Projects
None yet
Milestone
No milestone
3 participants
@nathadfield @abhishekbhakat @eladkal