Move the database configuration to a new section

[gitstart-airflow]

closes: 15930

This PR moves the following configurations from [core] to the new [database] section.

• sql_alchemy_conn
• sql_engine_encoding
• sql_engine_collation_for_ids
• sql_alchemy_pool_enabled
• sql_alchemy_pool_size
• sql_alchemy_max_overflow
• sql_alchemy_pool_recycle
• sql_alchemy_pool_pre_ping
• sql_alchemy_schema
• sql_alchemy_connect_args
• load_default_connections
• max_db_retries

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[potiuk]

This one will need much more. It's not enough to move the sections. And (unless I missed it) the change should handle deprecation - this is a very intrusive change and whoever has the current configurationm (either via config or environment variable) it should continue to work but raise a deprecation warning.

[gitstart-airflow]

Thanks @potiuk . Will make the necessary change.

[potiuk]

Information that they can still be accessed via "core" section but will raise a deprecation warning would be useful to add here.

[potiuk]

This is NIT. And it is good without it - feel free ot add it if you want though.

[potiuk]

Cool. One nit

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

Need rebase unfortunately.

[mik-laj]

Suggested change

	- name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
	valueFrom:
	- name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
	valueFrom:

[mik-laj]

This should help.

[boring-cyborg]

Awesome work, congrats on your first merged pull request!

[pingzh]

@gitstart-airflow it looks like this pr has caused this bug #23679

[pingzh]

@potiuk i am not quite sure about the expected behavior here for deprecating section.

airflow/airflow/configuration.py

Lines 518 to 528 in a7bccaa

	def _get_option_from_config_file(self, deprecated_key, deprecated_section, key, kwargs, section):
	# ...then the config file
	if super().has_option(section, key):
	# Use the parent's methods to get the actual config here to be able to
	# separate the config from default config.
	return expand_env_var(super().get(section, key, **kwargs))
	if deprecated_section:
	if super().has_option(deprecated_section, deprecated_key):
	self._warn_deprecate(section, key, deprecated_section, deprecated_key)
	return expand_env_var(super().get(deprecated_section, deprecated_key, **kwargs))
	return None

for the deprecation_section and key, for the above ^^ method, for database, sql_alchemy_conn, should it use the value from database.sql_alchemy_conn or core.sql_alchemy_conn?

the current behavior for my case #23679 (comment) is airflow tasks run --local process uses the value from core.sql_alchemy_conn, but airflow tasks run --raw process uses the value from database.sql_alchemy_conn

[potiuk]

Yes. I looked at it already and I could confirm that conf.as_dict() will work wrongly in this case (and this is the root cause). Currently when you run conf.as_dict(), the "database" entries will get the "defaults" (sqlite://) for MySQL and "core" will get the one that is set in the config. Which will effective replace the configuration specified in "defaults" when --raw is used.

I am not the author of this part, but after looking there I think simply conf.as_dict has not been written with such deprecations in mind (and likely some users could have some problems before because of other deprecations).

I think - "logically" speaking I think when the "new" value has a corresponding non-default "deprecated" value in the conf - it just should not be exported by the "as_dict()" (when it is taken from default) - but otherwise it should. i am looking at fixing it in an upcoming pr - but I have to learn how the system works first.

[potiuk]

If you want @pingzh you can try to learn it too, and come up with a good solution - maybe we can eventually compare ours and choose the better one.

[pingzh]

If you want @pingzh you can try to learn it too, and come up with a good solution - maybe we can eventually compare ours and choose the better one.

thanks @potiuk i definitely love to learn more about it. will keep you posted.

[potiuk]

thanks @potiuk i definitely love to learn more about it. will keep you posted.

The typing in #23716 would definitely make it easier :D. I've learned quite a lot while adding it