Introduced pyproject.toml and moved static metadata from setup.py #1592

pranavrth · 2023-06-20T08:33:12Z

Introduced pyproject.toml
Moved static metadata from setup.py
Removed Python 3.6 support
Updated RELEASE.md

cla-assistant · 2023-08-15T18:16:53Z

All committers have signed the CLA.

emasab

Given we're touching project dependencies we should do a cleanup of them, here's what I suggest

pyproject.toml

emasab · 2024-04-30T13:19:30Z

pyproject.toml

+[project.urls]
+Homepage = "https://github.com/confluentinc/confluent-kafka-python"
+
+[project.optional-dependencies]


To avoid duplicating dependencies we can move all requirements.txt to the project root folder and use them as dynamic dependencies, like:

[project] dynamic = ["dependencies", "optional-dependencies"] [tool.setuptools.dynamic] dependencies = {file = ["requirements.txt"]} optional-dependencies.schemaregistry = { file = ["requirements-schemaregistry.txt"] } optional-dependencies.avro = { file = ["requirements-avro.txt", "requirements-schemaregistry.txt"] }

You cannot reference
-r requirements-schemaregistry.txt in requirements-avro.txt so you can a requirements-dev-install.txt that lists the dependencies for the CI

-r requirements-schemaregistry.txt -r requirements-avro.txt -r requirements-json.txt -r requirements-protobuf.txt -r requirements-dev.txt

We should also change tox.ini to use
pip install .[dev]

deps = -rrequirements-dev-install.txt

and add py311. py312 lacks of the package pkg_resources needed by Trivup

Also replace this in DEVELOPER.md#Generate Documentation

Install doc dependencies: $ pip install .[doc]

You can use requirements-dev-install.txt when scripts are requiring different requirements files

Once we install with .[doc] we can remove this import in docs/conf.py
sys.path[:0] = [os.path.abspath(x) for x in glob('../build/lib.*')]

setup.py

Makefile

MANIFEST.in

pyproject.toml

confluent-cla-assistant · 2024-10-15T22:24:56Z

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

emasab · 2024-10-16T17:01:47Z

requirements/requirements-avro.txt

@@ -1,3 +1,3 @@
 fastavro>=0.23.0


There was a problem with fastavro and 3.7 in versions higher than 1.8.0, but maybe it's been fixed in 1.8.1

Suggested change

fastavro>=0.23.0

fastavro < 1.8.0; python_version == "3.7"

fastavro < 2; python_version > "3.7"

emasab · 2024-10-16T17:06:44Z

requirements/requirements-tests.txt

@@ -1,14 +1,17 @@
+# core test requirements
 urllib3<2.0.0;python_version<="3.7"
 urllib3


Suggested change

urllib3

urllib3 >= 2.0.0,<3; python_version > "3.7"

emasab · 2024-10-16T17:09:24Z

requirements/requirements-tests.txt

+# other requirements
 fastavro<1.8.0;python_version=="3.7"
 fastavro>=1.8.4;python_version>"3.7"
 fastavro
+requests
 avro>=1.11.1,<2
 jsonschema
-protobuf
+protobuf>=3.6.0


These dependencies would be duplicate, please remove them and add a requirements/requirements-tests-install.txt that one can use to install test dependencies plus SR ones.

-r requirements-tests.txt -r requirements-schemaregistry.txt -r requirements-avro.txt -r requirements-protobuf.txt -r requirements-json.txt

emasab · 2024-10-16T17:10:47Z

tests/README.md

+    $ pip install -r requirements/requirements-tests.txt
+    $ python3 -m pip install .


Suggested change

$ pip install -r requirements/requirements-tests.txt

$ python3 -m pip install .

$ python3 -m pip install .[tests]

emasab · 2024-10-16T17:12:59Z

tools/wheels/build-wheels.sh

@@ -8,7 +8,7 @@ this_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"


 # Skip PyPy, Python2, old Python3 versions, musl, and x86 builds.
-export CIBW_SKIP="pp* cp27-* cp35-* *i686 *musllinux* $CIBW_SKIP"
+export CIBW_SKIP="pp* cp27-* cp35-* cp36-* *i686 *musllinux* $CIBW_SKIP"


Below, replace
so that setup.py finds librdkafka.
with
so that cibuildwheel finds librdkafka.

In RELEASE.md there are these lines to change (confluent_kafka.h and pyproject.toml)

Commit these changes with a commit-message containing the version: $ git commit -m "Version v0.11.4rc1" src/confluent_kafka/src/confluent_kafka.h docs/conf.py pyproject.toml

Remember to change the confluent_kafka.c to confluent_kafka.h as it's what was modified just before and needs to be committed.

emasab · 2024-10-16T17:27:24Z

pyproject.toml

+readme = "README.md"
+license = { file = "LICENSE" }
+requires-python = ">=3.7"
+dynamic = ["optional-dependencies","dependencies"]


nit:

Suggested change

dynamic = ["optional-dependencies","dependencies"]

dynamic = ["dependencies", "optional-dependencies"]

emasab · 2024-10-16T17:31:02Z

pyproject.toml

+optional-dependencies.avro = { file = ["requirements/requirements-avro.txt", "requirements/requirements-schemaregistry.txt"] }
+optional-dependencies.json = { file = ["requirements/requirements-json.txt", "requirements/requirements-schemaregistry.txt"] }
+optional-dependencies.protobuf = { file = ["requirements/requirements-protobuf.txt", "requirements/requirements-schemaregistry.txt"] }
+optional-dependencies.dev = { file = ["requirements/requirements-tests.txt", "requirements/requirements-docs.txt", "requirements/requirements-examples.txt"] }


Please add the SR dependencies to dev (for testing) and break these long lines.
As it's not possible to use requirements-tests-install.txt, containing -r, here:

Suggested change

optional-dependencies.dev = { file = ["requirements/requirements-tests.txt", "requirements/requirements-docs.txt", "requirements/requirements-examples.txt"] }

optional-dependencies.dev = { file = [

"requirements/requirements-docs.txt",

"requirements/requirements-examples.txt",

"requirements/requirements-tests.txt",

"requirements/requirements-schemaregistry.txt",

"requirements/requirements-avro.txt",

"requirements/requirements-json.txt",

"requirements/requirements-protobuf.txt"] }

emasab · 2024-10-16T17:32:47Z

pyproject.toml

+optional-dependencies.json = { file = ["requirements/requirements-json.txt", "requirements/requirements-schemaregistry.txt"] }
+optional-dependencies.protobuf = { file = ["requirements/requirements-protobuf.txt", "requirements/requirements-schemaregistry.txt"] }
+optional-dependencies.dev = { file = ["requirements/requirements-tests.txt", "requirements/requirements-docs.txt", "requirements/requirements-examples.txt"] }
+optional-dependencies.docs = { file = ["requirements/requirements-docs.txt"] }


For generating docs Sphinx needs also the SR dependencies.

Suggested change

optional-dependencies.docs = { file = ["requirements/requirements-docs.txt"] }

optional-dependencies.docs = { file = [

"requirements/requirements-docs.txt",

"requirements/requirements-schemaregistry.txt",

"requirements/requirements-avro.txt",

"requirements/requirements-json.txt",

"requirements/requirements-protobuf.txt"] }

Add an all optional dependency

optional-dependencies.all = { file = [ "requirements/requirements-soaktest.txt", "requirements/requirements-docs.txt", "requirements/requirements-examples.txt", "requirements/requirements-tests.txt", "requirements/requirements-schemaregistry.txt", "requirements/requirements-avro.txt", "requirements/requirements-json.txt", "requirements/requirements-protobuf.txt"] }

emasab · 2024-10-16T17:34:07Z

pyproject.toml

+optional-dependencies.protobuf = { file = ["requirements/requirements-protobuf.txt", "requirements/requirements-schemaregistry.txt"] }
+optional-dependencies.dev = { file = ["requirements/requirements-tests.txt", "requirements/requirements-docs.txt", "requirements/requirements-examples.txt"] }
+optional-dependencies.docs = { file = ["requirements/requirements-docs.txt"] }
+optional-dependencies.tests = { file = ["requirements/requirements-tests.txt"] }


Add the SR dependencies here

Suggested change

optional-dependencies.tests = { file = ["requirements/requirements-tests.txt"] }

optional-dependencies.tests = { file = [

"requirements/requirements-tests.txt",

"requirements/requirements-schemaregistry.txt",

"requirements/requirements-avro.txt",

"requirements/requirements-json.txt",

"requirements/requirements-protobuf.txt"

] }

emasab · 2024-10-16T17:36:17Z

tox.ini

@@ -1,5 +1,5 @@
 [tox]
-envlist = flake8,py37,py38,py39,py310
+envlist = flake8,py37,py38,py39,py310,py311,py312


Below instead of installing
pip install . .[avro] .[schema-registry] .[json] .[protobuf]
we can install
pip install .[tests]

deps can be removed

deps = # https://docs.pytest.org/en/latest/changelog.html#id53 -rrequirements/requirement-dev.txt

Please add ,tmp-KafkaCluster/* to the flake8 exclusion list, folder created by trivup

Instead of pip install .[tests], I have added another block for src and added pip install .[src] instead as there are some extra dependencies in tests which are not main dependencies. Refer pyproject.toml.

But here it's a test so it's correct to use tests, if you don't it says that pytest isn't found (you need to remove the .tox folder first).

Could you also remove

setenv = CPPFLAGS=-I{toxinidir}/tmp-build/include LDFLAGS=-L{toxinidir}/tmp-build/lib C_INCLUDE_PATH={toxinidir}/tmp-build/include LD_LIBRARY_PATH={toxinidir}/tmp-build/lib

It overwrites the env variables, for example I've set C_INCLUDE_PATH and LD_LIBRARY_PATH so it uses librdkafka folder instead of installing with make install every time. And this way it doesn't use the defined variables.

emasab

LGTM, Thank Pranav!

pranavrth marked this pull request as ready for review June 23, 2023 06:10

pranavrth requested a review from a team as a code owner June 23, 2023 06:10

pranavrth force-pushed the dev_pyproject-toml branch 2 times, most recently from fea7154 to 665b2f0 Compare October 27, 2023 11:09

pranavrth requested a review from emasab October 27, 2023 11:55

pranavrth force-pushed the dev_pyproject-toml branch from 91878fe to 3265136 Compare April 18, 2024 11:19

emasab requested changes Apr 30, 2024

View reviewed changes

pranavrth force-pushed the dev_pyproject-toml branch from 3265136 to 2118262 Compare May 9, 2024 10:38

pranavrth requested a review from a team as a code owner July 1, 2024 10:08

pranavrth force-pushed the dev_pyproject-toml branch from d955327 to 9bbe5f6 Compare July 1, 2024 10:08

pranavrth added 2 commits October 16, 2024 03:41

Introduced pyproject.toml and moved static metadata from setup.py

a82aa41

Updated version

e656155

airlock-confluentinc bot force-pushed the dev_pyproject-toml branch from 878c308 to e656155 Compare October 15, 2024 22:24

Updated changelog

4c2a898

emasab requested changes Oct 16, 2024

View reviewed changes

pranavrth added 2 commits October 17, 2024 01:53

PR comments

d0f5645

PR comments

7739eb6

emasab approved these changes Oct 18, 2024

View reviewed changes

pranavrth merged commit 42f4487 into master Oct 18, 2024
3 checks passed

pranavrth deleted the dev_pyproject-toml branch October 18, 2024 06:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduced pyproject.toml and moved static metadata from setup.py #1592

Introduced pyproject.toml and moved static metadata from setup.py #1592

pranavrth commented Jun 20, 2023 •

edited

Loading

cla-assistant bot commented Aug 15, 2023 •

edited

Loading

emasab left a comment

emasab Apr 30, 2024

emasab Apr 30, 2024

emasab Apr 30, 2024

emasab Apr 30, 2024

emasab Apr 30, 2024

emasab Apr 30, 2024

confluent-cla-assistant bot commented Oct 15, 2024

emasab Oct 16, 2024

emasab Oct 16, 2024

emasab Oct 16, 2024

emasab Oct 16, 2024

emasab Oct 16, 2024

emasab Oct 16, 2024

emasab Oct 17, 2024

emasab Oct 16, 2024

emasab Oct 16, 2024

emasab Oct 16, 2024

emasab Oct 16, 2024

emasab Oct 16, 2024

emasab Oct 16, 2024

emasab Oct 16, 2024

emasab Oct 16, 2024

pranavrth Oct 16, 2024

emasab Oct 17, 2024

emasab Oct 17, 2024

emasab left a comment

	fastavro>=0.23.0
	fastavro < 1.8.0; python_version == "3.7"
	fastavro < 2; python_version > "3.7"

		$ pip install -r requirements/requirements-tests.txt
		$ python3 -m pip install .

	$ pip install -r requirements/requirements-tests.txt
	$ python3 -m pip install .
	$ python3 -m pip install .[tests]

	dynamic = ["optional-dependencies","dependencies"]
	dynamic = ["dependencies", "optional-dependencies"]

Introduced pyproject.toml and moved static metadata from setup.py #1592

Introduced pyproject.toml and moved static metadata from setup.py #1592

Conversation

pranavrth commented Jun 20, 2023 • edited Loading

cla-assistant bot commented Aug 15, 2023 • edited Loading

emasab left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

confluent-cla-assistant bot commented Oct 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emasab left a comment

Choose a reason for hiding this comment

pranavrth commented Jun 20, 2023 •

edited

Loading

cla-assistant bot commented Aug 15, 2023 •

edited

Loading