Memory leak (?) in Python in version 3.15.0+

[Atheuz]

What version of protobuf and what language are you using?
Version: 3.14.0/v3.15.0
Language: Python

What operating system (Linux, Windows, ...) and version?
Ubuntu Linux

What runtime / compiler are you using (e.g., python version or gcc version)
Python v3.9.5
Built protobuf schema with protoc 3.14.0

What did you do?
Steps to reproduce the behavior:

1. Go to https://github.com/Atheuz/proto_leak and clone it with: git clone https://github.com/Atheuz/proto_leak.git.
2. Go to the repository with cd proto_leak.
3. Create a virtual environment with virtualenv .venv --python=3.9 and activate it with source .venv/bin/activate.
4. Run: pip install -r requirements.txt. (installs protobuf==3.14.0)
5. Run python leak.py
6. Observe:

Memory consumed at the beginning 14.50390625
Memory consumed after parsed 735.421875
Memory consumed after deallocating 16.15625

7. Run: pip install protobuf==3.15.0
8. Run python leak.py
9. Observe:

Memory consumed at the beginning 17.1953125
Memory consumed after parsed 636.6875
Memory consumed after deallocating 536.734375

What did you expect to see
In 3.15.0, I expected it to free basically all memory used for the list of pb items.

What did you see instead?
In 3.15.0, it retains 500+ MB of memory that is not retained in 3.14.0.

[anandolee]

From 3.15.0 the install is using cpp extension by default. 3.14.0's default is pure python.

We have reference leak tests for cpp extension. Not sure why the memory retains so much... Will do more checks

[Atheuz]

From 3.15.0 the install is using cpp extension by default. 3.14.0's default is pure python.

We have reference leak tests for cpp extension. Not sure why the memory retains so much... Will do more checks

When you say "by default", does this mean that you can switch 3.15.0+ to use pure Python when using the package?

EDIT:
You can switch it to the Python implementation by setting an environment variable PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python. This fixes the leak on later versions too, so I expect that you are correct in your suspicion that the cpp extension is causing it:

(PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python python leak.py)
Memory consumed at the beginning 15.6484375
Memory consumed after parsed 736.5234375
Memory consumed after deallocating 17.64453125

[anandolee]

Yes, it can be switched to pure python by set the environment

[haberman]

This issue should be fixed in 4.21.0-rc1.

[Atheuz]

This issue should be fixed in 4.21.0-rc1.

@haberman please clarify. Did you mean 3.21.0-rc1? I don't see a tag for either 4.21.0 or 3.21.0 but 3.21.0 seems more reasonable. What was the commit that fixed this?

[haberman]

Our new release is 4.21.0-rc1: https://pypi.org/project/protobuf/4.21.0rc1/#files

There was not a single commit that fixes this, it is fixed by the C extension rewrite being released in 4.21: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

The release announcement also mentions more about the version numbering: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#versioning