PyThreadState and PyInterpreterState are not freed #99205
Description
The following test program gains about 10M per second under top, on Python 3.11 only (confirmed for all development stages: 3.11.0a4, 3.11.0b1, 3.11.0rc1, 3.11.0) . 3.10 shows no memory growth.
from threading import Thread
while True:
t1 = Thread()
t1.start()For direct results, here's the same program using the resource module that I just saw used over at #98467:
from threading import Thread
import resource
print("Before", resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, "MB")
i = 0
while True:
i += 1
t1 = Thread()
t1.start()
if i % 10000 == 0:
print(
f"memory after {i} iterations:",
resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024,
"MB",
)The above program under Py 3.11.0 prints:
$ .venv-3.11/bin/python test5.py
Before 8.14453125 MB
memory after 10000 iterations: 11.6875 MB
memory after 20000 iterations: 15.30078125 MB
memory after 30000 iterations: 18.65234375 MB
memory after 40000 iterations: 22.26171875 MB
memory after 50000 iterations: 25.87109375 MB
memory after 60000 iterations: 29.22265625 MB
memory after 70000 iterations: 32.83203125 MB
memory after 80000 iterations: 36.44140625 MB
memory after 90000 iterations: 39.78125 MB
memory after 100000 iterations: 43.390625 MB
...
under Python 3.10 it prints:
$ .venv-3.10/bin/python test5.py
Before 9.25 MB
memory after 10000 iterations: 9.46484375 MB
memory after 20000 iterations: 9.47265625 MB
memory after 30000 iterations: 9.47265625 MB
memory after 40000 iterations: 9.47265625 MB
memory after 50000 iterations: 9.47265625 MB
memory after 60000 iterations: 9.47265625 MB
memory after 70000 iterations: 9.47265625 MB
memory after 80000 iterations: 9.47265625 MB
the issue looks extremely similar to another one we just fixed in greenlet, over at python-greenlet/greenlet#328, although this one is much more surprising. Issue #98467 might also be related.
Environment:
$ uname -a
Linux photon3 5.17.14-200.fc35.x86_64 #1 SMP PREEMPT Thu Jun 9 14:02:42 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ python3
Python 3.11.0 (main, Nov 5 2022, 23:27:22) [GCC 11.3.1 20220421 (Red Hat 11.3.1-3)] on linux