pFad - Phone/Frame/Anonymizer/Declutterfier! Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

URL: http://github.com/pythonnet/pythonnet/pull/2721/files

="stylesheet" href="https://github.githubassets.com/assets/code-34e10031edc15af1.css" /> Python 3.14 free-threaded support by greateggsgreg · Pull Request #2721 · pythonnet/pythonnet · GitHub
Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
ebd0a95
Thread-safety prep for free-threading builds
greateggsgreg May 9, 2026
cf8372a
Initialise pythonnet on free-threaded Python (#2720)
greateggsgreg May 9, 2026
441954f
Make extension/CLR-object registries thread-safe
greateggsgreg May 9, 2026
19c7f59
Atomic type creation in ReflectedClrType.GetOrCreate / TypeManager.Ge…
greateggsgreg May 9, 2026
1029c62
Add free-threaded thread-stress tests and 3.14t to CI matrix
greateggsgreg May 9, 2026
3158ca5
Atomic GCHandle ownership and finalizer-thread shutdown guards
greateggsgreg May 10, 2026
1149295
Make additional internal registries thread-safe
greateggsgreg May 10, 2026
e8afa85
test_thread: join worker threads before returning
greateggsgreg May 10, 2026
3954df3
test_thread: cover ModuleObject thread-safe registries
greateggsgreg May 10, 2026
28013af
Wider thread-safety audit fixes for free-threaded Python
greateggsgreg May 10, 2026
9a3dd59
Document lock acquisition sites and strong->weak GCHandle swap
greateggsgreg May 10, 2026
cb1dcde
Preserve InternString single-write invariant under DEBUG
greateggsgreg May 10, 2026
74d12f1
test_thread: cover real-world consumer patterns
greateggsgreg May 10, 2026
a30c84e
Auto-detect free-threaded libpython in venv home
greateggsgreg May 12, 2026
be5ba34
Snapshot pypath, use ConcurrentDictionary for thunks and slot holders
greateggsgreg May 13, 2026
946785e
Fix handling of python runtime suffixes m/t
greateggsgreg May 13, 2026
77ca497
Fix threadtest race
greateggsgreg May 13, 2026
af518bb
Fix double-free in chained ClassDerived Finalize
greateggsgreg May 13, 2026
ed735ac
Enable Mono CI jobs on free-threaded Python 3.14
greateggsgreg May 13, 2026
bd3f0bf
Inline freethreaded_only as pytest.mark.skipif at call sites
greateggsgreg May 13, 2026
c4688b3
Fix InterruptTest assertion on free-threaded Python 3.14
greateggsgreg May 13, 2026
84cd07c
Add concurrent stress tests for PyBuffer.Dispose and CLR-cycle gc.col…
greateggsgreg May 15, 2026
d9b658d
Trim concurrent overhead on hot paths from free-threading prep
greateggsgreg May 15, 2026
20c51ad
Pre-warm ctor binder in concurrent-gc test to avoid first-call race
greateggsgreg May 15, 2026
f5f8ab0
Make MethodBinder.GetMethods lazy init thread-safe under free-threading
greateggsgreg May 15, 2026
c5097eb
Precompute method precedence to avoid quadratic GetParameters allocat…
greateggsgreg May 15, 2026
18e4901
Zero the slot in ClassDerived.tp_dealloc when tp_clear already ran to…
greateggsgreg May 15, 2026
e0569d0
Keep ClassDerived wrapper alive across the NewObjectToPython slot dem…
greateggsgreg May 15, 2026
2c9f861
Document private helpers added during free-threading prep
greateggsgreg May 16, 2026
ee3add4
Add debug echoes and a 6-minute step timeout to the Mono test job
greateggsgreg May 16, 2026
414ce70
Drop the per-loop CLR GC.Collect from concurrent-gc test to avoid Mon…
greateggsgreg May 16, 2026
99a99db
Add temporary Mono-step diagnostics on Linux/macOS to locate the x64-…
greateggsgreg May 16, 2026
8e7c2ba
Revert temporary Mono-step diagnostics now that the underlying race i…
greateggsgreg May 16, 2026
f8b69a2
Merge branch 'master' into freethreading-prep
greateggsgreg May 16, 2026
8f2804c
Add user-facing threading guide covering GIL, free-threading, and com…
greateggsgreg May 18, 2026
9c369ed
Harden CollectBasicObject against .NET-GC timing differences
greateggsgreg May 18, 2026
d1b4daa
Merge branch 'master' into freethreading-prep
greateggsgreg May 25, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
instance: macos-15
suffix: -macos-aarch64-none

python: ["3.10", "3.11", "3.12", "3.13", "3.14"]
python: ["3.10", "3.11", "3.12", "3.13", "3.14", "3.14t"]

exclude:
# Fails with initfs_encoding error
Expand All @@ -67,6 +67,17 @@ jobs:
platform: x86
python: '3.13'

# Free-threaded Python on Windows is not yet supported by pythonnet's
# native build chain; restrict 3.14t to Linux and macOS for now.
- os:
category: windows
platform: x86
python: '3.14t'
- os:
category: windows
platform: x64
python: '3.14t'

env:
NUGET_PACKAGES: ${{ github.workspace }}/.nuget/packages
steps:
Expand Down
1 change: 1 addition & 0 deletions doc/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ page. Use the `Python.NET issue tracker`_ to report issues.
python
dotnet
codecs
threading
pyreference
reference

Expand Down
214 changes: 214 additions & 0 deletions doc/source/threading.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,214 @@
Threading
=========

This page explains how Python.NET interacts with the Python Global Interpreter
Lock (GIL) and with managed threads, and what guarantees the runtime makes
when your code is multi-threaded. It covers both classic CPython builds and
the free-threaded build introduced in CPython 3.13 (``Py_GIL_DISABLED``).

The model in one paragraph
--------------------------

Python.NET embeds CPython, so every interaction with a Python object —
including reading a ``PyObject``'s attributes, calling a Python callable,
constructing a Python value, or letting a ``PyObject`` go out of scope — must
happen while the calling thread is *attached* to the interpreter. On a
classic (GIL-enabled) CPython build "attached" means "holds the GIL"; on a
free-threaded build it means "has an active thread state". In both cases the
attachment API is the same: ``Py.GIL()`` on the C# side and
``threading.Thread`` / ``_thread`` on the Python side. Forgetting to attach
will crash the process or corrupt memory.

Acquiring the GIL from C#
-------------------------

When .NET code calls into Python it must hold the GIL. Use the ``Py.GIL()``
disposable to acquire and release it::

using (Py.GIL())
{
dynamic np = Py.Import("numpy");
var arr = np.array(new[] { 1, 2, 3 });
// ... interact with arr ...
}

``Py.GIL()`` is re-entrant: nesting calls on the same thread is harmless and
cheap. Always pair acquisition with disposal — the ``using`` form does this
automatically, and you must release the GIL on the same thread that acquired
it.

If you need a Python object to outlive the ``using`` block, copy what you
need (e.g. ``.As<int[]>()`` or ``new PyObject(value)``) before releasing the
GIL.

Releasing the GIL for long-running .NET work
--------------------------------------------

If a managed call holds the GIL but then does long-running work that does not
touch Python (heavy CPU, blocking I/O, native interop), release the GIL so
other Python threads can run::

IntPtr threadState = PythonEngine.BeginAllowThreads();
try
{
DoCpuHeavyWork(); // safe: no Python C API calls
}
finally
{
PythonEngine.EndAllowThreads(threadState);
}

Inside the ``BeginAllowThreads``/``EndAllowThreads`` block you must not touch
any Python object. If you need to talk to Python from worker threads spawned
in this region, those threads must acquire the GIL themselves with
``Py.GIL()``.

Calling .NET from Python threads
--------------------------------

Calling a managed method from a Python ``threading.Thread`` works
transparently — Python.NET handles GIL acquisition/release around the
managed call. The managed code sees the GIL held on entry and is free to
release it via ``BeginAllowThreads`` if it does its own blocking work.

Calling Python from CLR threads
-------------------------------

A CLR thread that was *not* spawned by Python (a thread-pool task, a
``Thread`` started in C#, an ``async`` continuation that resumed on a
different thread, etc.) must acquire the GIL before touching any
``PyObject``::

Task.Run(() =>
{
using (Py.GIL())
{
// safe to use PyObjects here
}
});

Forgetting this is the most common pythonnet threading bug. Symptoms range
from immediate segfaults to subtle refcount corruption that crashes much
later.

Reference counting and finalizers
---------------------------------

``PyObject`` follows the .NET ``IDisposable`` pattern. ``Dispose()`` (or the
end of a ``using`` block) drops the underlying Python reference; the GC
finalizer queues the same release for the next time Python.NET is on the GIL.

Two practical consequences:

* **Don't share a single ``PyObject`` instance across threads without
serialising access.** ``PyObject`` is not internally locked. If multiple
threads concurrently dispose the same instance, the underlying refcount can
go negative.

* **Don't rely on the GC finalizer running promptly.** The PyObject is only
freed when a Python.NET API later reacquires the GIL. If your application
shuts down without that happening, finalizable PyObjects can be reported as
leaked.

Free-threaded Python (PEP 703)
------------------------------

Starting with the free-threaded CPython 3.13+ build (``Py_GIL_DISABLED``),
the GIL is no longer the serialisation point for Python C API calls.
Python.NET is tested against the ``3.14t`` (free-threaded) interpreter and
behaves as follows under that build:

* ``Py.GIL()`` still acquires a thread state. It is functionally a no-op
for mutual exclusion but is still required for thread-state attachment.
Existing code that uses ``using (Py.GIL())`` continues to work without
changes.
* ``PythonEngine.BeginAllowThreads`` / ``EndAllowThreads`` similarly
manage the thread state and are still needed if you want the GC and
other Python threads to run while you're in long-running unmanaged code.
* Internal Python.NET caches (the reflection cache, generic-type binding
cache, dynamic-dispatch cache, module attribute cache, the interned-
string table, etc.) are thread-safe. You may read and call CLR types
concurrently from any number of threads without external locking.
* The reference-counting protocol uses CPython's ``Py_REFCNT`` symbol on
3.14+, which returns the merged biased + shared refcount; values you read
from ``PyObject.Refcount`` are correct under free-threading.

Behaviour that is *unchanged* between GIL and free-threaded builds:

* A managed object exposed to Python (e.g. via ``System.Object`` or a
Python subclass of a CLR type) is still owned by a single CLR side: you
must not mutate its plain CLR fields from multiple threads without your
own locking. Python.NET only protects its own bookkeeping, not your
domain data.
* Operations on a single ``PyObject`` instance still require external
serialisation — see "Reference counting" above.

Patterns
--------

Concurrent CLR access from Python
"""""""""""""""""""""""""""""""""

Hammering CLR attributes / generic types from many threads is supported::

from threading import Thread
import System
from System.Collections.Generic import List

def worker():
for _ in range(1000):
_ = System.String.Empty
_ = List[int]()

threads = [Thread(target=worker) for _ in range(8)]
for t in threads: t.start()
for t in threads: t.join()

This works on both GIL and free-threaded builds.

Python callback invoked from a managed thread
"""""""""""""""""""""""""""""""""""""""""""""

If a managed component calls back into a Python delegate from a thread it
spawned, that callback path acquires the GIL internally — you do not need to
add ``Py.GIL()`` around the Python code in the delegate.

Spawning a managed thread from inside ``Py.GIL()``
""""""""""""""""""""""""""""""""""""""""""""""""""

If you start a managed thread while holding the GIL and the thread needs to
call back into Python, release the GIL first so the new thread can acquire
it::

using (Py.GIL())
{
var pyCallback = scope.Get("on_done");
PythonEngine.BeginAllowThreads(); // let workers acquire the GIL
try
{
// spawn workers, wait for them...
}
finally
{
PythonEngine.EndAllowThreads(...);
}
}

Without the ``BeginAllowThreads`` the spawned thread blocks forever waiting
for the GIL the parent thread is still holding.

Common pitfalls
---------------

* Holding ``Py.GIL()`` across ``Task.Run`` / ``await`` boundaries. Async
continuations can resume on a different thread; the GIL handle is
thread-bound and must be released on the same thread that acquired it.
* Passing a ``PyObject`` to a managed worker without taking ownership. If
the producer disposes its handle while the consumer is still using it,
the worker will operate on a freed object. Wrap the producer's
``PyObject`` with ``new PyObject(value)`` before handing it off, or use
``NewReference()``.
* Calling a Python callable that does CPU-bound work without releasing the
GIL. Other Python threads cannot make progress in that case, even on a
free-threaded build where the GIL is otherwise a no-op (the callable
itself may still touch contended Python state).
39 changes: 20 additions & 19 deletions src/embed_tests/TestFinalizer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ private static void FullGCCollect()
{
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect(); // reclaim objects whose finalizers just ran
}

[Test]
Expand All @@ -51,28 +52,28 @@ public void CollectBasicObject()
Finalizer.Instance.BeforeCollect += handler;

IntPtr pyObj = MakeAGarbage(out var shortWeak, out var longWeak);
FullGCCollect();
// The object has been resurrected
Warn.If(
shortWeak.IsAlive,
"The referenced object is alive although it should have been collected",
shortWeak
);
Assert.That(
longWeak.IsAlive,
Is.True,
$"The reference object is not alive although it should still be"
);

// The real contract: after the wrapper is GC'd, the underlying
// Python pointer must end up in Finalizer's queue. Poll because
// .NET Framework / .NET Core differ in how many GC cycles it takes.
List<IntPtr> garbage = null;
for (int attempt = 0; attempt < 10; attempt++)
{
var garbage = Finalizer.Instance.GetCollectedObjects();
Assert.NotZero(garbage.Count, "There should still be garbage around");
Warn.Unless(
garbage.Contains(pyObj),
$"The {nameof(longWeak)} reference doesn't show up in the garbage list",
garbage
);
FullGCCollect();
garbage = Finalizer.Instance.GetCollectedObjects();
if (garbage.Contains(pyObj)) break;
Thread.Sleep(20);
}

Warn.If(shortWeak.IsAlive,
"shortWeak is alive after FullGCCollect; runtime hasn't reclaimed the wrapper yet",
shortWeak);
// longWeak.IsAlive at this point is .NET-GC-implementation-defined
// (Framework reclaims post-finalize objects more eagerly than Core);
// intentionally not asserted.

Assert.That(garbage, Has.Member(pyObj),
"PyObject did not reach Finalizer.Instance.GetCollectedObjects()");
try
{
Finalizer.Instance.Collect();
Expand Down
12 changes: 11 additions & 1 deletion src/embed_tests/TestInterrupt.cs
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,17 @@ import time
Assert.That(asyncCall.Wait(TimeSpan.FromSeconds(5)), Is.True, "Async thread was not interrupted in time");
PythonEngine.EndAllowThreads(threadState);

Assert.That(asyncCall.Result, Is.EqualTo(0));
// On free-threaded CPython 3.14, PyRun_SimpleString may return -1 even
// when the script catches the async-injected KeyboardInterrupt — the
// C-level error indicator depends on which bytecode boundary the
// async-exc fires at, and isn't always cleared the way GIL builds clear
// it. The interrupt firing and the script terminating cleanly are what
// this test exercises; the return code is a side-effect that's only
// deterministic under the GIL.
if (Python.Runtime.Native.ABI.IsFreeThreaded)
Assert.That(asyncCall.Result, Is.AnyOf(0, -1));
else
Assert.That(asyncCall.Result, Is.EqualTo(0));
}
}
}
5 changes: 3 additions & 2 deletions src/embed_tests/TestNativeTypeOffset.cs
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,10 @@ public class TestNativeTypeOffset
public void LoadNativeTypeOffsetClass()
{
PyObject sys = Py.Import("sys");
// We can safely ignore the "m" abi flag
// "m" is benign; "t" (free-threaded) is handled via ABI.ObjectHeadOffset
// rather than the install-time-generated NativeTypeOffset class.
var abiflags = sys.HasAttr("abiflags") ? sys.GetAttr("abiflags").ToString() : "";
abiflags = abiflags.Replace("m", "");
abiflags = abiflags.Replace("m", "").Replace("t", "");
if (!string.IsNullOrEmpty(abiflags))
{
string typeName = "Python.Runtime.NativeTypeOffset, Python.Runtime";
Expand Down
37 changes: 37 additions & 0 deletions src/embed_tests/TestPyBuffer.cs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
using System;
using System.Runtime.CompilerServices;
using System.Text;
using System.Threading;
using NUnit.Framework;
using Python.Runtime;
using Python.Runtime.Codecs;
Expand Down Expand Up @@ -129,6 +130,42 @@ public void MultidimensionalNumPyArray()
});
}

[Test]
public void ConcurrentDispose()
{
// Two threads racing on Dispose() must not double-release the view —
// Interlocked.Exchange on disposedValue gates PyBuffer_Release.
// Smoke test: no crash, exception, or buffer-protocol violation.
using var _ = Py.GIL();
using var arr = ByteArrayFromAsciiString("hello world! !$%&/()=?");

const int iterations = 200;
for (int i = 0; i < iterations; i++)
{
PyBuffer buf = arr.GetBuffer();

IntPtr ts = PythonEngine.BeginAllowThreads();
using var barrier = new Barrier(2);
Exception captured = null;
Action race = () =>
{
try
{
barrier.SignalAndWait();
using (Py.GIL()) buf.Dispose();
}
catch (Exception ex) { Interlocked.CompareExchange(ref captured, ex, null); }
};
var t1 = new Thread(() => race());
var t2 = new Thread(() => race());
t1.Start(); t2.Start();
t1.Join(); t2.Join();
PythonEngine.EndAllowThreads(ts);

if (captured != null) throw captured;
}
}

[MethodImpl(MethodImplOptions.NoInlining)]
static void MakeBufAndLeak(PyObject bufProvider)
{
Expand Down
Loading
Loading
pFad - Phonifier reborn

Pfad - The Proxy pFad © 2024 Your Company Name. All rights reserved.





Check this box to remove all script contents from the fetched content.



Check this box to remove all images from the fetched content.


Check this box to remove all CSS styles from the fetched content.


Check this box to keep images inefficiently compressed and original size.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy