PEP 734: Multiple Interpreters in the Stdlib

eric.snow · 2024-09-26T22:10:47.553Z

No, you did the right thing. I just haven’t gotten to your post yet. It will probably be today. Sorry for the wait.

TkTech · 2024-11-11T05:57:34.085Z

When using InterpreterPoolExecutor, the started sub-processors do not seem to fully inherit the sys.path of the parent, which does work for all other types of executors. I ran into this when using sub-interpreters inside of a pytest-based tests, as the started jobs did not get the test directory in their sys.path. This is under 3.13 using the latest backport.

I’m not entirely convinced this is a bug as it’s a logical outcome, but it’s certainly unexpected when compared to the other executors.

eric.snow · 2024-11-11T22:05:10.698Z

I’d say this is worth a bug report. Please open an issue and feel free to CC me (@ericsnowcurrently) on it.

ZeroIntensity · 2024-11-13T22:12:26.387Z

For reference: InterpreterPoolExecutor workers do not inherit modifications made to sys.path before starting. · Issue #126714 · python/cpython · GitHub

a-reich · 2025-04-28T17:02:56.766Z

I saw the PEP was resubmitted to the SC - any expectation on whether they will make a decision in time for the 3.14 release?

barry · 2025-06-05T19:35:33.476Z

After discussing PEP 734 before and during PyCon, with you and many others, and confirming with @hugovk that he will issue a feature freeze exception for Python 3.14, the Steering Council approves PEP 734, with one suggested change (see below).

The Python community is incredibly grateful for your diligence, tenacity, and expertise in bringing the subinterpreters feature to fruition. Despite the Zen of Python’s “There’s Only One Way To Do It”, we believe that giving Python developers options for parallelism is ultimately in their best interest. Choosing which mechanism to use for concurrent programming in Python is a complex decision, and subinterpreters brings an important and useful option to the table. Besides that, all the work you’ve done “under the hood” has helped improve Python in general, and that is also greatly appreciated.

We have one request. We think rather than adding a new top-level subinterpreters module in the stdlib, a better place for it would be concurrent.interpreters. This should make the module easily discoverable, and it gives more validity to the concurrent package. We imagine that if we add additional APIs for free threading in the future, they might also go under the concurrent package.

If you’re agreeable, then congratulations! Please update the PEP accordingly and coordinate with Hugo on any PRs requiring an exception.

Barry, on behalf of the Steering Council

eric.snow · 2025-06-05T20:35:10.749Z

This is great news!!! Thanks for taking the time and making this possible for 3.14. I definitely look forward to all the things the community will be able to invent with multiple interpreters once we have this stdlib module.

As to concurrent.interpreters, I’m not sure it fits. The module is meant to minimally expose the existing runtime feature, rather than being specifically designed around supporting concurrency. Using interpreters to enable parallelism and human-friendly concurrency were certainly my motivations. However, interpreters don’t actually provide any concurrency themselves. Other features, like threads, provide the actual concurrency. Thus, I wouldn’t expect to find the module as concurrent.interpreters.

FWIW, I totally support making better use of concurrent. That’s a package where I would expect to find general, high-level, concurrency-related utilities. For example, if we were to add a CSP module some day, I’d expect to find it at concurrent.csp. Honestly, multiprocessing would probably be more appropriate as concurrent.multiprocessing (or similar), given that it is exactly a high-level abstraction of concurrency. I can totally see us adding different data structures and helpers related to concurrency, like thread-safe containers.

Interpreters aren’t that sort of thing though. Rather than a high-level abstraction, they are a fundamental, low-level runtime feature. It would be more appropriate to group the module with other modules that expose the runtime, like sys, builtins, atexit, types, inspect, dis, etc. I suppose it could be sys.interpreters. However, I’m not sure about setting that precedent (of a sys submodule) without more thought. Historically such modules have all been top-level.

That said, the queues that are a part of PEP 734 do specifically support concurrency. They are important because using multiple interpreters is substantially less meaningful without a way to communicate between them. Exposing cross-interpreter queues in a submodule of the concurrent package could definitely make sense, though I’m not sure what we’d call it. Probably not concurrent.queue, but maybe concurrent.isolated.queues? Or perhaps it would work under concurrent.queue if the type were named something like IsolatedQueue or CrossInterpreterQueue. Anyway, we could sort that out for 3.15 and just expose interpreters.create_queue() for 3.14 (which is what is already implemented).

So…here are the options that make sense to me:

keep it as interpreters, keep interpreters.create_queue(), change the name to CrossInterpreterQueue, and sort out a dedicated home for the queues in 3.15 (with a separate PEP or something)
likewise, but sys.interpreters
like (1) and (2), but sort out the home for queues in 3.14

Realistically, it’s too late to sort out queues for 3.14 other than to keep interpreters.create_queue(). It might be a bit late to be deciding about adding a sys submodule, but, honestly, the more I think about it the more I’m okay with it. (For 3.15, we could even consider moving more stuff to sys submodules.)

Ultimately, if this boils down to a decision between concurrent.interpreters or wait until 3.15 then I’ll go along with the former to get the module out sooner. I think it’s that important to get it into the hands of users.

-eric

encukou · 2025-06-05T21:23:42.185Z

AFAIK, (sub-)interpreters always were about isolation. Isolated GILs are a great application of course, but so is, for example, isolating sys.path to serve two web apps with conflicting versions of a common dependency.

ncoghlan · 2025-06-06T02:36:17.140Z

@ericsnowcurrently While interpreters aren’t necessarily about concurrent execution, they are about maintaining concurrent state (as opposed to repeatedly initialising and finalising the main interpreter, or mutating the main interpreter’s global state).

We certainly have more concurrency related things outside the concurrent namespace than inside, but putting interpreters there seems reasonable enough to me to not be worth worrying further about (resolving arguable naming decisions is one of the reasons we have the SC, after all).

encukou · 2025-06-06T06:01:19.121Z

Oh, of course, put it in under the name the SC chose! There’ll always be opposition to any idea, don’t feel blocked by that.

Alyssa Coghlan:

resolving arguable naming decisions is one of the reasons we have the SC, after all

Yeah, but usually there’s some prior public discussion.
The decision is arguable, but I have not seen the arguments. More importantly, I’m not sure the SC has seen the arguments :‍)

Alyssa Coghlan:

While interpreters aren’t necessarily about concurrent execution, they are about maintaining concurrent state (as opposed to repeatedly initialising and finalising the main interpreter, or mutating the main interpreter’s global state).

That’s certainly a meaning of the word concurrent. What’s less clear to me is whether Python should adopt this meaning.
Note that the proposed module provides a synchronous, blocking API (plus one 4-line helper that wraps call in a new thread).

Compare: In turtle, you usually call functions like forward to move the turtle around. But you can also create concurrent turtles, and control each one individually. It’s even possible to have concurrent turtle windows (TurtleScreens), though there isn’t a convenient Python API for that use case.

Or: list is concurrent—you can have several lists at once, and appending to one doesn’t affect the others.

I think that multiple is a better term, and it’s better to only use concurrent as an antonym to synchronous/blocking.
Of course, it’s an arguable opinion, and you don’t need to convince me that I’m wrong. All I ask is that a SC member is aware of my dissent.

pitrou · 2025-06-06T08:42:57.081Z

Putting the interpreters module under concurrent sounds a bit like putting subprocess under concurrent. It stretches the definitions and makes the structure less readable for the user.

That said, I’m ok with the idea that the SC gets to choose the color of the bikeshed in exchange for their hard work.

ncoghlan · 2025-06-06T08:52:36.789Z

We have lots of top level modules that are concurrency related (threading, asyncio, queue, multiprocessing are the ones that most readily come to mind). concurrent.futures was namespaced because a bare top level “futures” module was considered too ambiguous (especially in finance applications).

I don’t think the ambiguity concern for interpreters is as significant as that one, but I was also there for the original discussions when Eric started down this road, so I’m biased towards thinking of the term in the way PEP 734 uses it.

Since we mostly want the public interpreter module as a building block for further work on more inherently threadsafe concurrency models like CSP and the Actor model, namespacing it as being about concurrency strikes me as “not necessary, but not wrong either”.

TobiasHT · 2025-06-06T09:54:33.070Z

In my opinion, while individuals who employ multiprocessing, asyncio, or multithreading are explicitly aiming to develop concurrent programs, those who utilise subinterpreters may not necessarily have the same intention. For instance, an individual could create a compact runtime environment to emulate a secure container for executing arbitrary Python code. So I believe placing the multiple interpreters library under the concurrent namespace may limit its perception to solely concurrency, when in reality, its capabilities extend beyond that.

steve.dower · 2025-06-06T10:36:09.410Z

Even if the high-level interface goes in as concurrent.interpreters, there’ll be a low-level interface as _interpreters, which we can actually just document and recommend for special/undiscovered uses by libraries but not “regular” users. That also allows concurrent.interpreters to really lean into the concurrency aspects, e.g. we can add types to it that are useful for coordinating multiple subinterpreters without having to somehow interpret them as being strictly part of the subinterpreters mechanism.

One really easy way to clarify the separation would be to say that _interpreters provides isolation while concurrent.interpreters provides the ways to carefully bridge that isolation.

But I agree with Antoine. While it’s a stretch in terms of purity,^[1] it seems the most pragmatic way forward.

(Edit - fixed _interpreters module name.)

Keeping subprocess top-level and putting multiprocessing under concurrent would be more like what I’m proposing. ↩︎

gerardw · 2025-06-06T11:31:36.711Z

Eric Snow:

thread

This is likely a documentation issue. As an end user it’s unclear to me why I would use subinterpreters instead of simply using a free threaded thread.

TobiasHT · 2025-06-06T11:43:50.772Z

I have a feeling that when one chooses to use subinterpreters, they already know why they’ve chosen to do so and their problem fits the need for using subinterpreters.

AA-Turner · 2025-06-06T14:19:41.291Z

Editorial note: the PEP proposes the interpreters name for the module; the current (3.13+) low-level module is called _interpreters, rather than ‘subinterpreters’ for either.

A

eric.snow · 2025-06-06T17:38:44.909Z

FWIW, putting concurrency-related tools for multiple interpreters under concurrent does make sense to me. I can image a variety of future additions that could live in concurrent.interpreters (or concurrent.isolated or whatever). For example, I’d like to have a dedicated module for cross-interpreter queues, for a clearer API than just exposing interpreters.create_queue(), etc. That’s been implemented as interpreters.queues for quite a while (but not a part of PEP 734 nor what I plan to merge for 3.14). Adding that to 3.15 as something like concurrent.interpreters.queue makes some sense.

That said, the specific API from PEP 734–just the minimal interpreters module as proposed–still seems to me as very out of place as concurrent.interpreters. It doesn’t really fit conceptually and I’m not sure why there’s such a push to contort it into place. Perhaps I’m missing something. I do strongly agree with the sentiment that “we should make better use of the concurrent namespace”, but that goal shouldn’t mean we put things there that don’t belong.

FYI, I do plan on merging the change for 3.14 as concurrent.interpreters, to skirt any controversy and get it in now, but I hope I can convince those in the concurrent.interpreters camp in the next few days to move it back to interpreters (or even sys.interpreters if that’s more palatable) for 3.14+.

eric.snow · 2025-06-06T20:30:20.822Z

FTR, [PEP 734] Mark PEP 734 as Accepted by ericsnowcurrently · Pull Request #4451 · python/peps · GitHub.

brettcannon · 2025-06-06T21:13:51.073Z

TobiasHT:

For instance, an individual could create a compact runtime environment to emulate a secure container for executing arbitrary Python code.

I don’t think there’s any promises that the interpreters are sufficiently isolated that you can run arbitrary code in a subinterpreter safely.