Could we add Python to system PATH by default?

edison.abahurire · 2020-01-27T10:03:16.240Z

When installing python for Windows, I see a checkbox that gives an option of adding it to the ‘PATH’ which I presume is adding it to system environment variables.
Given that majority of users installing Python want it to be added to the PATH so that it can be called from the terminal and IDEs. I have been working with analysts learning python for data and it caused issues to almost all of them.

Could we move to have this option selected by default?

uranusjr · 2020-01-27T12:28:18.542Z

I believe the preference of core developers is to encourage users to use the py command. The option is deselected by default as a conscious decision to discourage it (and I believe there are core devs wishing to remove it outright), since adding entries to PATH has a lot more implications on Windows than UNIX-like operating systems. So the answer is very likely a definite “no.”

pf_moore · 2020-01-27T13:00:05.756Z

In addition, a per-user install of Python (the default) can only add to the user PATH setting. The user PATH is lower priority than the system PATH, so it’s possible to get a situation where "add python to PATH" doesn’t actually do what the user expects (because there is an older version of Python on the system PATH - not uncommon as per-user installs were not the default in older Python versions.

Basically, it’s possible to get confusing behaviour if you blindly add Python to the user PATH, so we don’t do so by default. Selecting the “Add python to PATH” checkbox acts as a confirmation that the user is at least aware of what they are doing (and so has a chance of debugging any unexpected issues).

oscarbenjamin · 2020-01-30T12:43:30.169Z

I thought that newer Windows has python.exe on PATH (even before you install Python!):

Python – 21 May 19

Who put Python in the Windows 10 May 2019 Update? | Python

Python might be hard to install on Windows, but with the latest Windows 10 update, you can type "python" to find it in the Microsoft Store.

pf_moore · 2020-01-30T13:32:55.906Z

It has a python.exe executable which will take you to the Windows Store to install the store version of Python (which will be on PATH when installed, I believe). The python.org installers don’t add themselves to PATH, though. (Which presumably does result in a rather odd UX, where you install python with the defaults from python.org, then try python and get the store…

@steve.dower I don’t imagine there’s any good way to avoid this, is there? Is the expectation that as time goes on, the store version of Python will become the “standard” that we recommend to end users?

steve.dower · 2020-01-30T22:22:53.613Z

(Thanks for mentioning me Paul, I have this category muted.)

There’s a very complicated bug that I’ve been trying to track down for months, involving a third party installer with an issue and some kind of user migration step in Windows, but without this issue the Windows Store package of Python does not interfere with a python.org install. (The issue appears as your WindowsApps folder being listed in PATH twice, and can be fixed by removing the first entry.)

I believe that for most people, either the Store package or Anaconda will eventually be the primary ways to get Python. Those who need the python.org installer can click a checkbox.

The real problem with modifying PATH is it enables (intentional or unintentional) DLL hijacking (see CVE-2020-8315 for example, which essentially requires a PATH modification to exploit). To fix this, we’d have to remove all the DLLs from the directory on PATH, which isn’t really feasible.

Alternatively, we could rename py.exe to python.exe so that (people can unlearn all of their muscle memory and) it looks like the actual executable. Though given the feedback about the Store package, we now see a lot of people making assumptions about how the executable works that would break again if we did that. And also I’m not that motivated to manage the change, since it’s such a stop-gap and doesn’t really help things overall like moving to more reliable installers.

So the answer is a no, for a mix of security, compatibility, and resourcing reasons.

eryksun · 2020-02-01T00:44:34.629Z

steve.dower:

the Windows Store package of Python does not interfere with a python.org install.

The 3.7 app distribution may take precedence over PATH when running “python” or “pythonw” using the shell execute family of functions and COM interfaces, or when getting a command line to run “python” by way of SHEvaluateSystemCommandTemplate. These API functions prefer registered application names in the system (but not user) “App Paths” key over searching PATH. A searched name either has to literally match a registered name or has to have no extension and match with “.EXE” appended (*).

Shell API execution is used, for example, by the Win+R dialog and Python’s os.startfile. On the other hand, CMD and PowerShell will search PATH first, so it’s not an issue with the regular command-line shell interface. Also, a direct CreateProcessW call has no knowledge of the shell API “App Paths”, so it’s not an issue there either.

The 3.7 app distribution registers system and user “App Paths” subkeys for “python.exe”, “pythonw.exe”, “idle.exe”, and “pip.exe”. The 3.8 app distribution registers only “3.8” specific names such as “python3.8.exe” and “pip3.8.exe”. These registered names run binaries from the corresponding system app directory under “%ProgramFiles%\WindowsApps”.

This is particularly problematic if both the 3.7 and 3.8 app distributions are installed. The “python.exe” name will be registered to run the “python.exe” binary of the 3.7 app. Executing this binary directly from a normal session context is only allowed if the user’s corresponding appexec link in “%LocalAppData%\Microsoft\WindowsApps” resolves to the same file. But when the 3.8 app is installed, the user’s “python.exe” appexec link instead resolves to the path of the 3.8 binary. Consequently running python from the Win+R dialog or os.startfile('python') will fail with a permission error.

(*) Running “python3.8.exe” via the registered “App Paths” key requires using the “.exe” extension. From the POV of the shell, “python3.8” has the extension “.8”, so it won’t try appending “.EXE” to match the registered name. However, a literal match works, so we can copy the “python3.8.exe” subkey to a new subkey named “python3.8”.

steve.dower · 2020-02-01T00:51:27.182Z

eryksun:

The 3.7 app distribution registers system and user “App Paths” subkeys for “python.exe”, “pythonw.exe”, “idle.exe”, and “pip.exe”. The 3.8 app distribution registers only “3.8” specific names such as “python3.8.exe” and “pip3.8.exe”.

This should be based on which app execution aliases are enabled. So if you configure the versionless aliases on 3.8 then they’ll point to the right place. (Or it may be due to renaming the actual executables to include versions, which was the best possible workaround for a Windows issue that may never be fixed…)

eryksun · 2020-02-01T04:58:28.999Z

steve.dower:

This should be based on which app execution aliases are enabled. So if you configure the versionless aliases on 3.8 then they’ll point to the right place.

I assumed that if both 3.7 and 3.8 were installed, the user would want the “python.exe” appexec link to target the 3.8 binary. If the link instead targets the 3.7 binary, then running python via the Win+R run dialog or os.startfile should work given the default settings (it does for me).

To reiterate, the execution of “python” via the shell API doesn’t directly consume the appexec link if a system “App Paths” subkey exists for “python.exe”. The 3.7 app sets the value of this key to the path of the 3.7 “python.exe” binary.

When the app binary gets directly executed by CreateProcessW in the normal session security context, it does indirectly use the appexec link of the same name – assuming it exists and targets the executed binary. If the link doesn’t exist or doesn’t target the executed binary, the call fails with a permission error. Otherwise, CreateProcessW extracts security attributes from the link that it sets in a custom access token for the child process. It also impersonates this token in the calling thread, which allows the NtCreateUserProcess system call to succeed.

The security attributes that it sets are required by a conditional access-control entry (ACE) that grants read and execute access to the app’s installed files. For the Python 3.8 app directory, this ACE has the following SDDL definition:

(XA;OICI;0x1200a9;;;BU;
    (WIN://SYSAPPID Contains
        "PYTHONSOFTWAREFOUNDATION.PYTHON.3.8_QBZ5N2KFRA8P0"))

“XA” is a conditional access-allowed ACE. “OI” means object (file) inherit, and “CI” means container (directory) inherit. The granted access is 0x1200a9, i.e. FILE_GENERIC_READ | FILE_GENERIC_EXECUTE. The security principle is “BU”, i.e. “BUILTIN\Users”. The condition is the presence of a security attribute named “WIN://SYSAPPID” in the token that contains the given app identifier.

If the user sets the “python.exe” appexec link to run 3.8, and the 3.7 app is also installed, then running “python” via the shell API will fail. It attempts to run the 3.7 binary, but there’s no matching “python.exe” appexec link. To correct this, the system (not user) “App Paths” subkey for “python.exe” would need to be manually modified to run the 3.8 binary. This in turn requires the “python3.8.exe” appexec link to be enabled, which should be the case by default. One could also simply remove the system “App Paths” key for “python.exe”, which allows the shell API to search PATH.

steve.dower:

Or it may be due to renaming the actual executables to include versions

Yes, it seems this determines the registered names in “App Paths” and the target path of appexec links. And it’s self-consistent in that the “App Paths” key runs the same binary that’s the target of the appexec links.

The 3.7 app installs a “python.exe” binary. The user’s “python.exe”, “python3.exe”, and “python3.7.exe” appexec links, if enabled for 3.7, all target this binary. It also registers a system “App Paths” key named “python.exe” that directly executes this binary. The latter works as long as the user has the “python.exe” appexec link set to 3.7.

The 3.8 app installs both “python.exe” and “python3.8.exe” binaries. All of the appexec links, if enabled for 3.8, target the “python3.8.exe” binary, as does the same-named system “App Paths” key. The binary can be directly executed in a normal session context, such as via os.startfile('python3.8.exe'), as long as the user has a “python3.8.exe” appexec link set to 3.8.

On a related note, apparently the app startup code has been modified to tell a white lie. Even though the process image is “python3.8.exe”, it pretends that sys.executable is “python.exe”:

>>> print(os.path.basename(_winapi.GetModuleFileName(0)))
python3.8.exe
>>> print(os.path.basename(sys.executable))
python.exe

This still works because “python.exe” exists and is allowed to be executed from the app security context.

Almenon · 2021-02-20T20:36:09.330Z

uranusjr:

I believe the preference of core developers is to encourage users to use the py command

Py is not installed for per-user installations I believe.

pf_moore:

(because there is an older version of Python on the system PATH - not uncommon as per-user installs were not the default in older Python versions.

Do you happen to remember what old versions of python did this? Py2? Maybe this could be fixed by checking if python is in the system path already and warning if so?

steve.dower:

I believe that for most people, either the Store package or Anaconda will eventually be the primary ways to get Python

Is there some changes happening that would make this the case? I would imagine most people would install python by googling “python” or “python install” or some variant thereof, which would take them to the installer on the python website. The python docs implicitly recommend using the python website installer as well.

“The full installer contains all components and is the best option for developers using Python for any kind of project.”

Some tutorials I found referenced the windows store python installation however, so you have a point there.

steve.dower:

The real problem with modifying PATH is it enables (intentional or unintentional) DLL hijacking (see CVE-2020-8315 for example, which essentially requires a PATH modification to exploit)

CVE-2020-8315 doesn’t affect windows 8 or later. Are there other vulnerabilities that effect more recent windows versions?

steve.dower · 2021-02-22T19:36:29.310Z

Almenon:

Py is not installed for per-user installations I believe.

It defaults to install for all users, but you can deselect the checkbox on the first page of the installer and it will be installed and put on PATH for the current user.

Almenon:

Do you happen to remember what old versions of python did this? Py2? Maybe this could be fixed by checking if python is in the system path already and warning if so?

Python 3.4 is the latest version that defaults to a system install.

The main thing such a warning needs is a detailed help document for users to follow to fix it, which has to include the variety of scenarios that may have caused them to end up with another Python on PATH (including unrelated applications that embed a copy). Once we have that document, and it’s been reviewed/contributed to by those with an interest here, we can look at modifying the default in the case where there is no python.exe to be superseded by (or to supersede) the new entry.

Almenon:

CVE-2020-8315 doesn’t affect windows 8 or later. Are there other vulnerabilities that effect more recent windows versions?

The CVE was just an example. Any application may be impacted by having its python3.dll or vcruntime140.dll overridden, or any DLL installed by a package into the prefix directory. The only way we can avoid breaking other apps is to avoid putting anything on PATH besides our main executable, which requires some (breaking) changes to move the filesystem layout around and potentially change the python.exe on PATH (incompatibly) into a redirector.

netaid270 · 2021-03-22T10:45:28.914Z

eryksun:

This is particularly problematic if both the 3.7 and 3.8 app distributions are installed. The “python.exe” name will be registered to run the “python.exe” binary of the 3.7 app. Executing this binary directly from a normal session context is only allowed if the user’s corresponding appexec link in “%LocalAppData%\Microsoft\WindowsApps” resolves to the same file. But when the 3.8 app is installed, the user’s “python.exe” appexec link instead resolves to the path of the 3.8 binary. Consequently running python from the Win+R dialog or .os.startfile(‘python’)` will fail with a permission error.

Yeah after fixing this, it is awesome thanks python