gh-106045: Keep searching from venv home when the resolved base executable has no stdlib by itamaro · Pull Request #160123 · python/cpython

itamaro · 2026-10-21T21:59:43Z

GH-156483 made getpath locate a venv's base installation relative to the resolved base executable instead of the home key in pyvenv.cfg. When the resolved executable lives outside the installation tree (e.g. a launcher binary symlinked into it), nothing is found there and discovery falls back to the compiled-in PREFIX. The venv then either fails to start or silently runs with another installation's standard library.

This PR:

Keeps gh-106045: Fix venv path discovery for symlinked executables #156483's behavior when the resolved executable's directory locates a standard library (pybuilddir.txt, the stdlib zip, or os.py). Otherwise it keeps searching from home, as before gh-106045: Fix venv path discovery for symlinked executables #156483.
In that fallback case, still searches the resolved executable's directory for lib-dynload when determining exec_prefix: after prefix, before home and the compiled-in EXEC_PREFIX. Layouts with a separate exec prefix (e.g. ./configure --exec-prefix) then keep loading extension modules from the right place, and a stale lib-dynload near the home symlink can't take precedence over the one beside the resolved executable.

The tests added by GH-156483 pass unchanged. Three new test_getpath cases cover the launcher-outside-the-installation and separate-exec-prefix layouts, and a misleading lib-dynload near home; each fails without this change.

Reproducer

On main (0a86a34), with a plain ./configure --prefix=$HOME/py-under-test && make && make install:

#!/bin/sh
# Usage: venv-home-repro.sh <installed-prefix> <X.Y>
set -eu
PREFIX=$1; VER=$2
W=$(mktemp -d)
mkdir -p "$W/runtime/bin" "$W/runtime/lib" "$W/launcher"
cp -R "$PREFIX/lib/python$VER" "$W/runtime/lib/"
cp "$PREFIX/bin/python$VER" "$W/launcher/python3"
ln -s "$W/launcher/python3" "$W/runtime/bin/python3"
PYTHONHOME="$W/runtime" "$W/runtime/bin/python3" -m venv --without-pip "$W/venv"
grep '^home' "$W/venv/pyvenv.cfg"
echo "expected base_prefix: $W/runtime"
"$W/venv/bin/python" -c 'import os, sys; print("actual base_prefix:", sys.base_prefix); print("os.py loaded from:", os.__file__)'

Before:

home = /tmp/tmp.NFug9wEr4h/runtime/bin
expected base_prefix: /tmp/tmp.NFug9wEr4h/runtime
actual base_prefix: /home/user/py-under-test
os.py loaded from: /home/user/py-under-test/lib/python3.16/os.py

(With $HOME/py-under-test/lib/python3.16 removed, the venv instead fails with ModuleNotFoundError: No module named 'encodings'.)

After:

home = /tmp/tmp.8Zne2otFrw/runtime/bin
expected base_prefix: /tmp/tmp.8Zne2otFrw/runtime
actual base_prefix: /tmp/tmp.8Zne2otFrw/runtime
os.py loaded from: /tmp/tmp.8Zne2otFrw/runtime/lib/python3.16/os.py

Written with assistance from Claude Code Fable 5.1

Issue: venv base path does not resolve symlinks using realpath() #106045

bedevere-app · 2026-10-21T22:45:49Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

ambv · 2026-10-21T23:38:32Z

If you rebase, the Hypo failure is fixed.

… executable has no stdlib pythonGH-156483 searches for a venv's base installation relative to the resolved base executable. When that executable lives outside the installation tree that 'home' in pyvenv.cfg points into (e.g. a launcher binary symlinked into it), nothing is found there and discovery falls back to the compiled-in PREFIX. Only prefer the resolved executable's directory when a standard library landmark is found from there; otherwise keep searching from 'home'. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A93JzpSGH27i6PNX8re4YV

Even when stdlib discovery falls back to 'home', the resolved base executable may live under a separate exec_prefix (e.g. ./configure --exec-prefix), so search there for platform dependent libraries before falling back to the compiled-in EXEC_PREFIX. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A93JzpSGH27i6PNX8re4YV

itamaro · 2026-10-22T00:41:57Z

I have made the requested changes; please review again.

bedevere-app · 2026-10-22T00:42:04Z

Thanks for making the requested changes!

@ambv: please review the changes made to this pull request.

Co-authored-by: Łukasz Langa <lukasz@langa.pl>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A93JzpSGH27i6PNX8re4YV

ambv · 2026-10-22T18:56:18Z

We can no longer "skip news" for this if it shipped to 3.14.9 and 3.15.1.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A93JzpSGH27i6PNX8re4YV

ambv · 2026-10-22T20:10:07Z

@itamaro nitpick: update the description before landing to match the code.

It says the resolved executable's directory is searched for lib-dynload "after prefix and home". Since your reorder, it's searched after prefix but before home (Modules/getpath.py:636-642). And there's three cases.

itamaro · 2026-10-22T20:25:09Z

@itamaro nitpick: update the description before landing to match the code.

It says the resolved executable's directory is searched for lib-dynload "after prefix and home". Since your reorder, it's searched after prefix but before home (Modules/getpath.py:636-642). And there's three cases.

thanks, updated the description!

miss-islington-app · 2026-10-22T21:35:24Z

Thanks @itamaro for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14, 3.15.
🐍🍒⛏🤖

bedevere-app · 2026-10-22T21:35:42Z

GH-160354 is a backport of this pull request to the 3.15 branch.

bedevere-app · 2026-10-22T21:35:48Z

GH-160355 is a backport of this pull request to the 3.14 branch.

…e executable has no stdlib (GH-160123) (#160355) gh-106045: Keep searching from venv home when the resolved base executable has no stdlib (GH-160123) GH-156483 made `getpath` locate a venv's base installation relative to the *resolved* base executable instead of the `home` key in `pyvenv.cfg`. When the resolved executable lives outside the installation tree (e.g. a launcher binary symlinked into it), nothing is found there and discovery falls back to the compiled-in `PREFIX`. The venv then either fails to start or silently runs with another installation's standard library. This PR: - Keeps GH-156483's behavior when the resolved executable's directory locates a standard library (`pybuilddir.txt`, the stdlib zip, or `os.py`). Otherwise it keeps searching from `home`, as before GH-156483. - In that fallback case, still searches the resolved executable's directory for `lib-dynload` when determining `exec_prefix`: after `prefix`, before `home` and the compiled-in `EXEC_PREFIX`. Layouts with a separate exec prefix (e.g. `./configure --exec-prefix`) then keep loading extension modules from the right place, and a stale `lib-dynload` near the `home` symlink can't take precedence over the one beside the resolved executable. The tests added by GH-156483 pass unchanged. Three new `test_getpath` cases cover the launcher-outside-the-installation and separate-exec-prefix layouts, and a misleading `lib-dynload` near `home`; each fails without this change. <details> <summary>Reproducer</summary> On `main` (0a86a34), with a plain `./configure --prefix=$HOME/py-under-test && make && make install`: ```sh #!/bin/sh # Usage: venv-home-repro.sh <installed-prefix> <X.Y> set -eu PREFIX=$1; VER=$2 W=$(mktemp -d) mkdir -p "$W/runtime/bin" "$W/runtime/lib" "$W/launcher" cp -R "$PREFIX/lib/python$VER" "$W/runtime/lib/" cp "$PREFIX/bin/python$VER" "$W/launcher/python3" ln -s "$W/launcher/python3" "$W/runtime/bin/python3" PYTHONHOME="$W/runtime" "$W/runtime/bin/python3" -m venv --without-pip "$W/venv" grep '^home' "$W/venv/pyvenv.cfg" echo "expected base_prefix: $W/runtime" "$W/venv/bin/python" -c 'import os, sys; print("actual base_prefix:", sys.base_prefix); print("os.py loaded from:", os.__file__)' ``` Before: ``` home = /tmp/tmp.NFug9wEr4h/runtime/bin expected base_prefix: /tmp/tmp.NFug9wEr4h/runtime actual base_prefix: /home/user/py-under-test os.py loaded from: /home/user/py-under-test/lib/python3.16/os.py ``` (With `$HOME/py-under-test/lib/python3.16` removed, the venv instead fails with `ModuleNotFoundError: No module named 'encodings'`.) After: ``` home = /tmp/tmp.8Zne2otFrw/runtime/bin expected base_prefix: /tmp/tmp.8Zne2otFrw/runtime actual base_prefix: /tmp/tmp.8Zne2otFrw/runtime os.py loaded from: /tmp/tmp.8Zne2otFrw/runtime/lib/python3.16/os.py ``` </details> (cherry picked from commit 5b55b50) Co-authored-by: Itamar Oren <itamarost@gmail.com>

…e executable has no stdlib (GH-160123) (#160354) gh-106045: Keep searching from venv home when the resolved base executable has no stdlib (GH-160123) GH-156483 made `getpath` locate a venv's base installation relative to the *resolved* base executable instead of the `home` key in `pyvenv.cfg`. When the resolved executable lives outside the installation tree (e.g. a launcher binary symlinked into it), nothing is found there and discovery falls back to the compiled-in `PREFIX`. The venv then either fails to start or silently runs with another installation's standard library. This PR: - Keeps GH-156483's behavior when the resolved executable's directory locates a standard library (`pybuilddir.txt`, the stdlib zip, or `os.py`). Otherwise it keeps searching from `home`, as before GH-156483. - In that fallback case, still searches the resolved executable's directory for `lib-dynload` when determining `exec_prefix`: after `prefix`, before `home` and the compiled-in `EXEC_PREFIX`. Layouts with a separate exec prefix (e.g. `./configure --exec-prefix`) then keep loading extension modules from the right place, and a stale `lib-dynload` near the `home` symlink can't take precedence over the one beside the resolved executable. The tests added by GH-156483 pass unchanged. Three new `test_getpath` cases cover the launcher-outside-the-installation and separate-exec-prefix layouts, and a misleading `lib-dynload` near `home`; each fails without this change. <details> <summary>Reproducer</summary> On `main` (0a86a34), with a plain `./configure --prefix=$HOME/py-under-test && make && make install`: ```sh #!/bin/sh # Usage: venv-home-repro.sh <installed-prefix> <X.Y> set -eu PREFIX=$1; VER=$2 W=$(mktemp -d) mkdir -p "$W/runtime/bin" "$W/runtime/lib" "$W/launcher" cp -R "$PREFIX/lib/python$VER" "$W/runtime/lib/" cp "$PREFIX/bin/python$VER" "$W/launcher/python3" ln -s "$W/launcher/python3" "$W/runtime/bin/python3" PYTHONHOME="$W/runtime" "$W/runtime/bin/python3" -m venv --without-pip "$W/venv" grep '^home' "$W/venv/pyvenv.cfg" echo "expected base_prefix: $W/runtime" "$W/venv/bin/python" -c 'import os, sys; print("actual base_prefix:", sys.base_prefix); print("os.py loaded from:", os.__file__)' ``` Before: ``` home = /tmp/tmp.NFug9wEr4h/runtime/bin expected base_prefix: /tmp/tmp.NFug9wEr4h/runtime actual base_prefix: /home/user/py-under-test os.py loaded from: /home/user/py-under-test/lib/python3.16/os.py ``` (With `$HOME/py-under-test/lib/python3.16` removed, the venv instead fails with `ModuleNotFoundError: No module named 'encodings'`.) After: ``` home = /tmp/tmp.8Zne2otFrw/runtime/bin expected base_prefix: /tmp/tmp.8Zne2otFrw/runtime actual base_prefix: /tmp/tmp.8Zne2otFrw/runtime os.py loaded from: /tmp/tmp.8Zne2otFrw/runtime/lib/python3.16/os.py ``` </details> (cherry picked from commit 5b55b50) Co-authored-by: Itamar Oren <itamarost@gmail.com>

itamaro requested a review from ambv October 21, 2026 21:59

itamaro requested a review from FFY00 as a code owner October 21, 2026 21:59

itamaro added skip news needs backport to 3.14 bugs and security fixes needs backport to 3.15 bugs and security fixes labels Oct 21, 2026

bedevere-app Bot mentioned this pull request Oct 21, 2026

venv base path does not resolve symlinks using realpath() #106045

Closed

bedevere-app Bot added the awaiting core review label Oct 21, 2026

ambv requested changes Oct 21, 2026

View reviewed changes

Comment thread Modules/getpath.py

bedevere-app Bot added awaiting changes and removed awaiting core review labels Oct 21, 2026

itamaro and others added 2 commits October 21, 2026 17:28

itamaro force-pushed the gh-106045-venv-home-fallback branch from 620ac1e to 72b32a4 Compare October 22, 2026 00:33

bedevere-app Bot added awaiting change review and removed awaiting changes labels Oct 22, 2026

bedevere-app Bot requested a review from ambv October 22, 2026 00:42

ambv reviewed Oct 22, 2026

View reviewed changes

Comment thread Modules/getpath.py Outdated

ambv reviewed Oct 22, 2026

View reviewed changes

Comment thread Lib/test/test_getpath.py

itamaro and others added 2 commits October 22, 2026 11:08

Prefer the resolved base executable for exec_prefix discovery

3efc8e8

Co-authored-by: Łukasz Langa <lukasz@langa.pl>

Fix syntax error and spacing from applied review suggestions

4a13369

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A93JzpSGH27i6PNX8re4YV

ambv removed the skip news label Oct 22, 2026

Add NEWS entry

4c0f76e

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A93JzpSGH27i6PNX8re4YV

ambv approved these changes Oct 22, 2026

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting change review labels Oct 22, 2026

ambv added release-blocker 3.14 bugs and security fixes 3.15 bugs and security fixes labels Oct 22, 2026

github-project-automation Bot added this to Release and Deferred blockers 🚫 Oct 22, 2026

github-project-automation Bot moved this to Todo in Release and Deferred blockers 🚫 Oct 22, 2026

itamaro enabled auto-merge (squash) October 22, 2026 20:26

itamaro merged commit 5b55b50 into python:main Oct 22, 2026
62 of 63 checks passed

github-project-automation Bot moved this from Todo to Done in Release and Deferred blockers 🚫 Oct 22, 2026

bedevere-app Bot removed the awaiting merge label Oct 22, 2026

bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label Oct 22, 2026

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Oct 22, 2026

Navigation Menu

Repository navigation

gh-106045: Keep searching from venv home when the resolved base executable has no stdlib - #160123

gh-106045: Keep searching from venv home when the resolved base executable has no stdlib#160123
itamaro merged 5 commits into
python:mainfrom
itamaro:gh-106045-venv-home-fallback

itamaro commented Oct 21, 2026 •

edited

Loading

Uh oh!

Uh oh!

bedevere-app Bot commented Oct 21, 2026

Uh oh!

ambv commented Oct 21, 2026

Uh oh!

itamaro commented Oct 22, 2026

Uh oh!

bedevere-app Bot commented Oct 22, 2026

Uh oh!

Uh oh!

Uh oh!

ambv commented Oct 22, 2026

Uh oh!

ambv commented Oct 22, 2026

Uh oh!

itamaro commented Oct 22, 2026

Uh oh!

Uh oh!

miss-islington-app Bot commented Oct 22, 2026

Uh oh!

bedevere-app Bot commented Oct 22, 2026

Uh oh!

bedevere-app Bot commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

itamaro commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

bedevere-app Bot commented Oct 21, 2026

Uh oh!

ambv commented Oct 21, 2026

Uh oh!

itamaro commented Oct 22, 2026

Uh oh!

bedevere-app Bot commented Oct 22, 2026

Uh oh!

Uh oh!

Uh oh!

ambv commented Oct 22, 2026

Uh oh!

ambv commented Oct 22, 2026

Uh oh!

itamaro commented Oct 22, 2026

Uh oh!

Uh oh!

miss-islington-app Bot commented Oct 22, 2026

Uh oh!

bedevere-app Bot commented Oct 22, 2026

Uh oh!

bedevere-app Bot commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

itamaro commented Oct 21, 2026 •

edited

Loading