jit.c may be built with an incomplete jit_stencils.h

@brandtbucher

Bug report

Bug description:

As discussed in https://discuss.python.org/t/pep-744-jit-compilation/50756/42 and further with @brandtbucher, I've tried to change the Fedora's Python 3.13 package to utilize --enable-experimental-jit=yes-off.

As a starting point, I took the 3.13.0b1-1 package from:

https://src.fedoraproject.org/rpms/python3.13/blob/1c24374362/f/python3.13.spec

And I changed:

diff --git a/python3.13.spec b/python3.13.spec
index 4bfe050..68b357d 100644
--- a/python3.13.spec
+++ b/python3.13.spec
@@ -304,6 +304,9 @@ BuildRequires: tcl-devel
 BuildRequires: tk-devel
 BuildRequires: tzdata
 
+BuildRequires: clang >= 18
+BuildRequires: llvm >= 18
+
 %if %{with valgrind}
 BuildRequires: valgrind-devel
 %endif
@@ -920,12 +923,12 @@ BuildPython() {
 # option produces too many warnings when compiling at the O0 optimization level.
 # See also: https://bugzilla.redhat.com/show_bug.cgi?id=1818857
 BuildPython debug \
-  "--without-ensurepip --with-pydebug" \
+  "--without-ensurepip --with-pydebug --enable-experimental-jit=yes-off" \
   "%{optflags_debug}"
 %endif # with debug_build
 
 BuildPython optimized \
-  "--without-ensurepip %{optimizations_flag}" \
+  "--without-ensurepip --enable-experimental-jit=yes-off %{optimizations_flag}" \
   "%{optflags_optimized}"
 
 %if %{with freethreading_build} && %{with debug_build}

That means I added --enable-experimental-jit=yes-off to the debug and optimized non-free-threaded builds and some extra build dependencies.

Running the build in Fedora Rawhide (41) chroot resulted in this error during the debug build (which happens first):

python3.13 /builddir/build/BUILD/Python-3.13.0b1/Tools/jit/build.py x86_64-redhat-linux-gnu --debug
==============================================================
JIT support for x86_64-redhat-linux-gnu is still experimental!
           Please report any issues you encounter.            
==============================================================
python3.13 /builddir/build/BUILD/Python-3.13.0b1/Tools/jit/build.py x86_64-redhat-linux-gnu --debug
==============================================================
JIT support for x86_64-redhat-linux-gnu is still experimental!
           Please report any issues you encounter.            
==============================================================
gcc -c -fno-strict-overflow -Wsign-compare -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -DDYNAMIC_ANNOTATIONS_ENABLED=1 -fcf-protection -fexceptions  -fcf-protection -fexceptions   -fcf-protection -fexceptions   -O0 -Wno-cpp -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer  -D_GNU_SOURCE -fPIC -fwrapv -D_Py_TIER2=3 -D_Py_JIT -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer  -D_GNU_SOURCE -fPIC -fwrapv -O0 -Wno-cpp -I/builddir/build/BUILD/Python-3.13.0b1/Include/internal -I/builddir/build/BUILD/Python-3.13.0b1/Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I/builddir/build/BUILD/Python-3.13.0b1/Include   -fPIC -DPy_BUILD_CORE -o Python/jit.o /builddir/build/BUILD/Python-3.13.0b1/Python/jit.c
/builddir/build/BUILD/Python-3.13.0b1/Python/jit.c: In function ‘emit__MATCH_MAPPING’:
/builddir/build/BUILD/Python-3.13.0b1/Python/jit.c:395:1: error: expected expression before ‘int’
  395 | int
      | ^~~
/builddir/build/BUILD/Python-3.13.0b1/Python/jit.c:479:1: error: expected declaration or statement at end of input
  479 | }
      | ^
In file included from /builddir/build/BUILD/Python-3.13.0b1/Python/jit.c:392:
./jit_stencils.h:30254:25: warning: unused variable ‘data_body’ [-Wunused-variable]
30254 |     const unsigned char data_body[24] = {
      |                         ^~~~~~~~~
./jit_stencils.h:30241:25: warning: unused variable ‘code_body’ [-Wunused-variable]
30241 |     const unsigned char code_body[53] = {
      |                         ^~~~~~~~~
/builddir/build/BUILD/Python-3.13.0b1/Python/jit.c: At top level:
/builddir/build/BUILD/Python-3.13.0b1/Python/jit.c:87:1: warning: ‘mark_executable’ defined but not used [-Wunused-function]
   87 | mark_executable(unsigned char *memory, size_t size)
      | ^~~~~~~~~~~~~~~
/builddir/build/BUILD/Python-3.13.0b1/Python/jit.c:70:1: warning: ‘jit_free’ defined but not used [-Wunused-function]
   70 | jit_free(unsigned char *memory, size_t size)
      | ^~~~~~~~
/builddir/build/BUILD/Python-3.13.0b1/Python/jit.c:49:1: warning: ‘jit_alloc’ defined but not used [-Wunused-function]
   49 | jit_alloc(size_t size)
      | ^~~~~~~~~
make: *** [Makefile:3018: Python/jit.o] Error 1
make: *** Waiting for unfinished jobs....

Full logs:

@brandtbucher suggested that "perhaps several builds are happening at once for some reason. If so, they could be stepping on each other and maybe truncating the generated file."

Anyway, here is the generated jit_stencils.h file as it was present after the failure.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

I've test-built the package in the Fedora build system.

https://koji.fedoraproject.org/koji/taskinfo?taskID=117554410 (this will eventually be garbage collected)

Some architectures succeeded and some failed with:

python3.13 /builddir/build/BUILD/Python-3.13.0b1/Tools/jit/build.py i686-redhat-linux-gnu --debug
usage: build.py [-h] [-d] [-f] [-v] target
build.py: error: argument target: invalid get_target value: 'i686-redhat-linux-gnu'

python3.13 /builddir/build/BUILD/Python-3.13.0b1/Tools/jit/build.py powerpc64le-redhat-linux-gnu --debug
usage: build.py [-h] [-d] [-f] [-v] target
build.py: error: argument target: invalid get_target value: 'powerpc64le-redhat-linux-gnu'

python3.13 /builddir/build/BUILD/Python-3.13.0b1/Tools/jit/build.py s390x-redhat-linux-gnu --debug
usage: build.py [-h] [-d] [-f] [-v] target
build.py: error: argument target: invalid get_target value: 's390x-redhat-linux-gnu'

I suppose this is expected.

But the problem reported here did not happen. Perhaps it only happens sometimes... ? I will trigger a couple of x86_64 builds to see if I can reproduce this again.

Theory:

I noticed we currently don't do an atomic rename of jit_stencils.h, and just write it incrementally instead. I think if make sees the partially-written file, it'll start compiling jit.c and the bottom of the included file will be missing?

Out of 10 builds, I did not reproduce this again.

I suppose this is expected.

Yeah, we're not attempting to support anything that isn't at least in PEP 11 right now.

But the problem reported here did not happen. Perhaps it only happens sometimes... ?

Well, yeah, these ones are intentionally failing early since we won't even attempt to build the JIT. So no jit_stencils.h is generated, and jit.c is never compiled, and the build stops there.

By the way, it appears my hunch is correct. With careful use of parallel build and manually inserted time.sleep calls, I've been able to coax out similar errors from the compiler (from including a half-generated jit_stencils.h). It happens if jit.c is scheduled to build after the file has been opened for writing but before it's finished writing.

So writing to a different file jit_stencils.h.new and doing a rename after is probably the fix here.

GH-118943: Fix a race condition when generating jit_stencils.h (GH-11…

pythonGH-118943: Fix a race condition when generating jit_stencils.h (p…

[3.13] GH-118943: Fix a race condition when generating jit_stencils.h (…

FWIW I just seen:

python3.13 /builddir/build/BUILD/python3.13-3.13.0_b2-build/Python-3.13.0b2/Tools/jit/build.py aarch64-redhat-linux-gnu --debug
===============================================================
JIT support for aarch64-redhat-linux-gnu is still experimental!
            Please report any issues you encounter.            
===============================================================
python3.13 /builddir/build/BUILD/python3.13-3.13.0_b2-build/Python-3.13.0b2/Tools/jit/build.py aarch64-redhat-linux-gnu --debug
===============================================================
JIT support for aarch64-redhat-linux-gnu is still experimental!
            Please report any issues you encounter.            
===============================================================
Traceback (most recent call last):
  File "/builddir/build/BUILD/python3.13-3.13.0_b2-build/Python-3.13.0b2/Tools/jit/build.py", line 28, in <module>
    args.target.build(pathlib.Path.cwd(), comment=comment, force=args.force)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/python3.13-3.13.0_b2-build/Python-3.13.0b2/Tools/jit/_targets.py", line 224, in build
    jit_stencils_new.replace(jit_stencils)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/pathlib/_local.py", line 780, in replace
    os.replace(self, target)
    ~~~~~~~~~~^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/builddir/build/BUILD/python3.13-3.13.0_b2-build/Python-3.13.0b2/build/debug/jit_stencils.h.new' -> '/builddir/build/BUILD/python3.13-3.13.0_b2-build/Python-3.13.0b2/build/debug/jit_stencils.h'
make: *** [Makefile:3026: regen-jit] Error 1
make: *** Waiting for unfinished jobs....

This is slightly better than before, but still not entirely OK. I guess when this happens, it means another process created and moved the new file at the same time.

Perhaps the file needs to have a uniq path, and/or be in tmp?

I got it on x86_64 as well, twice in a row. Both builders had 48 CPUs, so the make call was heavily parallelized.

Should I open a new issue?

pythongh-118943: Remove 'regen-jit' from the 'regen-all' target

pythongh-118943: Remove 'regen-jit' from the 'regen-all' make target

GH-118943: Handle races when moving jit_stencils.h (GH-120690)

pythonGH-118943: Handle races when moving jit_stencils.h (pythonGH-12…

00429: pythonGH-118943: Fix another race condition when generating ji…

pythonGH-118943: Handle races when moving jit_stencils.h (pythonGH-12…

GH-118943: Remove regen-jit from the regen-all make target (GH-122602)

pythonGH-118943: Remove regen-jit from the regen-all make target (pyt…

[3.13] GH-118943: Handle races when moving jit_stencils.h (GH-122709)

[3.13] GH-118943: Remove regen-jit from the regen-all make target (GH…

@hroncok

Thanks again, @hroncok!

pythonGH-118943: Handle races when moving jit_stencils.h (pythonGH-12…

pythonGH-118943: Remove regen-jit from the regen-all make target (pyt…

Navigation Menu

Repository navigation

`jit.c` may be built with an incomplete `jit_stencils.h` #118943

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

6 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

jit.c may be built with an incomplete jit_stencils.h #118943

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

hroncok commented on May 11, 2024

brandtbucher commented on May 11, 2024

hroncok commented on May 11, 2024

brandtbucher commented on May 11, 2024

hroncok commented on Jun 18, 2024

hroncok commented on Jun 18, 2024

6 remaining items

brandtbucher commented on Aug 8, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`jit.c` may be built with an incomplete `jit_stencils.h` #118943