dpctl built with latest oneapi releases (2023.0.0, libsycl 6) fail to submit jobs to cpu

Getting errors such as

/opt/venv/lib/python3.9/site-packages/numba_dpex/compiler.py:811: in __call__
    cfg(*args)
/opt/venv/lib/python3.9/site-packages/numba_dpex/compiler.py:515: in __call__
    self.sycl_queue.submit(
dpctl/_sycl_queue.pyx:716: in dpctl._sycl_queue.SyclQueue.submit
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   dpctl._sycl_queue.SyclKernelSubmitError: Kernel submission to Sycl queue failed.

dpctl/_sycl_queue.pyx:833: SyclKernelSubmitError

while it submits fine with gpu, both with opencl and level_zero backends 🤷

@fcharras

@fcharras It would be useful to know whether the compiler/tbb from oneAPI are activated (if so what is the output of sycl-ls), and what version of dpctl are you using (0.13.0 from Intel channel, or 0.14.0dev1 from dppy/label/dev)?

What is the output of python -c "import dpctl; dpctl.lsplatform(verbosity=2)"?

This build used to work only relying on python libraries such as dpcpp-cpp-rt and mkl-dpcpp + adding bin-llvm folder from the oneapi basekit to PATH.

Those packages are correctly bumped

root@3f55c48cd427:~/sklearn-numba-dpex# pip list |grep 2023 
daal                 2023.0.0
daal4py              2023.0.0
dpcpp-cpp-rt         2023.0.0
intel-cmplr-lib-rt   2023.0.0
intel-cmplr-lic-rt   2023.0.0
intel-fortran-rt     2023.0.0
intel-opencl-rt      2023.0.0
intel-openmp         2023.0.0
mkl                  2023.0.0
mkl-dpcpp            2023.0.0
scikit-learn-intelex 2023.0.0
root@3f55c48cd427:~/sklearn-numba-dpex# pip list |grep 2022
pytz                 2022.7

I tried activating entirely the oneapi basekit

source setvars.sh

but then get the same results.

root@3f55c48cd427:~/sklearn-numba-dpex# sycl-ls
[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device 1.2 [2022.15.12.0.01_081451]
[opencl:cpu:1] Intel(R) OpenCL, 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz 3.0 [2022.15.12.0.01_081451]
[opencl:gpu:2] Intel(R) OpenCL HD Graphics, Intel(R) Iris(R) Xe Graphics [0x9a49] 3.0 [22.38.24278]
[opencl:acc:3] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device 1.2 [2022.14.8.0.04_160000]
[opencl:cpu:4] Intel(R) OpenCL, 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz 3.0 [2022.14.8.0.04_160000]
[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Iris(R) Xe Graphics [0x9a49] 1.3 [1.3.24278]

root@3f55c48cd427:~/sklearn-numba-dpex# python -c "import dpctl; dpctl.lsplatform(verbosity=2)"
Platform  0 ::
    Name        Intel(R) FPGA Emulation Platform for OpenCL(TM)
    Version     OpenCL 1.2 Intel(R) FPGA SDK for OpenCL(TM), Version 20.3
    Vendor      Intel(R) Corporation
    Backend     opencl
    Num Devices 1
      # 0
        Name                Intel(R) FPGA Emulation Device
        Version             2022.15.12.0.01_081451
        Filter string       opencl:accelerator:0
Platform  1 ::
    Name        Intel(R) OpenCL
    Version     OpenCL 3.0 LINUX
    Vendor      Intel(R) Corporation
    Backend     opencl
    Num Devices 1
      # 0
        Name                11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
        Version             2022.15.12.0.01_081451
        Filter string       opencl:cpu:0
Platform  2 ::
    Name        Intel(R) OpenCL HD Graphics
    Version     OpenCL 3.0 
    Vendor      Intel(R) Corporation
    Backend     opencl
    Num Devices 1
      # 0
        Name                Intel(R) Iris(R) Xe Graphics [0x9a49]
        Version             22.38.24278
        Filter string       opencl:gpu:0
Platform  3 ::
    Name        Intel(R) FPGA Emulation Platform for OpenCL(TM)
    Version     OpenCL 1.2 Intel(R) FPGA SDK for OpenCL(TM), Version 20.3
    Vendor      Intel(R) Corporation
    Backend     opencl
    Num Devices 1
      # 0
        Name                Intel(R) FPGA Emulation Device
        Version             2022.14.8.0.04_160000
        Filter string       opencl:accelerator:1
Platform  4 ::
    Name        Intel(R) OpenCL
    Version     OpenCL 3.0 LINUX
    Vendor      Intel(R) Corporation
    Backend     opencl
    Num Devices 1
      # 0
        Name                11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
        Version             2022.14.8.0.04_160000
        Filter string       opencl:cpu:1
Platform  5 ::
    Name        Intel(R) Level-Zero
    Version     1.3
    Vendor      Intel(R) Corporation
    Backend     ext_oneapi_level_zero
    Num Devices 1
      # 0
        Name                Intel(R) Iris(R) Xe Graphics [0x9a49]
        Version             1.3.24278
        Filter string       level_zero:gpu:0

oneapi basekit is also bumped accordingly

root@3f55c48cd427:/opt/intel/oneapi/compiler# ls
2023.0.0  latest

The version of dpctl is 0.14.0dev1, built from source. Maybe the build instructions changed recently ?

I'll try a conda install to compare the results.

The issue is that DPC++ 2023 runtime is trying to submit a kernel to queue targeting CPU SYCL device built for DPC++ 2022 toolchain, the one with "2022.14.8.0.04_160000".

The DPC++ RT most likely picks it up from ICD files in /etc/OpenCL/vendors. If my hunch is correct, cat /etc/OpenCL/vendors/intel64.icd would show a path into /opt/intel/oneapi/compiler/2022.*.

There are several solutions, other directly choose CPU device with driver version "2022.15.12.0.01_081451", or find a way to hide CPU/FPGA-emu drivers from 2022 installation (say uninstall older compiler).

sudo apt list --installed | grep oneapi-dpcpp-cpp | grep 2022

and then sudo apt remove those listed.

I don't think I have older compilers installed (since the docker image is rebuilt from scratch after all bumps), and the issue persists when specifically selecting the cpu with the good driver version.

Also, some kernels continues running fine. (see https://github.com/soda-inria/sklearn-numba-dpex/actions/runs/3733104423/jobs/6333449877 )

I'll try setting up a minimal reproducer to understand what specifically triggers kernel rejection. It seems to be related to work_group_size: the test pipeline seems to hint that for cpu kernels submitted with a group size equal to max_work_group_size are rejected.

Remaining errors encountered are detailed in IntelPython/numba-dpex#868 IntelPython/numba-dpex#867 and IntelPython/numba-dpex#865 with workarounds.

Unless you want to keep track of it here I think we can close this issue.

@fcharras

Thanks you @fcharras, closing

Navigation Menu

Repository navigation

`dpctl` built with latest `oneapi` releases (`2023.0.0`, `libsycl 6`) fail to submit jobs to `cpu` #1023

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

dpctl built with latest oneapi releases (2023.0.0, libsycl 6) fail to submit jobs to cpu #1023

Description

Activity

oleksandr-pavlyk commented on Dec 28, 2022

fcharras commented on Dec 28, 2022

fcharras commented on Dec 28, 2022

fcharras commented on Dec 28, 2022

oleksandr-pavlyk commented on Dec 28, 2022

fcharras commented on Jan 7, 2023

fcharras commented on Jan 11, 2023

oleksandr-pavlyk commented on Jan 11, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`dpctl` built with latest `oneapi` releases (`2023.0.0`, `libsycl 6`) fail to submit jobs to `cpu` #1023