Created
February 11, 2026 17:09
-
-
Save Flamefire/9fe31719013c4593c93c3cf7289e3e7e to your computer and use it in GitHub Desktop.
(partial) EasyBuild log for failed build of /tmp/easybuild-tmp/eb-afuu32o6/files_pr25240/p/PyTorch/PyTorch-2.9.1-foss-2024a.eb (PR(s) #25240)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Running 1 items in this shard | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod7_cpu_float32 malloc(): smallbin double linked list corrupted | |
| Fatal Python error: Aborted | |
| Thread 0x00007f37f0fdb640 (most recent call first): | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1010 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1073 in _bootstrap_inner | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1030 in _bootstrap | |
| Current thread 0x00007f4244c2f740 (most recent call first): | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmpuhz6qbkg/ds/cds75bclpylkos3bl3nmorboslgvzrvvlbxfkq3pqfvpjeyts63k.py", line 1328 in call | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_inductor/output_code.py", line 613 in __call__ | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 526 in wrapper | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 724 in inner_fn | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/utils.py", line 129 in call_func_at_runtime_with_args | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 353 in runtime_wrapper | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1130 in forward | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1044 in _fn | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_dynamo/external_utils.py", line 66 in inner | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 832 in compile_wrapper | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 433 in run_test | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 950 in test_kv_batch_broadcast | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 426 in instantiated_test | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3224 in wrapper | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 589 in _callTestMethod | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 634 in run | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3376 in _run_custom | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3406 in run | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 517 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 690 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/unittest.py", line 344 in runtest | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 173 in pytest_runtest_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 241 in <lambda> | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 240 in call_and_report | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 135 in runtestprotocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/pytest-rerunfailures/15.0-GCCcore-13.3.0/lib/python3.12/site-packages/pytest_rerunfailures.py", line 566 in pytest_runtest_protocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 364 in pytest_runtestloop | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 339 in _main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/config/__init__.py", line 178 in main | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 1299 in run_tests | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_dynamo/test_case.py", line 62 in run_tests | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_inductor/test_case.py", line 15 in run_tests | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 2101 in <module> | |
| Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, gmpy2.gmpy2, psutil._psutil_linux, psutil._psutil_posix, markupsafe._speedups, c6chp3uecr3aj2s2wa7mncm7pfypjz4ajet23jntpnhwevs3kpq6.kernel (total: 29) | |
| Got exit code -6 (SIGIOT) | |
| Retrying single test... | |
| Test results will be stored in test-reports/python-pytest/inductor.test_flex_decoding/inductor.test_flex_decoding-d9b6f21ac3435486.xml | |
| ============================= test session starts ============================== | |
| platform linux -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0 -- /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/bin/python | |
| cachedir: .pytest_cache | |
| hypothesis profile 'dev' -> database=None, max_examples=10, suppress_health_check=[HealthCheck.too_slow] | |
| rootdir: /dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1 | |
| configfile: pytest.ini | |
| plugins: xdist-3.6.1, subtests-0.13.1, rerunfailures-15.0, flakefinder-1.1.0, hypothesis-6.103.1, typeguard-4.3.0 | |
| collecting ... collected 400 items / 399 deselected / 1 selected | |
| stepcurrent: skipping 213 already run items. Running only test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod7_cpu_float32 | |
| Running 1 items in this shard | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod7_cpu_float32 Command took >120min, returning 124 | |
| Got exit code 124 | |
| Test failed consistently, continuing with the rest of the tests due to continue-through-error being set | |
| Test results will be stored in test-reports/python-pytest/inductor.test_flex_decoding/inductor.test_flex_decoding-652b5b9f9dd55df1.xml | |
| ============================= test session starts ============================== | |
| platform linux -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0 -- /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/bin/python | |
| cachedir: .pytest_cache | |
| hypothesis profile 'dev' -> database=None, max_examples=10, suppress_health_check=[HealthCheck.too_slow] | |
| rootdir: /dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1 | |
| configfile: pytest.ini | |
| plugins: xdist-3.6.1, subtests-0.13.1, rerunfailures-15.0, flakefinder-1.1.0, hypothesis-6.103.1, typeguard-4.3.0 | |
| collecting ... collected 400 items / 218 deselected / 182 selected | |
| stepcurrent: skipping 214 already run items. | |
| Running 182 items in this shard | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod8_cpu_float32 malloc(): smallbin double linked list corrupted | |
| Fatal Python error: Aborted | |
| Thread 0x00007fc52cff1640 (most recent call first): | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1010 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1073 in _bootstrap_inner | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1030 in _bootstrap | |
| Current thread 0x00007fcf9315a740 (most recent call first): | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmpdvhxrl6q/dc/cdcfeael3jogyv7oa2uv4mq2mxzkcaj5k4kclrrzevota3wcjzul.py", line 1312 in call | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_inductor/output_code.py", line 613 in __call__ | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 526 in wrapper | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 724 in inner_fn | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/utils.py", line 129 in call_func_at_runtime_with_args | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 353 in runtime_wrapper | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1130 in forward | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1044 in _fn | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_dynamo/external_utils.py", line 66 in inner | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 832 in compile_wrapper | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 433 in run_test | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 950 in test_kv_batch_broadcast | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 426 in instantiated_test | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3224 in wrapper | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 589 in _callTestMethod | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 634 in run | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3376 in _run_custom | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3406 in run | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 517 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 690 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/unittest.py", line 344 in runtest | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 173 in pytest_runtest_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 241 in <lambda> | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 240 in call_and_report | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 135 in runtestprotocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/pytest-rerunfailures/15.0-GCCcore-13.3.0/lib/python3.12/site-packages/pytest_rerunfailures.py", line 566 in pytest_runtest_protocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 364 in pytest_runtestloop | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 339 in _main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/config/__init__.py", line 178 in main | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 1299 in run_tests | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_dynamo/test_case.py", line 62 in run_tests | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_inductor/test_case.py", line 15 in run_tests | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 2101 in <module> | |
| Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, gmpy2.gmpy2, psutil._psutil_linux, psutil._psutil_posix, markupsafe._speedups, cifrxnkidwdocxxw7op67ol6r2rgu5ylv4g6nh667pogaoxetiqw.kernel (total: 29) | |
| Got exit code -6 (SIGIOT) | |
| Retrying single test... | |
| Test results will be stored in test-reports/python-pytest/inductor.test_flex_decoding/inductor.test_flex_decoding-c808fe609f0e0c74.xml | |
| ============================= test session starts ============================== | |
| platform linux -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0 -- /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/bin/python | |
| cachedir: .pytest_cache | |
| hypothesis profile 'dev' -> database=None, max_examples=10, suppress_health_check=[HealthCheck.too_slow] | |
| rootdir: /dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1 | |
| configfile: pytest.ini | |
| plugins: xdist-3.6.1, subtests-0.13.1, rerunfailures-15.0, flakefinder-1.1.0, hypothesis-6.103.1, typeguard-4.3.0 | |
| collecting ... collected 400 items / 399 deselected / 1 selected | |
| stepcurrent: skipping 214 already run items. Running only test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod8_cpu_float32 | |
| Running 1 items in this shard | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod8_cpu_float32 free(): invalid size | |
| Fatal Python error: Aborted | |
| Thread 0x00007f0175d86640 (most recent call first): | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1010 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1073 in _bootstrap_inner | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1030 in _bootstrap | |
| Current thread 0x00007f0be5b63740 (most recent call first): | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmp61s73jjy/dc/cdcfeael3jogyv7oa2uv4mq2mxzkcaj5k4kclrrzevota3wcjzul.py", line 1312 in call | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_inductor/output_code.py", line 613 in __call__ | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 526 in wrapper | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 724 in inner_fn | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/utils.py", line 129 in call_func_at_runtime_with_args | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 353 in runtime_wrapper | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1130 in forward | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1044 in _fn | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_dynamo/external_utils.py", line 66 in inner | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 832 in compile_wrapper | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 433 in run_test | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 950 in test_kv_batch_broadcast | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 426 in instantiated_test | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3224 in wrapper | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 589 in _callTestMethod | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 634 in run | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3376 in _run_custom | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3406 in run | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 517 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 690 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/unittest.py", line 344 in runtest | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 173 in pytest_runtest_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 241 in <lambda> | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 240 in call_and_report | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 135 in runtestprotocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/pytest-rerunfailures/15.0-GCCcore-13.3.0/lib/python3.12/site-packages/pytest_rerunfailures.py", line 566 in pytest_runtest_protocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 364 in pytest_runtestloop | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 339 in _main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/config/__init__.py", line 178 in main | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 1299 in run_tests | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_dynamo/test_case.py", line 62 in run_tests | |
| File "/tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg/lib/python3.12/site-packages/torch/_inductor/test_case.py", line 15 in run_tests | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 2101 in <module> | |
| Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, gmpy2.gmpy2, psutil._psutil_linux, psutil._psutil_posix, markupsafe._speedups, cifrxnkidwdocxxw7op67ol6r2rgu5ylv4g6nh667pogaoxetiqw.kernel (total: 29) | |
| Got exit code -6 (SIGIOT) | |
| Retrying single test... | |
| Test results will be stored in test-reports/python-pytest/inductor.test_flex_decoding/inductor.test_flex_decoding-3e45722c789007e8.xml | |
| ============================= test session starts ============================== | |
| platform linux -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0 -- /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/bin/python | |
| cachedir: .pytest_cache | |
| hypothesis profile 'dev' -> database=None, max_examples=10, suppress_health_check=[HealthCheck.too_slow] | |
| rootdir: /dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1 | |
| configfile: pytest.ini | |
| plugins: xdist-3.6.1, subtests-0.13.1, rerunfailures-15.0, flakefinder-1.1.0, hypothesis-6.103.1, typeguard-4.3.0 | |
| collecting ... collected 400 items / 399 deselected / 1 selected | |
| stepcurrent: skipping 214 already run items. Running only test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod8_cpu_float32 | |
| Running 1 items in this shard | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod8_cpu_float32 Command took >120min, returning 124 | |
| Got exit code 124 | |
| Test failed consistently, continuing with the rest of the tests due to continue-through-error being set | |
| Test results will be stored in test-reports/python-pytest/inductor.test_flex_decoding/inductor.test_flex_decoding-c5d1e9774784417a.xml | |
| ============================= test session starts ============================== | |
| platform linux -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0 -- /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/bin/python | |
| cachedir: .pytest_cache | |
| hypothesis profile 'dev' -> database=None, max_examples=10, suppress_health_check=[HealthCheck.too_slow] | |
| rootdir: /dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1 | |
| configfile: pytest.ini | |
| plugins: xdist-3.6.1, subtests-0.13.1, rerunfailures-15.0, flakefinder-1.1.0, hypothesis-6.103.1, typeguard-4.3.0 | |
| collecting ... collected 400 items / 219 deselected / 181 selected | |
| stepcurrent: skipping 215 already run items. | |
| Running 181 items in this shard | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_larger_block_mask_bug_float32_cpu_float32 PASSED [27.1242s] [ 0%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_load_from_bias_head_seq_batch_float32_cpu_float32 PASSED [11.9744s] [ 1%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_load_from_bias_seq_batch_float32_cpu_float32 PASSED [10.0759s] [ 1%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_load_from_bias_seq_only_float32_cpu_float32 PASSED [9.7292s] [ 2%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_logsumexp_correctness_float32_score_mod0_cpu_float32 SKIPPED [0.0006s] [ 2%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_logsumexp_correctness_float32_score_mod1_cpu_float32 SKIPPED [0.0002s] [ 3%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_logsumexp_only_return_cpu SKIPPED [0.0002s] [ 3%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_max_autotune_cpu PASSED [15.8715s] [ 4%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_max_autotune_with_captured_cpu PASSED [10.2237s] [ 4%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_mixed_dtypes_fails_cpu PASSED [0.0209s] [ 5%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_multiple_score_mod_calls2_cpu PASSED [5.6279s] [ 6%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_multiple_score_mod_calls_cpu PASSED [5.2962s] [ 6%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_multiple_score_mod_calls_paged_attention2_cpu PASSED [6.9756s] [ 7%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_multiple_score_mod_calls_paged_attention_cpu PASSED [6.1362s] [ 7%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_njt_causal_float32_cpu_float32 PASSED [10.2381s] [ 8%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_divisible_multi_token_offset_mask_cpu PASSED [4.2003s] [ 8%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_divisible_multi_token_offset_mask_with_captured_buffer_cpu SKIPPED [0.0003s] [ 9%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_divisible_offset_mask_cpu PASSED [4.5295s] [ 9%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_divisible_offset_mask_with_captured_buffer_cpu PASSED [4.4202s] [ 10%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod0_float32_head_dims0_cpu_float32 PASSED [9.4789s] [ 11%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod0_float32_head_dims1_cpu_float32 PASSED [9.5339s] [ 11%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod1_float32_head_dims0_cpu_float32 PASSED [9.5304s] [ 12%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod1_float32_head_dims1_cpu_float32 PASSED [9.9620s] [ 12%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod2_float32_head_dims0_cpu_float32 PASSED [9.5097s] [ 13%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod2_float32_head_dims1_cpu_float32 PASSED [9.4629s] [ 13%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod3_float32_head_dims0_cpu_float32 PASSED [9.5393s] [ 14%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod3_float32_head_dims1_cpu_float32 PASSED [9.7863s] [ 14%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod4_float32_head_dims0_cpu_float32 PASSED [9.6158s] [ 15%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod4_float32_head_dims1_cpu_float32 PASSED [9.5402s] [ 16%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod5_float32_head_dims0_cpu_float32 PASSED [9.8553s] [ 16%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod5_float32_head_dims1_cpu_float32 PASSED [9.5243s] [ 17%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod6_float32_head_dims0_cpu_float32 PASSED [9.5599s] [ 17%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod6_float32_head_dims1_cpu_float32 PASSED [9.5666s] [ 18%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod7_float32_head_dims0_cpu_float32 PASSED [9.9738s] [ 18%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod7_float32_head_dims1_cpu_float32 PASSED [9.5532s] [ 19%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod8_float32_head_dims0_cpu_float32 PASSED [9.4774s] [ 19%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod8_float32_head_dims1_cpu_float32 PASSED [9.5046s] [ 20%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_sparse_mulitple_block_size_cpu PASSED [9.0708s] [ 20%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_not_pw_of_two_cpu SKIPPED [0.0002s] [ 21%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_padded_dense_causal_float32_cpu_float32 PASSED [9.7319s] [ 22%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims0_page_size_128_cpu_float32 PASSED [5.0003s] [ 22%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims0_page_size_256_cpu_float32 PASSED [5.0834s] [ 23%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims0_page_size_64_cpu_float32 PASSED [5.0759s] [ 23%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims1_page_size_128_cpu_float32 PASSED [5.0139s] [ 24%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims1_page_size_256_cpu_float32 PASSED [5.0148s] [ 24%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims1_page_size_64_cpu_float32 PASSED [5.3010s] [ 25%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims2_page_size_128_cpu_float32 PASSED [5.1116s] [ 25%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims2_page_size_256_cpu_float32 PASSED [5.1546s] [ 26%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims2_page_size_64_cpu_float32 PASSED [5.1157s] [ 27%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims0_page_size_128_cpu_float32 PASSED [5.0358s] [ 27%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims0_page_size_256_cpu_float32 PASSED [5.0243s] [ 28%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims0_page_size_64_cpu_float32 PASSED [5.0430s] [ 28%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims1_page_size_128_cpu_float32 PASSED [5.0869s] [ 29%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims1_page_size_256_cpu_float32 PASSED [5.0887s] [ 29%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims1_page_size_64_cpu_float32 PASSED [5.3444s] [ 30%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims2_page_size_128_cpu_float32 PASSED [5.1995s] [ 30%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims2_page_size_256_cpu_float32 PASSED [5.1882s] [ 31%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims2_page_size_64_cpu_float32 PASSED [5.1631s] [ 32%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims0_page_size_128_cpu_float32 PASSED [5.0541s] [ 32%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims0_page_size_256_cpu_float32 PASSED [5.1013s] [ 33%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims0_page_size_64_cpu_float32 PASSED [5.0956s] [ 33%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims1_page_size_128_cpu_float32 PASSED [5.0665s] [ 34%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims1_page_size_256_cpu_float32 PASSED [5.0599s] [ 34%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims1_page_size_64_cpu_float32 PASSED [5.3756s] [ 35%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims2_page_size_128_cpu_float32 PASSED [5.1713s] [ 35%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims2_page_size_256_cpu_float32 PASSED [5.1804s] [ 36%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims2_page_size_64_cpu_float32 PASSED [5.1684s] [ 37%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims0_page_size_128_cpu_float32 PASSED [5.1235s] [ 37%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims0_page_size_256_cpu_float32 PASSED [5.1429s] [ 38%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims0_page_size_64_cpu_float32 PASSED [5.1402s] [ 38%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims1_page_size_128_cpu_float32 PASSED [5.1294s] [ 39%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims1_page_size_256_cpu_float32 PASSED [5.1188s] [ 39%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims1_page_size_64_cpu_float32 PASSED [5.4541s] [ 40%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims2_page_size_128_cpu_float32 PASSED [5.2392s] [ 40%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims2_page_size_256_cpu_float32 PASSED [5.2451s] [ 41%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims2_page_size_64_cpu_float32 PASSED [5.2264s] [ 41%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims0_page_size_128_cpu_float32 PASSED [5.1138s] [ 42%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims0_page_size_256_cpu_float32 PASSED [5.1506s] [ 43%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims0_page_size_64_cpu_float32 PASSED [5.1317s] [ 43%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims1_page_size_128_cpu_float32 PASSED [5.0998s] [ 44%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims1_page_size_256_cpu_float32 PASSED [5.1023s] [ 44%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims1_page_size_64_cpu_float32 PASSED [5.4369s] [ 45%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims2_page_size_128_cpu_float32 PASSED [5.1852s] [ 45%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims2_page_size_256_cpu_float32 PASSED [5.1890s] [ 46%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims2_page_size_64_cpu_float32 PASSED [5.1749s] [ 46%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims0_page_size_128_cpu_float32 PASSED [5.0997s] [ 47%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims0_page_size_256_cpu_float32 PASSED [5.1138s] [ 48%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims0_page_size_64_cpu_float32 PASSED [5.0928s] [ 48%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims1_page_size_128_cpu_float32 PASSED [5.0855s] [ 49%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims1_page_size_256_cpu_float32 PASSED [5.1001s] [ 49%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims1_page_size_64_cpu_float32 PASSED [5.4243s] [ 50%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims2_page_size_128_cpu_float32 PASSED [5.1810s] [ 50%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims2_page_size_256_cpu_float32 PASSED [5.1635s] [ 51%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims2_page_size_64_cpu_float32 PASSED [5.1862s] [ 51%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims0_page_size_128_cpu_float32 PASSED [5.1617s] [ 52%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims0_page_size_256_cpu_float32 PASSED [5.1567s] [ 53%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims0_page_size_64_cpu_float32 PASSED [5.1969s] [ 53%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims1_page_size_128_cpu_float32 PASSED [5.2746s] [ 54%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims1_page_size_256_cpu_float32 PASSED [5.1411s] [ 54%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims1_page_size_64_cpu_float32 PASSED [5.4992s] [ 55%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims2_page_size_128_cpu_float32 PASSED [5.2246s] [ 55%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims2_page_size_256_cpu_float32 PASSED [5.2446s] [ 56%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims2_page_size_64_cpu_float32 PASSED [5.2502s] [ 56%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims0_page_size_128_cpu_float32 PASSED [5.1426s] [ 57%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims0_page_size_256_cpu_float32 PASSED [5.1650s] [ 58%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims0_page_size_64_cpu_float32 PASSED [5.1971s] [ 58%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims1_page_size_128_cpu_float32 PASSED [5.2173s] [ 59%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims1_page_size_256_cpu_float32 PASSED [5.1415s] [ 59%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims1_page_size_64_cpu_float32 PASSED [5.4998s] [ 60%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims2_page_size_128_cpu_float32 PASSED [5.2532s] [ 60%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims2_page_size_256_cpu_float32 PASSED [5.2387s] [ 61%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims2_page_size_64_cpu_float32 PASSED [5.2252s] [ 61%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims0_page_size_128_cpu_float32 PASSED [5.1097s] [ 62%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims0_page_size_256_cpu_float32 PASSED [5.2475s] [ 62%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims0_page_size_64_cpu_float32 PASSED [5.2432s] [ 63%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims1_page_size_128_cpu_float32 PASSED [5.2344s] [ 64%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims1_page_size_256_cpu_float32 PASSED [5.1150s] [ 64%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims1_page_size_64_cpu_float32 PASSED [5.0906s] [ 65%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims2_page_size_128_cpu_float32 PASSED [5.5558s] [ 65%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims2_page_size_256_cpu_float32 PASSED [5.3501s] [ 66%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims2_page_size_64_cpu_float32 PASSED [5.1876s] [ 66%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_recompile_changed_score_mod_float32_cpu_float32 PASSED [18.9635s] [ 67%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_seq_masking_float32_cpu_float32 PASSED [9.9786s] [ 67%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_silu_on_score_float32_cpu_float32 PASSED [9.8403s] [ 68%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_skip_odd_keys_float32_cpu_float32 PASSED [9.7691s] [ 69%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s0_head_dims0_cpu_float32 PASSED [8.9388s] [ 69%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s0_head_dims1_cpu_float32 PASSED [8.9597s] [ 70%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s0_head_dims2_cpu_float32 PASSED [9.1528s] [ 70%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s1_head_dims0_cpu_float32 PASSED [8.8891s] [ 71%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s1_head_dims1_cpu_float32 PASSED [9.2724s] [ 71%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s1_head_dims2_cpu_float32 PASSED [9.1657s] [ 72%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s2_head_dims0_cpu_float32 PASSED [8.9350s] [ 72%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s2_head_dims1_cpu_float32 PASSED [8.8304s] [ 73%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s2_head_dims2_cpu_float32 PASSED [9.0941s] [ 74%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s3_head_dims0_cpu_float32 PASSED [9.2947s] [ 74%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s3_head_dims1_cpu_float32 PASSED [8.7994s] [ 75%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s3_head_dims2_cpu_float32 PASSED [9.1905s] [ 75%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s0_head_dims0_cpu_float32 PASSED [8.8217s] [ 76%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s0_head_dims1_cpu_float32 PASSED [8.9285s] [ 76%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s0_head_dims2_cpu_float32 PASSED [9.0569s] [ 77%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s1_head_dims0_cpu_float32 PASSED [8.7564s] [ 77%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s1_head_dims1_cpu_float32 PASSED [9.3574s] [ 78%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s1_head_dims2_cpu_float32 PASSED [9.2438s] [ 79%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s2_head_dims0_cpu_float32 PASSED [8.8083s] [ 79%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s2_head_dims1_cpu_float32 PASSED [8.9068s] [ 80%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s2_head_dims2_cpu_float32 PASSED [9.1847s] [ 80%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s3_head_dims0_cpu_float32 PASSED [8.9455s] [ 81%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s3_head_dims1_cpu_float32 PASSED [9.3623s] [ 81%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s3_head_dims2_cpu_float32 PASSED [9.1908s] [ 82%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s0_head_dims0_cpu_float32 PASSED [8.7914s] [ 82%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s0_head_dims1_cpu_float32 PASSED [8.8266s] [ 83%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s0_head_dims2_cpu_float32 PASSED [9.0952s] [ 83%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s1_head_dims0_cpu_float32 PASSED [8.7029s] [ 84%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s1_head_dims1_cpu_float32 PASSED [9.3023s] [ 85%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s1_head_dims2_cpu_float32 PASSED [9.1272s] [ 85%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s2_head_dims0_cpu_float32 PASSED [8.7640s] [ 86%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s2_head_dims1_cpu_float32 PASSED [8.9222s] [ 86%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s2_head_dims2_cpu_float32 PASSED [9.0651s] [ 87%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s3_head_dims0_cpu_float32 PASSED [8.6922s] [ 87%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s3_head_dims1_cpu_float32 PASSED [9.3882s] [ 88%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s3_head_dims2_cpu_float32 PASSED [9.0810s] [ 88%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s0_head_dims0_cpu_float32 PASSED [8.7816s] [ 89%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s0_head_dims1_cpu_float32 PASSED [8.9457s] [ 90%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s0_head_dims2_cpu_float32 PASSED [9.0676s] [ 90%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s1_head_dims0_cpu_float32 PASSED [8.7771s] [ 91%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s1_head_dims1_cpu_float32 PASSED [8.9557s] [ 91%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s1_head_dims2_cpu_float32 PASSED [9.6117s] [ 92%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s2_head_dims0_cpu_float32 PASSED [8.7641s] [ 92%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s2_head_dims1_cpu_float32 PASSED [8.9076s] [ 93%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s2_head_dims2_cpu_float32 PASSED [9.0962s] [ 93%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s3_head_dims0_cpu_float32 PASSED [8.7500s] [ 94%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s3_head_dims1_cpu_float32 PASSED [8.8864s] [ 95%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s3_head_dims2_cpu_float32 PASSED [9.1274s] [ 95%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_subgraph_respect_decompostion_float32_cpu_float32 PASSED [1.1441s] [ 96%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_tma_decoding_float32_cpu_float32 SKIPPED [0.0003s] [ 96%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_full_mask_vs_sdpa_cpu PASSED [3.8134s] [ 97%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_full_mask_vs_sdpa_paged_attention_cpu PASSED [4.7360s] [ 97%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_no_mask_vs_sdpa_cpu PASSED [3.9586s] [ 98%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_no_mask_vs_sdpa_paged_attention_cpu PASSED [4.6857s] [ 98%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_partial_block_vs_sdpa_cpu PASSED [3.7479s] [ 99%] | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_partial_block_vs_sdpa_paged_attention_cpu PASSED [4.6698s] [100%] | |
| - generated xml file: /dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/test-reports/python-pytest/inductor.test_flex_decoding/inductor.test_flex_decoding-c5d1e9774784417a.xml - | |
| ========= 175 passed, 6 skipped, 219 deselected in 1252.22s (0:20:52) ========== | |
| The following tests failed consistently: ['test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_fully_masked_out_rows_0_check_gqa_cpu', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod4_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod5_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod6_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod7_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod8_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod4_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod5_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod6_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod7_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod8_cpu_float32'] | |
| FINISHED PRINTING LOG FILE of inductor/test_flex_decoding 1/1 (test/test-reports/inductor.test_flex_decoding_1.1_651a6cbf9932f79f_.log) | |
| Finished inductor/test_flex_decoding 1/1 in 1401 minutes | |
| inductor/test_flex_decoding 1/1 failed! | |
| Running test batch 'tests to run' cost 112190.64 seconds | |
| distributed/tensor/test_dtensor_ops 1/1 failed! | |
| inductor/test_flex_attention 1/1 failed! | |
| functorch/test_ops 1/1 failed! | |
| inductor/test_flex_decoding 1/1 failed! | |
| == 2026-02-11 17:24:37,814 filetools.py:423 INFO Path /tmp/easybuild-tmp/eb-afuu32o6/tmprp6j_fpg successfully removed. | |
| == 2026-02-11 17:24:44,405 pytorch.py:712 WARNING Found 3 individual tests with failed assertions: TestDTensorOpsCPU.test_dtensor_op_db_nn_functional_multi_head_attention_forward_cpu_float32, TestOperatorsCPU.test_vmapvjp_linalg_tensorsolve_cpu_float32, TestOperatorsCPU.test_vmap_autograd_grad_nn_functional_conv2d_cpu_float32 | |
| == 2026-02-11 17:24:44,621 build_log.py:233 ERROR EasyBuild encountered an error: Failing because not all failed tests could be determined.Tests failed to start, crashed or the test accounting in the PyTorch EasyBlock needs updating! | |
| Missing (2): inductor/test_flex_attention, inductor/test_flex_decoding | |
| You can check the test failures (in the log) manually and if they are harmless, use --ignore-test-failure to make the test step pass. | |
| 3 test failures, 0 test errors (out of 262061): | |
| Failed tests (suites/files): | |
| distributed/tensor/test_dtensor_ops (1 failed, 275 passed, 397 skipped, 0 errors) | |
| functorch/test_ops (2 failed, 7495 passed, 2725 skipped, 0 errors) | |
| Could not count failed tests for the following test suites/files: | |
| inductor/test_flex_attention (Undetected or did not run properly) | |
| inductor/test_flex_decoding (Undetected or did not run properly) (at easybuild/easyblocks/p/pytorch.py:800 in test_step) | |
| Callstack: | |
| easybuild/easyblocks/p/pytorch.py:800 in test_step | |
| easybuild/framework/easyblock.py:3069 in _test_step | |
| easybuild/framework/easyblock.py:4782 in run_step | |
| easybuild/framework/easyblock.py:4941 in run_all_steps | |
| easybuild/framework/easyblock.py:5129 in build_and_install_one | |
| easybuild/main.py:178 in build_and_install_software | |
| easybuild/main.py:611 in process_eb_args | |
| easybuild/main.py:795 in main | |
| easybuild/main.py:844 in main_with_hooks | |
| easybuild/main.py:859 in <module> | |
| == 2026-02-11 17:24:44,623 build_log.py:233 ERROR EasyBuild encountered an error: An error was raised during test step: Failing because not all failed tests could be determined.Tests failed to start, crashed or the test accounting in the PyTorch EasyBlock needs updating! | |
| Missing (2): inductor/test_flex_attention, inductor/test_flex_decoding | |
| You can check the test failures (in the log) manually and if they are harmless, use --ignore-test-failure to make the test step pass. | |
| 3 test failures, 0 test errors (out of 262061): | |
| Failed tests (suites/files): | |
| distributed/tensor/test_dtensor_ops (1 failed, 275 passed, 397 skipped, 0 errors) | |
| functorch/test_ops (2 failed, 7495 passed, 2725 skipped, 0 errors) | |
| Could not count failed tests for the following test suites/files: | |
| inductor/test_flex_attention (Undetected or did not run properly) | |
| inductor/test_flex_decoding (Undetected or did not run properly) (at easybuild/framework/easyblock.py:2572 in report_test_failure) | |
| Callstack: | |
| easybuild/framework/easyblock.py:2572 in report_test_failure | |
| easybuild/framework/easyblock.py:3071 in _test_step | |
| easybuild/framework/easyblock.py:4782 in run_step | |
| easybuild/framework/easyblock.py:4941 in run_all_steps | |
| easybuild/framework/easyblock.py:5129 in build_and_install_one | |
| easybuild/main.py:178 in build_and_install_software | |
| easybuild/main.py:611 in process_eb_args | |
| easybuild/main.py:795 in main | |
| easybuild/main.py:844 in main_with_hooks | |
| easybuild/main.py:859 in <module> | |
| == 2026-02-11 17:24:44,623 build_log.py:329 INFO ... (took 31 hours 12 mins 5 secs) | |
| == 2026-02-11 17:24:44,623 filetools.py:2172 INFO Removing lock /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/.locks/_data_horse_ws_s3248973-EasyBuild_easybuild-rome_software_PyTorch_2.9.1-foss-2024a.lock... | |
| == 2026-02-11 17:24:44,630 filetools.py:423 INFO Path /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/.locks/_data_horse_ws_s3248973-EasyBuild_easybuild-rome_software_PyTorch_2.9.1-foss-2024a.lock successfully removed. | |
| == 2026-02-11 17:24:44,630 filetools.py:2176 INFO Lock removed: /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/.locks/_data_horse_ws_s3248973-EasyBuild_easybuild-rome_software_PyTorch_2.9.1-foss-2024a.lock | |
| == 2026-02-11 17:24:44,630 easyblock.py:392 INFO Closing log for application name PyTorch version 2.9.1 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment