Atomics on 16 bits: prevent reading 4 bytes for 2-byte locations. #3005
+37
−7
ROCm Repo Management API / Jenkins
failed
Feb 27, 2026 in 5h 46m 41s
Tests/Test Inductor/Run pytorch_inductor_null: warning in 'junit' step
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Shell Script
Error in sh step, with arguments ./test_pytorch_test.sh.
script returned exit code 1
Build log
Build log truncated.
[2026-02-27T00:16:36.113Z]
[2026-02-27T00:16:36.113Z] Finished test_autograd 1/1 ... [2026-02-27 00:16:34.876167][49131.288475807], took 2.98min
[2026-02-27T00:19:04.714Z]
[2026-02-27T00:19:04.714Z] PRINTING LOG FILE of test_cuda 1/1 (test/test-reports/test_cuda_1.1_411d3fd0adc32727_.log)
[2026-02-27T00:19:04.714Z] Test results will be stored in test-reports/python-pytest/test_cuda/test_cuda-620f30639e2d67b4.xml
[2026-02-27T00:19:04.714Z] ============================= test session starts ==============================
[2026-02-27T00:19:04.714Z] platform linux -- Python 3.12.12, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-02-27T00:19:04.714Z] cachedir: .pytest_cache
[2026-02-27T00:19:04.714Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-02-27T00:19:04.714Z] rootdir: /var/lib/jenkins/pytorch
[2026-02-27T00:19:04.714Z] configfile: pytest.ini
[2026-02-27T00:19:04.714Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-02-27T00:19:04.714Z] collecting ... collected 286 items / 34 deselected / 252 selected
[2026-02-27T00:19:04.714Z] stepcurrent: Cannot find last run test, not skipping
[2026-02-27T00:19:04.714Z] Running 252 items in this shard: test/test_cuda.py::TestCuda::test_arithmetic_large_tensor, test/test_cuda.py::TestCuda::test_batch_norm_gather_stats, test/test_cuda.py::TestCuda::test_bincount_ext, test/test_cuda.py::TestCuda::test_caching_allocator_record_stream_oom, test/test_cuda.py::TestCuda::test_caching_pinned_memory, test/test_cuda.py::TestCuda::test_check_error, test/test_cuda.py::TestCuda::test_copy_non_blocking, test/test_cuda.py::TestCuda::test_copy_non_blocking_type_conversion, test/test_cuda.py::TestCuda::test_cublas_allow_bf16_reduced_precision_reduction_get_set, test/test_cuda.py::TestCuda::test_cublas_allow_fp16_accumulation_get_set, test/test_cuda.py::TestCuda::test_cublas_allow_fp16_reduced_precision_reduction_get_set, test/test_cuda.py::TestCuda::test_cublas_allow_tf32_get_set, test/test_cuda.py::TestCuda::test_cublas_multiple_threads_same_device, test/test_cuda.py::TestCuda::test_cublas_workspace_explicit_allocation, test/test_cuda.py::TestCuda::test_cuda_get_device_capability, test/test_cuda.py::TestCuda::test_cuda_get_device_name, test/test_cuda.py::TestCuda::test_cuda_get_device_properties, test/test_cuda.py::TestCuda::test_cuda_graph_allocator_propagates_stream, test/test_cuda.py::TestCuda::test_cuda_graph_error_options, test/test_cuda.py::TestCuda::test_cuda_graph_inference_mode, test/test_cuda.py::TestCuda::test_cuda_graph_raw_graph, test/test_cuda.py::TestCuda::test_cuda_graph_raw_graph_exec_keep_graph_False, test/test_cuda.py::TestCuda::test_cuda_graph_raw_graph_exec_keep_graph_True, test/test_cuda.py::TestCuda::test_cuda_graph_raw_graph_keep_graph_false, test/test_cuda.py::TestCuda::test_cuda_graph_raw_graph_reset_and_recapture, test/test_cuda.py::TestCuda::test_cuda_graph_tensor_item_not_allowed, test/test_cuda.py::TestCuda::test_cuda_memory_leak_detection_propagates_errors, test/test_cuda.py::TestCuda::test_cuda_stream_protocol, test/test_cuda.py::TestCuda::test_cudart_register, test/test_cuda.py::TestCuda::test_cudnn_allow_tf32_get_set, test/test_cuda.py::TestCuda::test_cudnn_multiple_threads_same_device, test/test_cuda.py::TestCuda::test_cusparse_multiple_threads_same_device, test/test_cuda.py::TestCuda::test_device_context_manager, test/test_cuda.py::TestCuda::test_device_count_not_cached_pre_init, test/test_cuda.py::TestCuda::test_events, test/test_cuda.py::TestCuda::test_events_elapsedtime, test/test_cuda.py::TestCuda::test_fixed_cuda_assert_async, test/test_cuda.py::TestCuda::test_float32_matmul_precision_get_set, test/test_cuda.py::TestCuda::test_fp32_precision_with_tf32, test/test_cuda.py::TestCuda::test_gather_bool, test/test_cuda.py::TestCuda::test_gds_fails_in_ci, test/test_cuda.py::TestCuda::test_generic_stream_event, test/test_cuda.py::TestCuda::test_get_device_index, test/test_cuda.py::TestCuda::test_graph_capture_oom, test/test_cuda.py::TestCuda::test_graph_capture_reset_recapture, test/test_cuda.py::TestCuda::test_graph_capture_simple, test/test_cuda.py::TestCuda::test_graph_concurrent_replay, test/test_cuda.py::TestCuda::test_graph_debugdump, test/test_cuda.py::TestCuda::test_graph_error, test/test_cuda.py::TestCuda::test_graph_is_current_stream_capturing, test/test_cuda.py::TestCuda::test_graph_make_graphed_callables_same_pool, test/test_cuda.py::TestCuda::test_graph_memory_stats_and_use_result_after_destroy_graph, test/test_cuda.py::TestCuda::test_graph_optims_with_explicitly_capturable_param_groups, test/test_cuda.py::TestCuda::test_graph_record_stream, test/test_cuda.py::TestCuda::test_graph_rng_distributions, test/test_cuda.py::TestCuda::test_graph_rng_functional, test/test_cuda.py::TestCuda::test_graph_three_successive, test/test_cuda.py::TestCuda::test_graph_timing, test/test_cuda.py::TestCuda::test_graph_two_successive, test/test_cuda.py::TestCuda::test_graph_warn_if_has_zero_nodes, test/test_cuda.py::TestCuda::test_graphsafe_set_get_rng_state, test/test_cuda.py::TestCuda::test_hip_device_count, test/test_cuda.py::TestCuda::test_index_out_of_bounds_exception_cuda, test/test_cuda.py::TestCuda::test_is_pinned_no_context, test/test_cuda.py::TestCuda::test_lazy_init, test/test_cuda.py::TestCuda::test_manual_seed, test/test_cuda.py::TestCuda::test_matmul_device_mismatch, test/test_cuda.py::TestCuda::test_matmul_memory_use, test/test_cuda.py::TestCuda::test_mean_fp16, test/test_cuda.py::TestCuda::test_memory_allocation, test/test_cuda.py::TestCuda::test_memory_stats, test/test_cuda.py::TestCuda::test_memory_stats_of_multiple_generators_and_graphs, test/test_cuda.py::TestCuda::test_min_max_inits, test/test_cuda.py::TestCuda::test_multi_device_context_manager, test/test_cuda.py::TestCuda::test_multi_device_stream_context_manager, test/test_cuda.py::TestCuda::test_multinomial_ext, test/test_cuda.py::TestCuda::test_multinomial_invalid_probs_cuda, test/test_cuda.py::TestCuda::test_noncontiguous_pinned_memory, test/test_cuda.py::TestCuda::test_norm_type_conversion, test/test_cuda.py::TestCuda::test_nvtx, test/test_cuda.py::TestCuda::test_out_of_memory, test/test_cuda.py::TestCuda::test_pinned_memory_empty_cache, test/test_cuda.py::TestCuda::test_pinned_memory_use_background_threads, test/test_cuda.py::TestCuda::test_pinned_memory_with_cudaregister, test/test_cuda.py::TestCuda::test_pinned_memory_with_cudaregister_multithread, test/test_cuda.py::TestCuda::test_preferred_blas_library_settings, test/test_cuda.py::TestCuda::test_prod_large, test/test_cuda.py::TestCuda::test_randint_randomness_for_large_range, test/test_cuda.py::TestCuda::test_random_no_reused_random_states_float32, test/test_cuda.py::TestCuda::test_random_no_reused_random_states_float64, test/test_cuda.py::TestCuda::test_record_stream, test/test_cuda.py::TestCuda::test_record_stream_on_shifted_view, test/test_cuda.py::TestCuda::test_reduction_gpu_memory_accessing, test/test_cuda.py::TestCuda::test_rocm_backward_pass_guard, test/test_cuda.py::TestCuda::test_serialization_array_with_empty, test/test_cuda.py::TestCuda::test_serialization_array_with_storage, test/test_cuda.py::TestCuda::test_specify_improper_device_name, test/test_cuda.py::TestCuda::test_stream_compatibility, test/test_cuda.py::TestCuda::test_stream_context_manager, test/test_cuda.py::TestCuda::test_stream_event_repr, test/test_cuda.py::TestCuda::test_streaming_backwards_callback, test/test_cuda.py::TestCuda::test_streaming_backwards_multiple_streams, test/test_cuda.py::TestCuda::test_streaming_backwards_sync, test/test_cuda.py::TestCuda::test_streaming_backwards_sync_graph_root, test/test_cuda.py::TestCuda::test_streams, test/test_cuda.py::TestCuda::test_sum_fp16, test/test_cuda.py::TestCuda::test_tiny_half_norm_, test/test_cuda.py::TestCuda::test_to_cpu_blocking_by_default, test/test_cuda.py::TestCuda::test_to_numpy, test/test_cuda.py::TestCuda::test_torch_manual_seed_seeds_cuda_devices, test/test_cuda.py::TestCuda::test_type_conversions, test/test_cuda.py::TestCuda::test_uuid, test/test_cuda.py::TestCudaMallocAsync::test_allocator_backend, test/test_cuda.py::TestCudaMallocAsync::test_allocator_fuzz, test/test_cuda.py::TestCudaMallocAsync::test_allocator_memory_fraction_setting, test/test_cuda.py::TestCudaMallocAsync::test_allocator_settings, test/test_cuda.py::TestCudaMallocAsync::test_cachingAllocator_raw_alloc, test/test_cuda.py::TestCudaMallocAsync::test_clock_speed, test/test_cuda.py::TestCudaMallocAsync::test_cpp_memory_snapshot_pickle, test/test_cuda.py::TestCudaMallocAsync::test_cycles, test/test_cuda.py::TestCudaMallocAsync::test_device_memory_used, test/test_cuda.py::TestCudaMallocAsync::test_direct_traceback, test/test_cuda.py::TestCudaMallocAsync::test_memory_plots, test/test_cuda.py::TestCudaMallocAsync::test_memory_plots_free_segment_stack, test/test_cuda.py::TestCudaMallocAsync::test_memory_plots_free_stack, test/test_cuda.py::TestCudaMallocAsync::test_memory_plots_history_context, test/test_cuda.py::TestCudaMallocAsync::test_memory_plots_metadata, test/test_cuda.py::TestCudaMallocAsync::test_memory_profiler_viz, test/test_cuda.py::TestCudaMallocAsync::test_memory_snapshot, test/test_cuda.py::TestCudaMallocAsync::test_memory_snapshot_script, test/test_cuda.py::TestCudaMallocAsync::test_memory_snapshot_skip_actions, test/test_cuda.py::TestCudaMallocAsync::test_memory_snapshot_with_cpp, test/test_cuda.py::TestCudaMallocAsync::test_notifies_oom, test/test_cuda.py::TestCudaMallocAsync::test_nvml_get_handler, test/test_cuda.py::TestCudaMallocAsync::test_power_draw, test/test_cuda.py::TestCudaMallocAsync::test_raises_oom_max_split_size_mb_setting_False, test/test_cuda.py::TestCudaMallocAsync::test_raises_oom_max_split_size_mb_setting_True, test/test_cuda.py::TestCudaMallocAsync::test_raw_amdsmi_device_count, test/test_cuda.py::TestCudaMallocAsync::test_raw_amdsmi_device_uuids, test/test_cuda.py::TestCudaMallocAsync::test_temperature, test/test_cuda.py::TestCudaMallocAsync::test_uuid_visible_devices, test/test_cuda.py::TestBlockStateAbsorption::test_additional_free_following_checkpoint, test/test_cuda.py::TestBlockStateAbsorption::test_allocate_in_thread_to_pool, test/test_cuda.py::TestBlockStateAbsorption::test_allocated_in_middle_of_segment, test/test_cuda.py::TestBlockStateAbsorption::test_assigning_back_deleter_fns_to_tensor, test/test_cuda.py::TestBlockStateAbsorption::test_check_pool_live_allocations, test/test_cuda.py::TestBlockStateAbsorption::test_expandable_segment_checkpoint_growth, test/test_cuda.py::TestBlockStateAbsorption::test_middle_allocations_contiguous, test/test_cuda.py::TestBlockStateAbsorption::test_multiple_middle_allocations, test/test_cuda.py::TestBlockStateAbsorption::test_no_triton_on_import, test/test_cuda.py::TestBlockStateAbsorption::test_resnet, test/test_cuda.py::TestBlockStateAbsorption::test_simple, test/test_cuda.py::TestBlockStateAbsorption::test_tensor_dies_after_checkpoint, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_pin_memory_no_use2_use_cuda_host_register_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_pin_memory_no_use2_use_cuda_host_register_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_pin_memory_no_use_use_cuda_host_register_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_pin_memory_no_use_use_cuda_host_register_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_pin_memory_use_use_cuda_host_register_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_pin_memory_use_use_cuda_host_register_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_False_use_memory_False_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_False_use_memory_False_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_False_use_memory_True_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_False_use_memory_True_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_True_use_memory_False_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_True_use_memory_False_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_True_use_memory_True_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_False_use_background_threads_True_use_memory_True_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_False_use_memory_False_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_False_use_memory_False_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_False_use_memory_True_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_False_use_memory_True_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_True_use_memory_False_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_True_use_memory_False_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_True_use_memory_True_delete_memory_False, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_two_graphs_use_cuda_host_register_True_use_background_threads_True_use_memory_True_delete_memory_True, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_unpinned_memory_use, test/test_cuda.py::TestCachingHostAllocatorCudaGraph::test_unpinned_memory_use_device_to_host, test/test_cuda.py::TestMemPool::test_graph_capture_reclaim_2_streams, test/test_cuda.py::TestMemPool::test_graph_capture_reclaim_4_streams, test/test_cuda.py::TestMemPool::test_mempool_empty_cache, test/test_cuda.py::TestMemPool::test_mempool_emptycache_multithread, test/test_cuda.py::TestMemPool::test_mempool_expandable, test/test_cuda.py::TestMemPool::test_mempool_id, test/test_cuda.py::TestMemPool::test_mempool_multithread, test/test_cuda.py::TestMemPool::test_snapshot_include_traces, test/test_cuda.py::TestMemPool::test_snapshot_include_traces_correctness, test/test_cuda.py::TestGreenContext::test_greencontext_restores_stream, test/test_cuda.py::TestGDS::test_gds_read_write_tensors, test/test_cuda.py::TestCudaAutocast::test_autocast_banned, test/test_cuda.py::TestCudaAutocast::test_autocast_cat_jit, test/test_cuda.py::TestCudaAutocast::test_autocast_checkpointing, test/test_cuda.py::TestCudaAutocast::test_autocast_custom_cast_inputs, test/test_cuda.py::TestCudaAutocast::test_autocast_custom_deprecated_warning, test/test_cuda.py::TestCudaAutocast::test_autocast_custom_enabled, test/test_cuda.py::TestCudaAutocast::test_autocast_ignored_types, test/test_cuda.py::TestCudaAutocast::test_autocast_linalg_fp16, test/test_cuda.py::TestCudaAutocast::test_autocast_methods_expect_builtin_promote, test/test_cuda.py::TestCudaAutocast::test_autocast_methods_fp16, test/test_cuda.py::TestCudaAutocast::test_autocast_methods_fp32, test/test_cuda.py::TestCudaAutocast::test_autocast_nn_bf16, test/test_cuda.py::TestCudaAutocast::test_autocast_nn_fp16, test/test_cuda.py::TestCudaAutocast::test_autocast_nn_fp32, test/test_cuda.py::TestCudaAutocast::test_autocast_rnn, test/test_cuda.py::TestCudaAutocast::test_autocast_torch_bf16, test/test_cuda.py::TestCudaAutocast::test_autocast_torch_expect_builtin_promote, test/test_cuda.py::TestCudaAutocast::test_autocast_torch_fp16, test/test_cuda.py::TestCudaAutocast::test_autocast_torch_fp32, test/test_cuda.py::TestCudaAutocast::test_autocast_torch_need_autocast_promote, test/test_cuda.py::TestCudaAutocast::test_cuda_autocast_deprecated_warning, test/test_cuda.py::TestCudaAutocast::test_rnn_packed_sequence_batch_sizes_must_be_cpu, test/test_cuda.py::TestCompileKernel::test_compile_kernel, test/test_cuda.py::TestCompileKernel::test_compile_kernel_advanced, test/test_cuda.py::TestCompileKernel::test_compile_kernel_as_custom_op, test/test_cuda.py::TestCompileKernel::test_compile_kernel_cuda_headers, test/test_cuda.py::TestCompileKernel::test_compile_kernel_custom_op_validation, test/test_cuda.py::TestCompileKernel::test_compile_kernel_dlpack, test/test_cuda.py::TestCompileKernel::test_compile_kernel_double_precision, test/test_cuda.py::TestCompileKernel::test_compile_kernel_large_shared_memory, test/test_cuda.py::TestCompileKernel::test_compile_kernel_template, test/test_cuda.py::TestFXMemoryProfiler::test_fx_memory_profiler_augmentation, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_False_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_False_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_False_SGD_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_True_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_True_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaler_with_preset_grad_scale_in_place_unscale_True_SGD_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaling_autocast_fused_optimizers_Adagrad_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaling_autocast_fused_optimizers_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaling_autocast_fused_optimizers_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_grad_scaling_autocast_fused_optimizers_SGD_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_False_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_False_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_False_SGD_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_True_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_True_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_False_fused_True_SGD_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_True_fused_False_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_True_fused_False_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_grad_scaling_foreach_True_fused_False_SGD_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_ASGD_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_Adadelta_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_Adamax_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_NAdam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_RAdam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_RMSprop_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_optims_Rprop_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_scaling_fused_optimizers_AdamW_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_scaling_fused_optimizers_Adam_cuda_float32, test/test_cuda.py::TestCudaOptimsCUDA::test_graph_scaling_fused_optimizers_SGD_cuda_float32, test/test_cuda.py::TestCudaDeviceParametrizedCUDA::test_graph_external_wait_and_record_cuda
[2026-02-27T00:19:04.714Z]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_arithmetic_large_tensor SKIPPED [0.0004s] (was disabled due to not enough memory, but actually it always fail) [ 0%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_batch_norm_gather_stats PASSED [0.1901s] [ 0%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_bincount_ext PASSED [0.4929s] [ 1%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_caching_allocator_record_stream_oom PASSED [0.0598s] [ 1%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_caching_pinned_memory PASSED [1.0071s] [ 1%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_check_error PASSED [0.0012s] [ 2%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_copy_non_blocking PASSED [0.2004s] [ 2%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_copy_non_blocking_type_conversion PASSED [0.1058s] [ 3%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cublas_allow_bf16_reduced_precision_reduction_get_set PASSED [0.0017s] [ 3%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cublas_allow_fp16_accumulation_get_set PASSED [0.0011s] [ 3%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cublas_allow_fp16_reduced_precision_reduction_get_set PASSED [0.0012s] [ 4%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cublas_allow_tf32_get_set PASSED [0.0010s] [ 4%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cublas_multiple_threads_same_device PASSED [0.4856s] [ 5%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cublas_workspace_explicit_allocation [W227 00:13:42.157635398 Context.cpp:541] Warning: torch.backends.cuda.preferred_blas_library is an experimental feature. If you see any error or unexpected behavior when this flag is set please file an issue on GitHub. (function operator())
[2026-02-27T00:19:04.714Z] PASSED [0.0803s] [ 5%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cuda_get_device_capability PASSED [0.0015s] [ 5%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cuda_get_device_name PASSED [0.0010s] [ 6%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cuda_get_device_properties PASSED [0.0011s] [ 6%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cuda_graph_allocator_propagates_stream PASSED [0.0212s] [ 7%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cuda_graph_error_options PASSED [0.0350s] [ 7%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cuda_graph_inference_mode PASSED [0.0091s] [ 7%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cuda_graph_raw_graph SKIPPED [0.0002s] (CUDA >= 11.0 or ROCM >= 5.3 required for graphs, cuda-bindings must be installed) [ 8%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cuda_graph_raw_graph_exec_keep_graph_False SKIPPED [0.0002s] (CUDA >= 11.0 or ROCM >= 5.3 required for graphs, cuda-bindings must be installed) [ 8%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cuda_graph_raw_graph_exec_keep_graph_True SKIPPED [0.0005s] (CUDA >= 11.0 or ROCM >= 5.3 required for graphs, cuda-bindings must be installed) [ 9%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cuda_graph_raw_graph_keep_graph_false PASSED [0.0020s] [ 9%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cuda_graph_raw_graph_reset_and_recapture PASSED [0.0030s] [ 9%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cuda_graph_tensor_item_not_allowed /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/cuda/graphs.py:126: UserWarning: The CUDA Graph is empty. This usually means that the graph was attempted to be captured on wrong device or stream. (Triggered internally at /var/lib/jenkins/pytorch/aten/src/ATen/hip/HIPGraph.cpp:148.)
[2026-02-27T00:19:04.714Z] super().capture_end()
[2026-02-27T00:19:04.714Z] Traceback (most recent call last):
[2026-02-27T00:19:04.714Z] File "<string>", line 17, in <module>
[2026-02-27T00:19:04.714Z] File "<string>", line 7, in my_func
[2026-02-27T00:19:04.714Z] torch.AcceleratorError: CUDA error: operation not permitted when stream is capturing
[2026-02-27T00:19:04.714Z] Search for `hipErrorStreamCaptureUnsupported' in https://rocm.docs.amd.com/projects/HIP/en/latest/index.html for more information.
[2026-02-27T00:19:04.714Z] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[2026-02-27T00:19:04.714Z] For debugging consider passing AMD_SERIALIZE_KERNEL=3
[2026-02-27T00:19:04.714Z] Device-side assertion tracking was not enabled by user.
[2026-02-27T00:19:04.714Z] PASSED [1.7428s] [ 10%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cuda_memory_leak_detection_propagates_errors PASSED [0.0019s] [ 10%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cuda_stream_protocol PASSED [0.0072s] [ 11%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cudart_register PASSED [0.0013s] [ 11%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cudnn_allow_tf32_get_set PASSED [0.0010s] [ 11%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cudnn_multiple_threads_same_device PASSED [217.3224s] [ 12%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_cusparse_multiple_threads_same_device PASSED [9.9195s] [ 12%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_device_context_manager PASSED [0.0025s] [ 13%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_device_count_not_cached_pre_init SKIPPED [0.0004s] (requires multiple devices) [ 13%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_events PASSED [0.0519s] [ 13%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_events_elapsedtime PASSED [0.0011s] [ 14%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_fixed_cuda_assert_async GPU coredump: execvp failed: No such file or directory
[2026-02-27T00:19:04.714Z] Failed to write segment data to pipe: Broken pipe
[2026-02-27T00:19:04.714Z] GPU coredump: handler exited with error (status: 1)
[2026-02-27T00:19:04.714Z] GPU core dump failed
[2026-02-27T00:19:04.714Z] :0:rocdevice.cpp :3586: 49189728093 us: Callback: Queue 0x7f314d200000 aborting with error : HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception. code: 0x1016
[2026-02-27T00:19:04.714Z] GPU coredump: execvp failed: No such file or directory
[2026-02-27T00:19:04.714Z] Failed to write segment data to pipe: Broken pipe
[2026-02-27T00:19:04.714Z] GPU coredump: handler exited with error (status: 1)
[2026-02-27T00:19:04.714Z] GPU core dump failed
[2026-02-27T00:19:04.714Z] :0:rocdevice.cpp :3586: 49199433410 us: Callback: Queue 0x7f688ac00000 aborting with error : HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception. code: 0x1016
[2026-02-27T00:19:04.714Z] GPU coredump: execvp failed: No such file or directory
[2026-02-27T00:19:04.714Z] Failed to write segment data to pipe: Broken pipe
[2026-02-27T00:19:04.714Z] GPU coredump: handler exited with error (status: 1)
[2026-02-27T00:19:04.714Z] GPU core dump failed
[2026-02-27T00:19:04.714Z] :0:rocdevice.cpp :3586: 49200960360 us: Callback: Queue 0x7f3af8200000 aborting with error : HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception. code: 0x1016
[2026-02-27T00:19:04.714Z] GPU coredump: execvp failed: No such file or directory
[2026-02-27T00:19:04.714Z] Failed to write segment data to pipe: Broken pipe
[2026-02-27T00:19:04.714Z] GPU coredump: handler exited with error (status: 1)
[2026-02-27T00:19:04.714Z] GPU core dump failed
[2026-02-27T00:19:04.714Z] :0:rocdevice.cpp :3586: 49210736141 us: Callback: Queue 0x7fa481c00000 aborting with error : HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception. code: 0x1016
[2026-02-27T00:19:04.714Z] PASSED [22.5435s] [ 14%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_float32_matmul_precision_get_set PASSED [0.0025s] [ 15%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_fp32_precision_with_tf32 PASSED [0.0024s] [ 15%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_gather_bool PASSED [0.0306s] [ 15%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_gds_fails_in_ci PASSED [0.0014s] [ 16%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_generic_stream_event PASSED [0.0016s] [ 16%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_get_device_index PASSED [0.0009s] [ 17%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_graph_capture_oom PASSED [0.0027s] [ 17%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_graph_capture_reset_recapture PASSED [0.0086s] [ 17%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_graph_capture_simple PASSED [0.0016s] [ 18%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_graph_concurrent_replay SKIPPED [0.0003s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/104055 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 18%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_graph_debugdump PASSED [0.0127s] [ 19%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_graph_error PASSED [1.5535s] [ 19%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_graph_is_current_stream_capturing PASSED [0.0009s] [ 19%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_graph_make_graphed_callables_same_pool PASSED [0.4840s] [ 20%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_graph_memory_stats_and_use_result_after_destroy_graph PASSED [1.5652s] [ 20%]
[2026-02-27T00:19:04.714Z] test_cuda.py::TestCuda::test_graph_optims_with_explicitly_capturable_param_groups PASSED [0.2620s] [ 21%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_graph_record_stream PASSED [0.0314s] [ 21%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_graph_rng_distributions PASSED [0.1158s] [ 21%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_graph_rng_functional PASSED [0.0237s] [ 22%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_graph_three_successive PASSED [0.0057s] [ 22%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_graph_timing SKIPPED [0.0002s] (CUDA >= 11.0 required for external events in cuda graphs. rocm does not support external events) [ 23%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_graph_two_successive PASSED [0.0059s] [ 23%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_graph_warn_if_has_zero_nodes PASSED [0.0011s] [ 23%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_graphsafe_set_get_rng_state PASSED [0.0033s] [ 24%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_hip_device_count PASSED [5.7332s] [ 24%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_index_out_of_bounds_exception_cuda SKIPPED [0.0012s] (test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test) [ 25%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_is_pinned_no_context PASSED [1.6259s] [ 25%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_lazy_init PASSED [1.4075s] [ 25%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_manual_seed PASSED [0.0035s] [ 26%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_matmul_device_mismatch PASSED [0.0032s] [ 26%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_matmul_memory_use PASSED [0.0061s] [ 26%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_mean_fp16 PASSED [0.0145s] [ 27%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_memory_allocation PASSED [0.1671s] [ 27%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_memory_stats PASSED [0.2903s] [ 28%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_memory_stats_of_multiple_generators_and_graphs PASSED [0.4445s] [ 28%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_min_max_inits PASSED [0.0020s] [ 28%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_multi_device_context_manager SKIPPED [0.0002s] (only one GPU detected) [ 29%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_multi_device_stream_context_manager SKIPPED [0.0001s] (only one GPU detected) [ 29%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_multinomial_ext PASSED [0.1506s] [ 30%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_multinomial_invalid_probs_cuda SKIPPED [0.0012s] (test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test) [ 30%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_noncontiguous_pinned_memory PASSED [0.0014s] [ 30%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_norm_type_conversion PASSED [0.0086s] [ 31%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_nvtx PASSED [0.0010s] [ 31%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_out_of_memory PASSED [0.0036s] [ 32%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_pinned_memory_empty_cache PASSED [0.0214s] [ 32%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_pinned_memory_use_background_threads PASSED [1.7571s] [ 32%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_pinned_memory_with_cudaregister PASSED [0.0592s] [ 33%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_pinned_memory_with_cudaregister_multithread PASSED [0.1062s] [ 33%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_preferred_blas_library_settings PASSED [3.2698s] [ 34%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_prod_large PASSED [0.0096s] [ 34%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_randint_randomness_for_large_range PASSED [0.6342s] [ 34%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_random_no_reused_random_states_float32 PASSED [0.7156s] [ 35%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_random_no_reused_random_states_float64 PASSED [0.6986s] [ 35%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_record_stream PASSED [0.0515s] [ 36%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_record_stream_on_shifted_view SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/120318 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 36%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_reduction_gpu_memory_accessing PASSED [0.0012s] [ 36%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_rocm_backward_pass_guard PASSED [0.0017s] [ 37%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_serialization_array_with_empty PASSED [0.0042s] [ 37%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_serialization_array_with_storage PASSED [0.0034s] [ 38%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_specify_improper_device_name PASSED [0.0018s] [ 38%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_stream_compatibility PASSED [0.0013s] [ 38%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_stream_context_manager PASSED [0.0013s] [ 39%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_stream_event_repr PASSED [0.0010s] [ 39%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_streaming_backwards_callback PASSED [0.0059s] [ 40%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_streaming_backwards_multiple_streams PASSED [0.0654s] [ 40%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_streaming_backwards_sync PASSED [0.0094s] [ 40%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_streaming_backwards_sync_graph_root PASSED [0.2519s] [ 41%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_streams PASSED [0.0011s] [ 41%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_sum_fp16 PASSED [0.0872s] [ 42%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_tiny_half_norm_ PASSED [0.0016s] [ 42%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_to_cpu_blocking_by_default PASSED [0.3283s] [ 42%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_to_numpy PASSED [0.0017s] [ 43%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_torch_manual_seed_seeds_cuda_devices PASSED [0.0028s] [ 43%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_type_conversions PASSED [0.0020s] [ 44%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCuda::test_uuid PASSED [0.0010s] [ 44%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_allocator_backend PASSED [1.6858s] [ 44%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_allocator_fuzz PASSED [0.5595s] [ 45%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_allocator_memory_fraction_setting PASSED [7.8046s] [ 45%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_allocator_settings PASSED [0.0061s] [ 46%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_cachingAllocator_raw_alloc PASSED [0.0027s] [ 46%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_clock_speed PASSED [0.0027s] [ 46%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_cpp_memory_snapshot_pickle PASSED [13.2798s] [ 47%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_cycles W0227 00:18:40.292000 916800 site-packages/torch/utils/viz/_cycles.py:59] CUDA Memory changed during GC, 512 bytes freed.
[2026-02-27T00:19:04.715Z] PASSED [0.3269s] [ 47%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used ('RERUN', {'yellow': True}) [0.1453s] [ 48%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used ('RERUN', {'yellow': True}) [0.1422s] [ 48%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used FAILED [0.1420s] [ 48%]
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] ==================================== RERUNS ====================================
[2026-02-27T00:19:04.715Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-02-27T00:19:04.715Z] Traceback (most recent call last):
[2026-02-27T00:19:04.715Z] File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 4960, in test_device_memory_used
[2026-02-27T00:19:04.715Z] self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-02-27T00:19:04.715Z] File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-02-27T00:19:04.715Z] raise self.failureException(msg)
[2026-02-27T00:19:04.715Z] AssertionError: False is not true
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] To execute this test, run the following from the base repo dir:
[2026-02-27T00:19:04.715Z] PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-02-27T00:19:04.715Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-02-27T00:19:04.715Z] Traceback (most recent call last):
[2026-02-27T00:19:04.715Z] File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 4960, in test_device_memory_used
[2026-02-27T00:19:04.715Z] self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-02-27T00:19:04.715Z] File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-02-27T00:19:04.715Z] raise self.failureException(msg)
[2026-02-27T00:19:04.715Z] AssertionError: False is not true
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] To execute this test, run the following from the base repo dir:
[2026-02-27T00:19:04.715Z] PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-02-27T00:19:04.715Z] =================================== FAILURES ===================================
[2026-02-27T00:19:04.715Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-02-27T00:19:04.715Z] Traceback (most recent call last):
[2026-02-27T00:19:04.715Z] File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 4960, in test_device_memory_used
[2026-02-27T00:19:04.715Z] self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-02-27T00:19:04.715Z] File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-02-27T00:19:04.715Z] raise self.failureException(msg)
[2026-02-27T00:19:04.715Z] AssertionError: False is not true
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] To execute this test, run the following from the base repo dir:
[2026-02-27T00:19:04.715Z] PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-02-27T00:19:04.715Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_cuda/test_cuda-620f30639e2d67b4.xml -
[2026-02-27T00:19:04.715Z] =========================== short test summary info ============================
[2026-02-27T00:19:04.715Z] FAILED [0.1420s] test_cuda.py::TestCudaMallocAsync::test_device_memory_used - AssertionError: False is not true
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] To execute this test, run the following from the base repo dir:
[2026-02-27T00:19:04.715Z] PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-02-27T00:19:04.715Z] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
[2026-02-27T00:19:04.715Z] = 1 failed, 108 passed, 12 skipped, 34 deselected, 2 rerun in 300.65s (0:05:00) =
[2026-02-27T00:19:04.715Z] Got exit code 1
[2026-02-27T00:19:04.715Z] Retrying single test...
[2026-02-27T00:19:04.715Z] Test results will be stored in test-reports/python-pytest/test_cuda/test_cuda-7057e520d8db2913.xml
[2026-02-27T00:19:04.715Z] ============================= test session starts ==============================
[2026-02-27T00:19:04.715Z] platform linux -- Python 3.12.12, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-02-27T00:19:04.715Z] cachedir: .pytest_cache
[2026-02-27T00:19:04.715Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-02-27T00:19:04.715Z] rootdir: /var/lib/jenkins/pytorch
[2026-02-27T00:19:04.715Z] configfile: pytest.ini
[2026-02-27T00:19:04.715Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-02-27T00:19:04.715Z] collecting ... collected 286 items / 285 deselected / 1 selected
[2026-02-27T00:19:04.715Z] stepcurrent: skipping 142 already run items. Running only test/test_cuda.py::TestCudaMallocAsync::test_device_memory_used
[2026-02-27T00:19:04.715Z] Running 1 items in this shard
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used ('RERUN', {'yellow': True}) [0.1364s] [100%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used ('RERUN', {'yellow': True}) [0.1204s] [100%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used FAILED [0.1184s] [100%]
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] ==================================== RERUNS ====================================
[2026-02-27T00:19:04.715Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-02-27T00:19:04.715Z] Traceback (most recent call last):
[2026-02-27T00:19:04.715Z] File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 4960, in test_device_memory_used
[2026-02-27T00:19:04.715Z] self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-02-27T00:19:04.715Z] File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-02-27T00:19:04.715Z] raise self.failureException(msg)
[2026-02-27T00:19:04.715Z] AssertionError: False is not true
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] To execute this test, run the following from the base repo dir:
[2026-02-27T00:19:04.715Z] PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-02-27T00:19:04.715Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-02-27T00:19:04.715Z] Traceback (most recent call last):
[2026-02-27T00:19:04.715Z] File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 4960, in test_device_memory_used
[2026-02-27T00:19:04.715Z] self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-02-27T00:19:04.715Z] File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-02-27T00:19:04.715Z] raise self.failureException(msg)
[2026-02-27T00:19:04.715Z] AssertionError: False is not true
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] To execute this test, run the following from the base repo dir:
[2026-02-27T00:19:04.715Z] PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-02-27T00:19:04.715Z] =================================== FAILURES ===================================
[2026-02-27T00:19:04.715Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-02-27T00:19:04.715Z] Traceback (most recent call last):
[2026-02-27T00:19:04.715Z] File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 4960, in test_device_memory_used
[2026-02-27T00:19:04.715Z] self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-02-27T00:19:04.715Z] File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-02-27T00:19:04.715Z] raise self.failureException(msg)
[2026-02-27T00:19:04.715Z] AssertionError: False is not true
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] To execute this test, run the following from the base repo dir:
[2026-02-27T00:19:04.715Z] PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-02-27T00:19:04.715Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_cuda/test_cuda-7057e520d8db2913.xml -
[2026-02-27T00:19:04.715Z] =========================== short test summary info ============================
[2026-02-27T00:19:04.715Z] FAILED [0.1184s] test_cuda.py::TestCudaMallocAsync::test_device_memory_used - AssertionError: False is not true
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] To execute this test, run the following from the base repo dir:
[2026-02-27T00:19:04.715Z] PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-02-27T00:19:04.715Z] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
[2026-02-27T00:19:04.715Z] ================== 1 failed, 285 deselected, 2 rerun in 0.41s ==================
[2026-02-27T00:19:04.715Z] Got exit code 1
[2026-02-27T00:19:04.715Z] Retrying single test...
[2026-02-27T00:19:04.715Z] Test results will be stored in test-reports/python-pytest/test_cuda/test_cuda-421da65ce09309e6.xml
[2026-02-27T00:19:04.715Z] ============================= test session starts ==============================
[2026-02-27T00:19:04.715Z] platform linux -- Python 3.12.12, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
[2026-02-27T00:19:04.715Z] cachedir: .pytest_cache
[2026-02-27T00:19:04.715Z] hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
[2026-02-27T00:19:04.715Z] rootdir: /var/lib/jenkins/pytorch
[2026-02-27T00:19:04.715Z] configfile: pytest.ini
[2026-02-27T00:19:04.715Z] plugins: subtests-0.13.1, hypothesis-6.56.4, xdist-3.3.1, flakefinder-1.1.0, xdoctest-1.3.0, rerunfailures-14.0, cpp-2.3.0, typeguard-4.3.0
[2026-02-27T00:19:04.715Z] collecting ... collected 286 items / 285 deselected / 1 selected
[2026-02-27T00:19:04.715Z] stepcurrent: skipping 142 already run items. Running only test/test_cuda.py::TestCudaMallocAsync::test_device_memory_used
[2026-02-27T00:19:04.715Z] Running 1 items in this shard
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used ('RERUN', {'yellow': True}) [0.1336s] [100%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used ('RERUN', {'yellow': True}) [0.1182s] [100%]
[2026-02-27T00:19:04.715Z] test_cuda.py::TestCudaMallocAsync::test_device_memory_used FAILED [0.1153s] [100%]
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] ==================================== RERUNS ====================================
[2026-02-27T00:19:04.715Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-02-27T00:19:04.715Z] Traceback (most recent call last):
[2026-02-27T00:19:04.715Z] File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 4960, in test_device_memory_used
[2026-02-27T00:19:04.715Z] self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-02-27T00:19:04.715Z] File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-02-27T00:19:04.715Z] raise self.failureException(msg)
[2026-02-27T00:19:04.715Z] AssertionError: False is not true
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] To execute this test, run the following from the base repo dir:
[2026-02-27T00:19:04.715Z] PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-02-27T00:19:04.715Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-02-27T00:19:04.715Z] Traceback (most recent call last):
[2026-02-27T00:19:04.715Z] File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 4960, in test_device_memory_used
[2026-02-27T00:19:04.715Z] self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-02-27T00:19:04.715Z] File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-02-27T00:19:04.715Z] raise self.failureException(msg)
[2026-02-27T00:19:04.715Z] AssertionError: False is not true
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] To execute this test, run the following from the base repo dir:
[2026-02-27T00:19:04.715Z] PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-02-27T00:19:04.715Z] =================================== FAILURES ===================================
[2026-02-27T00:19:04.715Z] _________________ TestCudaMallocAsync.test_device_memory_used __________________
[2026-02-27T00:19:04.715Z] Traceback (most recent call last):
[2026-02-27T00:19:04.715Z] File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 4960, in test_device_memory_used
[2026-02-27T00:19:04.715Z] self.assertTrue(num_bytes // 32 <= mem_bytes <= num_bytes * 32)
[2026-02-27T00:19:04.715Z] File "/opt/conda/envs/py_3.12/lib/python3.12/unittest/case.py", line 727, in assertTrue
[2026-02-27T00:19:04.715Z] raise self.failureException(msg)
[2026-02-27T00:19:04.715Z] AssertionError: False is not true
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] To execute this test, run the following from the base repo dir:
[2026-02-27T00:19:04.715Z] PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-02-27T00:19:04.715Z] - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_cuda/test_cuda-421da65ce09309e6.xml -
[2026-02-27T00:19:04.715Z] =========================== short test summary info ============================
[2026-02-27T00:19:04.715Z] FAILED [0.1153s] test_cuda.py::TestCudaMallocAsync::test_device_memory_used - AssertionError: False is not true
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] To execute this test, run the following from the base repo dir:
[2026-02-27T00:19:04.715Z] PYTORCH_TEST_WITH_ROCM=1 python test/test_cuda.py TestCudaMallocAsync.test_device_memory_used
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
[2026-02-27T00:19:04.715Z] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
[2026-02-27T00:19:04.715Z] ================== 1 failed, 285 deselected, 2 rerun in 0.40s ==================
[2026-02-27T00:19:04.715Z] Got exit code 1
[2026-02-27T00:19:04.715Z] FAILED CONSISTENTLY: test/test_cuda.py::TestCudaMallocAsync::test_device_memory_used
[2026-02-27T00:19:04.715Z] Stopping at first consistent failure
[2026-02-27T00:19:04.715Z] The following tests failed consistently: ['test/test_cuda.py::TestCudaMallocAsync::test_device_memory_used']
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] FINISHED PRINTING LOG FILE of test_cuda 1/1 (test/test-reports/test_cuda_1.1_411d3fd0adc32727_.log)
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] Finished test_cuda 1/1 ... [2026-02-27 00:18:51.298996][49267.711309258], took 5.25min
[2026-02-27T00:19:04.715Z] test_cuda 1/1 failed!
[2026-02-27T00:19:04.715Z] Running test batch 'tests to run' cost 415.41 seconds
[2026-02-27T00:19:04.715Z] Emitting td_test_failure_stats_v2
[2026-02-27T00:19:04.715Z] /var/lib/jenkins/pytorch/tools/stats/upload_metrics.py:140: UserWarning: Not emitting metrics for td_test_failure_stats_v2. Missing repo. Please set the GITHUB_REPOSITORY environment variable to pass in this value.
[2026-02-27T00:19:04.715Z] warn(f"Not emitting metrics for {metric_name}. {e}")
[2026-02-27T00:19:04.715Z] test_cuda 1/1 failed!
[2026-02-27T00:19:04.715Z] Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_unary_ufuncs/test_unary_ufuncs-e8d9282057aa3d9f.xml
[2026-02-27T00:19:04.715Z] Found job id: None
[2026-02-27T00:19:04.715Z] Failed to parse and upload json test reports: Unable to locate credentials
[2026-02-27T00:19:04.715Z] Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/test_unary_ufuncs/test_unary_ufuncs-e8d9282057aa3d9f.xml
[2026-02-27T00:19:04.715Z] Found job id: None
[2026-02-27T00:19:04.715Z] Failed to parse and upload json test reports: Unable to locate credentials
[2026-02-27T00:19:04.715Z] GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
[2026-02-27T00:19:04.715Z] Uploading artifacts took 0.00 seconds
[2026-02-27T00:19:04.715Z]
[2026-02-27T00:19:04.715Z] real 6m59.367s
[2026-02-27T00:19:04.715Z] user 90m43.683s
[2026-02-27T00:19:04.715Z] sys 6m4.575s
[2026-02-27T00:19:04.715Z] + sccache_epilogue
[2026-02-27T00:19:04.715Z] + echo '::group::Sccache Compilation Log'
[2026-02-27T00:19:04.715Z] + echo '=================== sccache compilation log ==================='
[2026-02-27T00:19:04.715Z] ::group::Sccache Compilation Log
[2026-02-27T00:19:04.715Z] =================== sccache compilation log ===================
[2026-02-27T00:19:04.715Z] + python /var/lib/jenkins/pytorch/.ci/pytorch/print_sccache_log.py /root/sccache_error.log
[2026-02-27T00:19:04.715Z] + echo '=========== If your build fails, please take a look at the log above for possible reasons ==========='
[2026-02-27T00:19:04.715Z] + sccache --show-stats
[2026-02-27T00:19:04.715Z] =========== If your build fails, please take a look at the log above for possible reasons ===========
[2026-02-27T00:19:04.715Z] Compile requests 313
[2026-02-27T00:19:04.715Z] Compile requests executed 100
[2026-02-27T00:19:04.715Z] Cache hits 32
[2026-02-27T00:19:04.715Z] Cache misses 68
[2026-02-27T00:19:04.715Z] Cache timeouts 0
[2026-02-27T00:19:04.715Z] Cache read errors 0
[2026-02-27T00:19:04.715Z] Forced recaches 0
[2026-02-27T00:19:04.715Z] Cache write errors 0
[2026-02-27T00:19:04.715Z] Compilation failures 0
[2026-02-27T00:19:04.715Z] Cache errors 0
[2026-02-27T00:19:04.715Z] Non-cacheable compilations 0
[2026-02-27T00:19:04.715Z] Non-cacheable calls 32
[2026-02-27T00:19:04.715Z] Non-compilation calls 181
[2026-02-27T00:19:04.715Z] Unsupported compiler calls 0
[2026-02-27T00:19:04.715Z] Average cache write 0.002 s
[2026-02-27T00:19:04.715Z] Average cache read miss 9.998 s
[2026-02-27T00:19:04.715Z] Average cache read hit 0.001 s
[2026-02-27T00:19:04.715Z] Cache location Local disk: "/root/.cache/sccache"
[2026-02-27T00:19:04.715Z] Cache size 5 MiB
[2026-02-27T00:19:04.715Z] Max cache size 10 GiB
[2026-02-27T00:19:04.715Z] + sccache --stop-server
[2026-02-27T00:19:04.715Z] Stopping sccache server...
[2026-02-27T00:19:04.715Z] Compile requests 313
[2026-02-27T00:19:04.715Z] Compile requests executed 100
[2026-02-27T00:19:04.715Z] Cache hits 32
[2026-02-27T00:19:04.715Z] Cache misses 68
[2026-02-27T00:19:04.715Z] Cache timeouts 0
[2026-02-27T00:19:04.715Z] Cache read errors 0
[2026-02-27T00:19:04.715Z] Forced recaches 0
[2026-02-27T00:19:04.715Z] Cache write errors 0
[2026-02-27T00:19:04.715Z] Compilation failures 0
[2026-02-27T00:19:04.715Z] Cache errors 0
[2026-02-27T00:19:04.715Z] Non-cacheable compilations 0
[2026-02-27T00:19:04.715Z] Non-cacheable calls 32
[2026-02-27T00:19:04.715Z] Non-compilation calls 181
[2026-02-27T00:19:04.715Z] Unsupported compiler calls 0
[2026-02-27T00:19:04.715Z] Average cache write 0.002 s
[2026-02-27T00:19:04.715Z] Average cache read miss 9.998 s
[2026-02-27T00:19:04.715Z] Average cache read hit 0.001 s
[2026-02-27T00:19:04.715Z] Cache location Local disk: "/root/.cache/sccache"
[2026-02-27T00:19:04.715Z] Cache size 5 MiB
[2026-02-27T00:19:04.715Z] Max cache size 10 GiB
[2026-02-27T00:19:04.715Z] ::endgroup::
[2026-02-27T00:19:04.715Z] + echo ::endgroup::
[2026-02-27T00:19:04.715Z] + cp -RT test/test-reports /host_workspace/pytorch_reports
[2026-02-27T00:19:04.715Z] + chmod -R 777 /host_workspace/pytorch_log /host_workspace/pytorch_reports
[2026-02-27T00:19:04.715Z] + git clean -fdx
[2026-02-27T00:19:04.715Z] Removing .additional_ci_files/
[2026-02-27T00:19:04.715Z] Removing .pytest_cache/
[2026-02-27T00:19:04.715Z] Removing build/
[2026-02-27T00:19:04.715Z] Removing dist/
[2026-02-27T00:19:04.715Z] Removing test/.pytorch-disabled-tests.json
[2026-02-27T00:19:04.715Z] Removing test/__pycache__/
[2026-02-27T00:19:04.715Z] Removing test/autograd/__pycache__/
[2026-02-27T00:19:04.715Z] Removing test/test-reports/
[2026-02-27T00:19:04.715Z] Removing test/torch_compile_debug/
[2026-02-27T00:19:04.715Z] Removing test_artifacts.zip
[2026-02-27T00:19:04.715Z] Removing tools/__pycache__/
[2026-02-27T00:19:04.715Z] Removing tools/stats/__pycache__/
[2026-02-27T00:19:04.715Z] Removing tools/testing/__pycache__/
[2026-02-27T00:19:04.715Z] Removing tools/testing/target_determination/__pycache__/
[2026-02-27T00:19:04.715Z] Removing tools/testing/target_determination/heuristics/__pycache__/
[2026-02-27T00:19:04.715Z] Removing torch-2.11.0a0+git8e34cc5-cp312-cp312-linux_x86_64.whl
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Error signal
Error in error step, with arguments pytorch_test_2 failed.
pytorch_test_2 failed
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Archive JUnit-formatted test results
Warning in junit step.
4 tests failed
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Shell Script
Error in sh step, with arguments ./test_pytorch_test_distributed.sh.
script returned exit code 1
Build log
Build log truncated.
[2026-02-27T01:41:20.305Z]
[2026-02-27T01:41:20.305Z] distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad <- ../../../../opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py I0227 01:38:21.208000 1116265 site-packages/torch/testing/_internal/common_distributed.py:854] Started process 0 with pid 1116336
[2026-02-27T01:41:20.305Z] I0227 01:38:21.209000 1116265 site-packages/torch/testing/_internal/common_distributed.py:854] Started process 1 with pid 1116337
[2026-02-27T01:41:20.305Z] I0227 01:38:21.210000 1116265 site-packages/torch/testing/_internal/common_distributed.py:854] Started process 2 with pid 1116338
[2026-02-27T01:41:20.305Z] I0227 01:38:21.210000 1116265 site-packages/torch/testing/_internal/common_distributed.py:854] Started process 3 with pid 1116339
[2026-02-27T01:41:20.305Z] /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/parallel/distributed.py:949: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
[2026-02-27T01:41:20.305Z] self._set_static_graph()
[2026-02-27T01:41:20.305Z] /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/parallel/distributed.py:949: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
[2026-02-27T01:41:20.305Z] self._set_static_graph()
[2026-02-27T01:41:20.305Z] /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/parallel/distributed.py:949: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
[2026-02-27T01:41:20.305Z] self._set_static_graph()
Output truncated.
Details
- Kill older PR Builds (1.6 sec)
- Initialize (1 hr 26 min)
- Download CI scripts (2 min 59 sec)
- Checkout Pytorch (7 min 2 sec)
- Check base Docker image existence (13 sec)
- Pull Docker Image (7 min 31 sec)
- Build PyTorch (1 hr 5 min)
- Tests (4 hr 19 min)
- Test PyTorch (7 ms)
- Test Distributed (6 ms)
- Test Inductor (7 ms)
- Test Inductor (4 hr 19 min)
- Run pytorch_inductor_null (4 hr 19 min)
Error: Found 1 failure(s) in pytorch_reports/python-pytest/inductor.test_aot_inductor/inductor.test_aot_inductor-9cf5b991a99195ee.xml report - logs
Error: Some tests are failed or errored - logs
Error: pytorch_inductor_null failed - logs
Unstable: 1 tests failed - logs
- Run pytorch_inductor_null (4 hr 19 min)
- Test Inductor (4 hr 19 min)
- Test PyTorch Slow (7 ms)
- Test PyTorch Slow (7 sec)
- Microbenchmark (14 sec)
- Microbenchmark (7.1 sec)
- Post Build (1.4 sec)
- Declarative: Post Actions (3.4 sec)
Loading