CARVIEW |
Navigation Menu
-
Notifications
You must be signed in to change notification settings - Fork 24.7k
Friendly catch exception when fail to initialize XPU devices #141658
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141658
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 87d974a with merge base 6e61ff4 ( FLAKY - The following job failed but was likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
c10/xpu/XPUFunctions.cpp
Outdated
enumDevices(gDevicePool.devices); | ||
} catch (const sycl::exception& e) { | ||
TORCH_WARN( | ||
"Failed to initialize XPU devices. Did you install the driver correctly?"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The message is not informative. Could you help refine the message?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated. Do you think it is OK?
TORCH_WARN( | ||
"Failed to initialize XPU devices. The driver may not be installed, installed incorrectly, or incompatible with the current setup. ", | ||
"Please refer to the guideline (https://github.com/pytorch/pytorch?tab=readme-ov-file#intel-gpu-support) for proper installation and configuration."); | ||
return; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With this change, torch will not crash itself even without any XPU device.
Sounds good to provide a smooth warning, instead of ugly crash.
I assume we have well-defined behavior for below cases in this scenario,
- The init value of device pool is always empty
- No undefined behavior happens even if user does not check the device count.
Pls. note, initDevicePoolCallOnce is called once, that means, the warning will be raised only once.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the device pool is always initialized to empty. PyTorch will raise RuntimeError in the following code if the user would like to call the other runtime API. We deliberately warn this message only once to avoid interrupting the user over time.
pytorch/c10/xpu/XPUFunctions.cpp
Lines 173 to 178 in f35bb55
DeviceIndex device_count_ensure_non_zero() { | |
auto count = device_count(); | |
// Zero gpus could produce a warning in `device_count` but we fail here. | |
TORCH_CHECK(count, "No XPU devices are available."); | |
return count; | |
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
"Unrelated failure, there is an issue to track it #141705" |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
…#141658) Pull Request resolved: pytorch#141658 Approved by: https://github.com/EikanWang
Stack from ghstack (oldest at bottom):
cc @gujinghui @EikanWang @fengyuan14