CARVIEW |
Navigation Menu
-
Notifications
You must be signed in to change notification settings - Fork 24.7k
fix codecache write_atomic path issue on Windows. #138331
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix codecache write_atomic path issue on Windows. #138331
Conversation
đź”— Helpful Linksđź§Ş See artifacts and rendered test results at hud.pytorch.org/pr/138331
Note: Links to docs will display an error until the docs builds have been completed. ❌ 5 New Failures, 1 Unrelated FailureAs of commit 92ee32e with merge base aa3ae50 ( NEW FAILURES - The following jobs have failed:
FLAKY - The following job failed but was likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
94b3999
to
6113e3e
Compare
6ff4335
to
ac8408a
Compare
ac8408a
to
31470a5
Compare
@pytorchbot rebase |
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
1. handle FileExistsError on Windows. 2. Breakdown this code into equal Windows code.
Successfully rebased |
31470a5
to
76ff6f1
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, few comments though (also, I guess we test it in prod...)
I tested the new code both on Windows/Linux. And confirmed they are works. @malfet import os
import sys
from pathlib import Path
import shutil
_IS_WINDOWS = sys.platform == "win32"
def test_case():
cwd = os.getcwd()
path1 = os.path.join(cwd, "haha1.txt")
path2 = Path(os.path.join(cwd, "haha2.txt"))
try:
path2.rename(target=path1)
except FileExistsError as e_file_exist:
if not _IS_WINDOWS:
raise
# On Windows file exist is expected: https://docs.python.org/3/library/pathlib.html#pathlib.Path.rename
# Below two lines code is equal to `tmp_path.rename(path)` on non-Windows OS.
# 1. Copy tmp_file to Target(Dst) file.
shutil.copy2(src=path2, dst=path1)
# 2. Delete tmp_file.
os.remove(path2)
print("run here.")
if __name__ == "__main__":
test_case() |
@pytorchmergebot merge -f "skip unrelated error." |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
2.5.1 is an emergency release to address large regression, so fixes to a new features do not meet the criteria to be included into it. |
Fixes pytorch#138211 `Path.rename` function has Windows OS specific behavior, that will raise `FileExistsError` when the target file existing. This behavior is not happened on Linux, so I write a small repoduce code to figure out what happened. After stepping trace the repo code: ```python import os import sys from pathlib import Path _IS_WINDOWS = sys.platform == "win32" def test_case(): cwd = os.getcwd() path1 = os.path.join(cwd, "haha1.txt") path2 = Path(os.path.join(cwd, "haha2.txt")) try: path2.rename(path1) except FileExistsError as e_file_exist: if _IS_WINDOWS: # on Windows file exist is expected: https://docs.python.org/3/library/pathlib.html#pathlib.Path.rename shutil.copy2(path2, path1) os.remove(path2) else: raise e_file_exist except BaseException as e: raise e print("run here.") if __name__ == "__main__": test_case() ``` We found the code `path2.rename(path1)` can breakdown into: 1. copy file2's content to file1. 2. delete file2. So, we can implemented equal code on Windows path: ```python shutil.copy2(src=tmp_path, dst=path) os.remove(tmp_path) ``` So, we can get current PR. TODO: need cherry-pick to release/2.5 branch, CC: @atalman . Pull Request resolved: pytorch#138331 Approved by: https://github.com/malfet Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com>
Fixes #138211
Path.rename
function has Windows OS specific behavior, that will raiseFileExistsError
when the target file existing.This behavior is not happened on Linux, so I write a small repoduce code to figure out what happened.
After stepping trace the repo code:
We found the code
path2.rename(path1)
can breakdown into:So, we can implemented equal code on Windows path:
So, we can get current PR.
TODO: need cherry-pick to release/2.5 branch, CC: @atalman .
cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @iremyux @Blackhex @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @voznesenskym @penguinwu @EikanWang @Guobing-Chen @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang @aakhundov