Add README for torch._inductor.runtime #141492

ezyang · 2024-11-25T15:37:54Z

Stack from ghstack (oldest at bottom):

Signed-off-by: Edward Z. Yang ezyang@meta.com

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang @aakhundov

[ghstack-poisoned]

pytorch-bot · 2024-11-25T15:37:58Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141492

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit 35e86f1 with merge base 4c1f50a ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
convnext_base

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

inductor / cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#141498)
convnext_base

This comment was automatically generated by Dr. CI and updates every 15 minutes.

jansel · 2024-11-25T19:02:21Z

torch/_inductor/runtime/README.md

+There's not really a coherent concept for this folder; in general it contains
+code for:
+
+1. Out-of-process Triton compilation (e.g., when doing multiprocess parallel
+   Triton compile)
+
+2. Code that is accessed by the code we generated in Inductor codegen (the
+   "runtime", although not really a true runtime because we don't actually run
+   this code when running, it's what the Triton compilation uses).
+
+It isn't really a true runtime (aka, code that needs to be invoked when
+actually running Inductor compiled kernels).
+
+One of the original motivations for this directory split was so that the
+subprocesses could access Triton and our compiler support code without having
+to import torch.  However, we never actually go this to work, so there are
+still plenty of torch imports in this folder.


Suggested change

There's not really a coherent concept for this folder; in general it contains

code for:

1. Out-of-process Triton compilation (e.g., when doing multiprocess parallel

Triton compile)

2. Code that is accessed by the code we generated in Inductor codegen (the

"runtime", although not really a true runtime because we don't actually run

this code when running, it's what the Triton compilation uses).

It isn't really a true runtime (aka, code that needs to be invoked when

actually running Inductor compiled kernels).

One of the original motivations for this directory split was so that the

subprocesses could access Triton and our compiler support code without having

to import torch. However, we never actually go this to work, so there are

still plenty of torch imports in this folder.

This folder contains code needed at runtime by the output code of TorchInductor. The output code of TorchInductor will import `torch` and `torch._inductor.runtime`, but should not import from other files in `torch._inductor.*`.

Runtime includes Triton/C++ compiles, which are compiled (sometimes in parallel) when the output code of TorchInductor is imported. It also includes the autotuning code and heuristics to decide block sizes of generated code.

One of the original motivations for this directory split was so that the Triton compile

subprocesses could access Triton and our compiler support code while mocking out most of `torch`, which can take seconds to import (sometimes more than a Triton compile itself). An abandoned prototype of this can be found [here](https://github.com/pytorch/pytorch/pull/124682/files).

This directory does contain the code needed to run Inductor compiled kernels/code. In that sense I think it is a true runtime and I think this directory structure makes sense even without the MockTorch stuff landing.

Not importing torch is not a requirement of the prototype in [inductor] Add thread+subproc-based worker_start_method #124682 -- triton imports torch, so that requirement is clearly not attainable. Instead, the prototype mocks out torch by doing:

sys.modules["torch"] = MockTorch()

with some trickery to allow only files inside torch to be imported.

I took this suggestion and reworded a little bit of it

[ghstack-poisoned]

Signed-off-by: Edward Z. Yang <ezyang@meta.com> ghstack-source-id: 98af2af Pull Request resolved: #141492

ezyang · 2024-11-26T14:41:11Z

@pytorchbot merge -f "unrelated failures"

pytorchmergebot · 2024-11-26T14:42:42Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Do code review with whitespace changes off. Check comments for what I changed. Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: #141574 Approved by: https://github.com/bobrenjc93, https://github.com/jansel ghstack dependencies: #141491, #141492

Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: #141654 Approved by: https://github.com/aorenste, https://github.com/jansel ghstack dependencies: #141491, #141492, #141574

Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: pytorch#141492 Approved by: https://github.com/jansel ghstack dependencies: pytorch#141491

Do code review with whitespace changes off. Check comments for what I changed. Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: pytorch#141574 Approved by: https://github.com/bobrenjc93, https://github.com/jansel ghstack dependencies: pytorch#141491, pytorch#141492

Signed-off-by: Edward Z. Yang <ezyang@meta.com> Pull Request resolved: pytorch#141654 Approved by: https://github.com/aorenste, https://github.com/jansel ghstack dependencies: pytorch#141491, pytorch#141492, pytorch#141574

Update

8520c46

[ghstack-poisoned]

ezyang mentioned this pull request Nov 25, 2024

AFAICT, compile workers never actually mocked torch #141491

Closed

pytorch-bot bot added ciflow/inductor module: inductor topic: not user facing topic category labels Nov 25, 2024

github-actions bot requested review from albanD, antoniojkim, bdhirsh, miladm and SherlockNoMad November 25, 2024 15:38

ezyang requested review from jansel and masnesral November 25, 2024 15:38

jansel requested changes Nov 25, 2024

View reviewed changes

Update

35e86f1

[ghstack-poisoned]

ezyang added a commit that referenced this pull request Nov 25, 2024

Add README for torch._inductor.runtime

6af9549

Signed-off-by: Edward Z. Yang <ezyang@meta.com> ghstack-source-id: 98af2af Pull Request resolved: #141492

jansel approved these changes Nov 25, 2024

View reviewed changes

pytorchmergebot added the merging label Nov 26, 2024

pytorchmergebot added the Merged label Nov 26, 2024

pytorchmergebot closed this in 42ab612 Nov 26, 2024

pytorchmergebot removed the merging label Nov 26, 2024

github-actions bot deleted the gh/ezyang/3014/head branch December 27, 2024 02:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add README for torch._inductor.runtime #141492

Add README for torch._inductor.runtime #141492

Uh oh!

ezyang commented Nov 25, 2024 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Nov 25, 2024 •

edited

Loading

Uh oh!

jansel Nov 25, 2024

Uh oh!

ezyang Nov 25, 2024

Uh oh!

ezyang commented Nov 26, 2024

Uh oh!

pytorchmergebot commented Nov 26, 2024

Uh oh!

Uh oh!

Add README for torch._inductor.runtime #141492

Add README for torch._inductor.runtime #141492

Uh oh!

Conversation

ezyang commented Nov 25, 2024 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141492

✅ You can merge normally! (2 Unrelated Failures)

Uh oh!

jansel Nov 25, 2024

Choose a reason for hiding this comment

Uh oh!

ezyang Nov 25, 2024

Choose a reason for hiding this comment

Uh oh!

ezyang commented Nov 26, 2024

Uh oh!

pytorchmergebot commented Nov 26, 2024

Merge started

Uh oh!

Uh oh!

ezyang commented Nov 25, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Nov 25, 2024 •

edited

Loading