[Typing] 设置环境变量 `SKIP_STUB_GEN` 阻止 `gen_tensor_stub.py` 生成 stub 文件 #65657

megemini · 2024-07-03T05:39:20Z

PR Category

User Experience

PR Types

Improvements

Description

设置环境变量 SKIP_STUB_GEN 阻止 gen_tensor_stub.py 生成 stub 文件

如果接受此 PR 的方案，则可以关闭 #65646

@SigureMo

closes #65646

paddle-bot · 2024-07-03T05:39:24Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

… tensor_stub_env

megemini · 2024-07-03T10:52:33Z

情况1：默认编译，编译通过。

命令：> time make -j6 2>&1 | tee ../../pmake.log

日志：

changing mode of build/bdist.linux-x86_64/wheel/paddlepaddle_gpu-0.0.0.data/scripts/paddle to 775
-- Generate stub file tensor.pyi ... 
-- End Generate stub file tensor.pyi ... 
[100%] Built target paddle_copy
make -j6 2>&1  60.03s user 8.60s system 140% cpu 48.859 total
tee ../../pmake.log  0.00s user 0.04s system 0% cpu 48.859 total

情况2：默认编译，编译不通过。

如导入 paddle 失败等情况。

命令同上：> time make -j6 2>&1 | tee ../../pmake.log

日志：

[100%] Built target inference_lib_dist
Traceback (most recent call last):
  File "/home/shun/Documents/Projects/paddle/megemini/Paddle/tools/gen_tensor_stub.py", line 411, in try_import_paddle
    return importlib.import_module('paddlea')
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'paddlea'
ERROR: Can NOT import paddle from `tools/gen_tensor_stub.py` before installation.
    So the stub file `python/paddle/tensor/tensor.pyi` of `paddle.Tensor` may be lost.
    We COULD import paddle without installation with all libs (.dll or .so) copied into dir `paddle/libs`,
    or path already been set for the system. Try the following steps to locate the problem.
    1. Build with `SKIP_STUB_GEN=ON make -j$(nproc)`.
    2. Install the wheel from `build/python/dist`.
    3. Try to `import paddle` and check the problems.
            -- Generate stub file tensor.pyi ... 
Traceback (most recent call last):
  File "setup.py", line 1197, in <module>
    generate_tensor_stub('/home/shun/Documents/Projects/paddle/megemini/Paddle/build', '/home/shun/Documents/Projects/paddle/megemini/Paddle')
  File "setup.py", line 1183, in generate_tensor_stub
    gen_tensor_stub.generate_stub_file(
  File "/home/shun/Documents/Projects/paddle/megemini/Paddle/tools/gen_tensor_stub.py", line 566, in generate_stub_file
    tensor_members = get_tensor_members(module)
  File "/home/shun/Documents/Projects/paddle/megemini/Paddle/tools/gen_tensor_stub.py", line 432, in get_tensor_members
    raise (
ModuleNotFoundError: Can NOT import paddle from tools/gen_tensor_stub.py.
make[2]: *** [python/CMakeFiles/paddle_copy.dir/build.make:73: python/build/.timestamp_wheel] Error 1
make[1]: *** [CMakeFiles/Makefile2:55102: python/CMakeFiles/paddle_copy.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
make -j6 2>&1  27.52s user 5.71s system 233% cpu 14.250 total
tee ../../pmake.log  0.00s user 0.02s system 0% cpu 14.250 total

情况3：跳过生成 stub，编译通过。

命令：> time SKIP_STUB_GEN=ON make -j6 2>&1 | tee ../../pmake.log

日志：

removing build/bdist.linux-x86_64/wheel
[100%] Built target paddle_copy

megemini · 2024-07-03T14:02:53Z

Update 20240703

修改 tensor.prototype.pyi ，不再使用 Protocol

发现个问题，如果使用 Protocol ，vscode 会推断 Tensor 为 Any ，但是使用 mypy 检查是没问题的：

使用 Protocol：

改为类继承：

这里已经标注了类属性：

class Uniform(distribution.Distribution):
    low: Tensor
    high: Tensor

另外，把 Tensor 的 docstring 从 AbstractTensor 移到 Tensor 里面，不然 vscode 获取不到～

SigureMo · 2024-07-03T16:41:06Z

修改 tensor.prototype.pyi ，不再使用 Protocol

emmm，之后这种完全不相关的事情最好拆出去单独一个 PR

This reverts commit c93447b.

SigureMo · 2024-07-04T07:28:02Z

python/setup.py.in

@@ -1193,7 +1193,8 @@ def generate_tensor_stub(paddle_binary_dir, paddle_source_dir):
    print('-'*2, 'End Generate stub file tensor.pyi ... ')

 # generate stub file `tensor.pyi`
-generate_tensor_stub('${PADDLE_BINARY_DIR}', '${PADDLE_SOURCE_DIR}')
+if not os.getenv("SKIP_STUB_GEN", None) == 'ON':


Suggested change

if not os.getenv("SKIP_STUB_GEN", None) == 'ON':

if not os.getenv("SKIP_STUB_GEN", None).lower() in ['y', 'yes', 't', 'true', 'on', '1']:

稍微增加下容错吧，这些都是常见的表示环境变量开启的 value，setup.py 同

SigureMo

LGTMeow

zyfncg

LGTM for setup.py.in

[Change] gen stub env

4ce6a60

paddle-bot bot added the contributor External developers label Jul 3, 2024

luotao1 added HappyOpenSource 快乐开源活动issue与PR PaddlePaddle Hackathon and removed HappyOpenSource 快乐开源活动issue与PR labels Jul 3, 2024

luotao1 assigned luotao1 and SigureMo Jul 3, 2024

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

da388f3

… tensor_stub_env

[Update] tensor.prototype.pyi

c93447b

Revert "[Update] tensor.prototype.pyi"

9a5b57e

This reverts commit c93447b.

megemini mentioned this pull request Jul 4, 2024

[Typing] 修改 tensor.prototype.pyi 不再使用 Protocol 以及缺失的参数类型 #65697

Merged

SigureMo reviewed Jul 4, 2024

View reviewed changes

[Update] getenv ops

a501703

risemeup1 approved these changes Jul 5, 2024

View reviewed changes

megemini mentioned this pull request Jul 5, 2024

[tmp][Typing] gen_tensor_stub.py 如果无法导入 paddle 则使用默认的模板文件 #65646

Closed

SigureMo approved these changes Jul 5, 2024

View reviewed changes

zyfncg approved these changes Jul 5, 2024

View reviewed changes

SigureMo merged commit c34f484 into PaddlePaddle:develop Jul 5, 2024
31 of 32 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Typing] 设置环境变量 `SKIP_STUB_GEN` 阻止 `gen_tensor_stub.py` 生成 stub 文件 #65657

[Typing] 设置环境变量 `SKIP_STUB_GEN` 阻止 `gen_tensor_stub.py` 生成 stub 文件 #65657

Uh oh!

megemini commented Jul 3, 2024 •

edited by SigureMo

Loading

Uh oh!

paddle-bot bot commented Jul 3, 2024

Uh oh!

megemini commented Jul 3, 2024 •

edited

Loading

Uh oh!

megemini commented Jul 3, 2024

Uh oh!

SigureMo commented Jul 3, 2024

Uh oh!

SigureMo Jul 4, 2024

Uh oh!

SigureMo left a comment

Uh oh!

zyfncg left a comment

Uh oh!

Uh oh!

Uh oh!

	if not os.getenv("SKIP_STUB_GEN", None) == 'ON':
	if not os.getenv("SKIP_STUB_GEN", None).lower() in ['y', 'yes', 't', 'true', 'on', '1']:

[Typing] 设置环境变量 SKIP_STUB_GEN 阻止 gen_tensor_stub.py 生成 stub 文件 #65657

[Typing] 设置环境变量 SKIP_STUB_GEN 阻止 gen_tensor_stub.py 生成 stub 文件 #65657

Uh oh!

Conversation

megemini commented Jul 3, 2024 • edited by SigureMo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Category

PR Types

Description

Uh oh!

paddle-bot bot commented Jul 3, 2024

Uh oh!

megemini commented Jul 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

情况1：默认编译，编译通过。

情况2：默认编译，编译不通过。

情况3：跳过生成 stub，编译通过。

Uh oh!

megemini commented Jul 3, 2024

Update 20240703

Uh oh!

SigureMo commented Jul 3, 2024

Uh oh!

SigureMo Jul 4, 2024

Choose a reason for hiding this comment

Uh oh!

SigureMo left a comment

Choose a reason for hiding this comment

Uh oh!

zyfncg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

[Typing] 设置环境变量 `SKIP_STUB_GEN` 阻止 `gen_tensor_stub.py` 生成 stub 文件 #65657

[Typing] 设置环境变量 `SKIP_STUB_GEN` 阻止 `gen_tensor_stub.py` 生成 stub 文件 #65657

megemini commented Jul 3, 2024 •

edited by SigureMo

Loading

megemini commented Jul 3, 2024 •

edited

Loading