[SOT] Split NumpyVariable for separate array/number handling #71216

DrRyanHuang · 2025-02-20T08:53:52Z

PR Category

Execute Infrastructure

PR Types

Bug fixes

Description

Currently, when creating numpy.ndarray within the dynamic-to-static function, both SOT-based and AST-based dynamic-to-static approaches encounter segmentation faults when processing numpy ndarrays.

import paddle
import numpy as np
def fn(x):
    y = np.random.rand(11, 1)
    z = x + y
    return z
static_fn = paddle.jit.to_static(fn, full_graph=False)
x = paddle.rand([11, 1], dtype="float32")
out = static_fn(x)
print(out)

--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
0   paddle::pybind::static_api_full(_object*, _object*, _object*)
1   paddle::pybind::PyObject_ToDouble(_object*)
----------------------
Error Message Summary:
----------------------
FatalError: `Segmentation fault` is detected by the operating system.
  [TimeInfo: *** Aborted at 1740040223 (unix time) try "date -d @1740040223" if you are using GNU date ***]
  [SignalInfo: *** SIGSEGV (@0x0) received by PID 35955 (TID 0x7fbdbd58d740) from PID 0 ***]
Segmentation fault (core dumped)

The solution involves splitting NumpyVariable handling into two distinct cases: numpy.Number (already supported in SOT) and numpy.ndarray (to be supported subsequently).

When encountering operations between paddle.Tensor and numpy.ndarray, it will fall back to dynamic graph execution.

This issue currently causes identical segmentation faults in the Co-Deformable-DETR-R50_CINN model of PaddleDetection.

https://github.com/PaddlePaddle/PaddleDetection/blob/59d5f5ebebc2a380f5f07dd413a056b75af01f2a/ppdet/modeling/assigners/hungarian_assigner.py#L415

        # regression iou cost, defaultly giou is used in official DETR.
        bboxes = bbox_cxcywh_to_xyxy(bbox_pred) * factor
        iou_cost = self.iou_cost(bboxes, gt_bboxes)
        # weighted sum of above three costs
        cost = cls_cost + reg_cost + iou_cost            # <---------------- 这里报错 iou_cost 是 numpy array
        # 3. do Hungarian matching on CPU using linear_sum_assignment
        cost = cost.detach().cpu()
        if linear_sum_assignment is None:
            raise ImportError('Please run "pip install scipy" '

[EXECUTOR RUN] Start execute opcode: <code object $resume_51@assign_218fe at 0x7f5fc7fe0190, file "/workspace/tmp/PaddleX/paddlex/repo_manager/repos/PaddleDetection/ppdet/modeling/assigners/hungarian_assigner.py", line 344>
[Translate Executor]: (line 415) LOAD_FAST    cls_cost, stack is []
[Translate Executor]: (line 415) LOAD_FAST    reg_cost, stack is [TensorVariable([300, 1], float32, False, var_2485, None, cls_cost, object_72417)]
[Translate Executor]: (line 415) BINARY_ADD   None, stack is [TensorVariable([300, 1], float32, False, var_2485, None, cls_cost, object_72417), TensorVariable([300, 1], float32, False, var_2486, None, reg_cost, object_72421)]
         inputs : ((var_2485, var_2486), {}) 
[Translate Executor]: (line 415) LOAD_FAST    resume_51_stack_0, stack is [TensorVariable([300, 1], float32, False, var_2490, None, tmp_var_cls_cost_reg_cost, object_72611)]
[Translate Executor]: (line 415) BINARY_ADD   None, stack is [TensorVariable([300, 1], float32, False, var_2490, None, tmp_var_cls_cost_reg_cost, object_72611), NumpyArrVariable([[-1.15825517]
 [-1.23576199]
 [-1.28707552]
 [ 0.27751816]], resume_51_stack_0, object_72416)]
--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
0   paddle::pybind::static_api_full(_object*, _object*, _object*)
1   paddle::pybind::PyObject_ToDouble(_object*)
----------------------
Error Message Summary:
----------------------
FatalError: `Segmentation fault` is detected by the operating system.
  [TimeInfo: *** Aborted at 1740040843 (unix time) try "date -d @1740040843" if you are using GNU date ***]
  [SignalInfo: *** SIGSEGV (@0x0) received by PID 8417 (TID 0x7f5fe5bd2740) from PID 0 ***]

This PR is under @SigureMo ’s guidance.

PCard-66972

paddle-bot · 2025-02-20T08:53:59Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

python/paddle/jit/sot/opcode_translator/executor/variables/__init__.py

python/paddle/jit/sot/opcode_translator/executor/variables/basic.py

SigureMo

LGTMeow

SigureMo · 2025-02-20T13:44:33Z

python/paddle/jit/sot/opcode_translator/executor/variables/basic.py

+        return None
+
+
+class NumpyNumberVariable(NumpyVariable):
    @check_guard
    def make_stringified_guard(self) -> list[StringifiedExpression]:
        frame_value_tracer = self.tracker.trace_value_from_frame()
        obj_free_var_name = f"__{self.id}"


这行用不到，后面可以删了

…addle#71216)

split NumpyVariable

ae13598

paddle-bot bot added the contributor External developers label Feb 20, 2025

SigureMo reviewed Feb 20, 2025

View reviewed changes

python/paddle/jit/sot/opcode_translator/executor/variables/__init__.py Outdated Show resolved Hide resolved

python/paddle/jit/sot/opcode_translator/executor/variables/basic.py Outdated Show resolved Hide resolved

create subclass make_stringified_guard

7513d95

SigureMo changed the title ~~[SOT] Split NumpyVariable for separate array/number handling~~ [SOT][3.13] Split NumpyVariable for separate array/number handling Feb 20, 2025

SigureMo approved these changes Feb 20, 2025

View reviewed changes

SigureMo changed the title ~~[SOT][3.13] Split NumpyVariable for separate array/number handling~~ [SOT] Split NumpyVariable for separate array/number handling Feb 20, 2025

SigureMo merged commit 399da2a into PaddlePaddle:develop Feb 20, 2025
35 checks passed

DrRyanHuang deleted the numpyVar branch February 20, 2025 16:02

Enigmatisms pushed a commit to Enigmatisms/Paddle that referenced this pull request Mar 6, 2025

[SOT] Split NumpyVariable for separate array/number handling (PaddleP…

538d9ba

…addle#71216)

gouzil mentioned this pull request Mar 28, 2025

SOT Guard 机制性能优化 #69264

Open

YqGe585 pushed a commit to YqGe585/Paddle that referenced this pull request May 7, 2025

[SOT] Split NumpyVariable for separate array/number handling (PaddleP…

2f4d947

…addle#71216)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SOT] Split NumpyVariable for separate array/number handling #71216

[SOT] Split NumpyVariable for separate array/number handling #71216

Uh oh!

DrRyanHuang commented Feb 20, 2025

Uh oh!

paddle-bot bot commented Feb 20, 2025

Uh oh!

Uh oh!

Uh oh!

SigureMo left a comment

Uh oh!

SigureMo Feb 20, 2025

Uh oh!

Uh oh!

Uh oh!

[SOT] Split NumpyVariable for separate array/number handling #71216

[SOT] Split NumpyVariable for separate array/number handling #71216

Uh oh!

Conversation

DrRyanHuang commented Feb 20, 2025

PR Category

PR Types

Description

Uh oh!

paddle-bot bot commented Feb 20, 2025

Uh oh!

Uh oh!

Uh oh!

SigureMo left a comment

Choose a reason for hiding this comment

Uh oh!

SigureMo Feb 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!