add acos_double_grad in dygraph composite #70409

zhiminzhang0830 · 2024-12-23T08:51:57Z

PR Category

Operator Mechanism

PR Types

New features

Description

Pcard-75624
添加acos的动态图二阶反向组合

精度对齐时，参考acos测试代码，使用[-0.95, 0.95]范围内数据作为输入。

与torch对齐验证代码：

import paddle
import numpy as np
import torch
np.random.seed(42)
x_np = np.random.uniform(-0.95, 0.95, size=(100, 10000)).astype("float32")
dout_np = np.random.rand(*x_np.shape).astype("float32")
dout_np2 = np.random.rand(*x_np.shape).astype("float32")
# paddle
x = paddle.to_tensor(x_np)
x.stop_gradient = False
dout_pd = paddle.to_tensor(dout_np)
dout_pd.stop_gradient = False
dout_pd2 = paddle.to_tensor(dout_np2)
dout_pd2.stop_gradient = False
y_pd = paddle.acos(x)
y_grad_pd = paddle.grad(y_pd, x, dout_pd, create_graph=True, retain_graph=True)[0]
y_grad_grad_pd = paddle.grad(y_grad_pd, x, dout_pd2, create_graph=True, retain_graph=True)[0]
# torch
x = torch.tensor(x_np, requires_grad=True)
y_pt = torch.acos(x)
dout_pt = torch.as_tensor(dout_np)
dout_pt.requires_grad = True
dout_pt2 = torch.as_tensor(dout_np2)
dout_pt2.requires_grad = True
y_grad_pt = torch.autograd.grad(y_pt, x, dout_pt, create_graph=True, retain_graph=True)[0]
y_grad_grad_pt = torch.autograd.grad(y_grad_pt, x, dout_pt2, create_graph=True, retain_graph=True)[0]
np.testing.assert_allclose(y_grad_pd.numpy(), y_grad_pt.detach().numpy(), rtol=1e-6)
print('1st order grad check passed')
np.testing.assert_allclose(y_grad_grad_pd.numpy(), y_grad_grad_pt.detach().numpy(), rtol=1e-6)
print('2nd order grad check passed')

paddle-bot · 2024-12-23T08:52:01Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

CLAassistant · 2024-12-23T08:52:03Z

All committers have signed the CLA.

HydrogenSulfate · 2024-12-24T05:00:41Z

paddle/fluid/prim/api/composite_backward/composite_double_backward_api.h

+  if (x_grad) {
+    auto x_grad_tmp = (grad_out * (-x) * pow<T>(1 - x * x, -1.5) * grad_x_grad);
+    set_output<T>(x_grad_tmp, x_grad);
+  }
+
+  if (grad_out_grad) {
+    auto grad_out_grad_tmp = -pow<T>(1 - x * x, -0.5) * grad_x_grad;
+    set_output<T>(grad_out_grad_tmp, grad_out_grad);
+  }


参考:

Paddle/paddle/fluid/prim/api/composite_backward/composite_double_backward_api.h

Lines 625 to 645 in b3afa7b

template <typename T>

void silu_double_grad(const Tensor& x,

const Tensor& out,

const Tensor& out_grad,

const Tensor& grad_x_grad,

Tensor* grad_x,

Tensor* grad_out_grad) {

auto s = sigmoid<T>(x);

auto tmp1 = scale<T>(s, -1.0, 1.0);

auto tmp2 = scale<T>(tmp1 * x, 1.0, 1.0);

auto grad_x_grad_mul_sigmoid = grad_x_grad * s;

if (grad_out_grad) {

auto ddout = grad_x_grad_mul_sigmoid * tmp2;

set_output<T>(ddout, grad_out_grad);

}

if (grad_x) {

auto dx = grad_x_grad_mul_sigmoid * out_grad *

(scale<T>(tmp2 - out, 1.0, 1.0)) * tmp1;

set_output<T>(dx, grad_x);

}

}

，把两个分支计算的公共部分提取出来（如1-x*x)，这样能避免重复的kernel调用和计算开销

HydrogenSulfate · 2024-12-24T05:02:31Z

另外PR描述里的验证代码，构造的张量形状太小，可以在百万级别的矩阵上验证一下。

HydrogenSulfate · 2024-12-24T05:05:20Z

paddle/fluid/prim/api/composite_backward/composite_double_backward_api.h

+  // acos grad grad : ddout = -((1-x*x)^(-0.5)) * ddx, dx = dy *
+  // (-x)*((1-x*x)^(-1.5)) * ddx


Suggested change

// acos grad grad : ddout = -((1-x*x)^(-0.5)) * ddx, dx = dy *

// (-x)*((1-x*x)^(-1.5)) * ddx

// ddout = -((1-x*x)^(-0.5)) * ddx

// dx = dy * (-x)*((1-x*x)^(-1.5)) * ddx

zyfncg · 2024-12-26T08:35:04Z

paddle/phi/ops/yaml/backward.yaml

+  output : Tensor(x_grad), Tensor(grad_out_grad)
+  infer_meta :
+    func : GeneralBinaryGradInferMeta
+    param : [x, x]


第二个x最好写成grad_out，以防后面静态图使用出现一些奇怪的问题

已修改，辛苦再看下

add acos_double_grad in dygraph composite

7adc1da

zhiminzhang0830 requested review from xiaoguoguo626807, JiabinYang and phlrain as code owners December 23, 2024 08:51

zhiminzhang0830 requested a review from HydrogenSulfate December 23, 2024 08:56

del acos at paddle/fluid/prim/api/api.yaml

95c3375

HydrogenSulfate reviewed Dec 24, 2024

View reviewed changes

zhiminzhang0830 added 2 commits December 24, 2024 07:09

fix: update acos_double_grad

e8abf76

fix: test_acos_double_grad_comp_dygraph

2288f40

cubehan3 previously approved these changes Dec 24, 2024

View reviewed changes

fix bugs

b524296

zhiminzhang0830 dismissed cubehan3’s stale review via b524296 December 25, 2024 02:39

cubehan3 previously approved these changes Dec 26, 2024

View reviewed changes

zyfncg previously approved these changes Dec 26, 2024

View reviewed changes

fix: chang infer_meta param

6fb252f

zhiminzhang0830 dismissed stale reviews from zyfncg and cubehan3 via 6fb252f December 26, 2024 14:06

zyfncg approved these changes Dec 27, 2024

View reviewed changes

cubehan3 approved these changes Dec 27, 2024

View reviewed changes

HydrogenSulfate merged commit fe4d055 into PaddlePaddle:develop Dec 27, 2024
29 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add acos_double_grad in dygraph composite #70409

add acos_double_grad in dygraph composite #70409

Uh oh!

zhiminzhang0830 commented Dec 23, 2024 •

edited

Loading

Uh oh!

paddle-bot bot commented Dec 23, 2024

Uh oh!

CLAassistant commented Dec 23, 2024 •

edited

Loading

Uh oh!

HydrogenSulfate Dec 24, 2024

Uh oh!

zhiminzhang0830 Dec 24, 2024

Uh oh!

HydrogenSulfate commented Dec 24, 2024

Uh oh!

HydrogenSulfate Dec 24, 2024

Uh oh!

zhiminzhang0830 Dec 24, 2024

Uh oh!

zyfncg Dec 26, 2024

Uh oh!

zhiminzhang0830 Dec 27, 2024

Uh oh!

Uh oh!

Uh oh!

	template <typename T>
	void silu_double_grad(const Tensor& x,
	const Tensor& out,
	const Tensor& out_grad,
	const Tensor& grad_x_grad,
	Tensor* grad_x,
	Tensor* grad_out_grad) {
	auto s = sigmoid<T>(x);
	auto tmp1 = scale<T>(s, -1.0, 1.0);
	auto tmp2 = scale<T>(tmp1 * x, 1.0, 1.0);
	auto grad_x_grad_mul_sigmoid = grad_x_grad * s;
	if (grad_out_grad) {
	auto ddout = grad_x_grad_mul_sigmoid * tmp2;
	set_output<T>(ddout, grad_out_grad);
	}
	if (grad_x) {
	auto dx = grad_x_grad_mul_sigmoid * out_grad *
	(scale<T>(tmp2 - out, 1.0, 1.0)) * tmp1;
	set_output<T>(dx, grad_x);
	}
	}

		// acos grad grad : ddout = -((1-xx)^(-0.5)) ddx, dx = dy *
		// (-x)((1-xx)^(-1.5)) * ddx

add acos_double_grad in dygraph composite #70409

add acos_double_grad in dygraph composite #70409

Uh oh!

Conversation

zhiminzhang0830 commented Dec 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Category

PR Types

Description

Uh oh!

paddle-bot bot commented Dec 23, 2024

Uh oh!

CLAassistant commented Dec 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HydrogenSulfate Dec 24, 2024

Choose a reason for hiding this comment

Uh oh!

zhiminzhang0830 Dec 24, 2024

Choose a reason for hiding this comment

Uh oh!

HydrogenSulfate commented Dec 24, 2024

Uh oh!

HydrogenSulfate Dec 24, 2024

Choose a reason for hiding this comment

Uh oh!

zhiminzhang0830 Dec 24, 2024

Choose a reason for hiding this comment

Uh oh!

zyfncg Dec 26, 2024

Choose a reason for hiding this comment

Uh oh!

zhiminzhang0830 Dec 27, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zhiminzhang0830 commented Dec 23, 2024 •

edited

Loading

CLAassistant commented Dec 23, 2024 •

edited

Loading