@fengyuentau, excellent! is there such a dramatic difference really?

btw, I played a bit with 'erf' approximation, as GELU can be computed exactly via erf: GELU(x) = x/2*(1 + erf(x/sqrt(2))) and found that erf can be computed very accurately without any v_select() and without v_exp():

#include <math.h>
#include <algorithm>
#include <stdio.h>
int main(int argc, char** argv) {
    float maxerr = 0.f;
    for (int i = -100000; i < 100000000; i++) {
        float x0 = i*0.0001f;
        float y0 = erff(x0);
        float x = fabsf(x0), sx = x0 >= 0 ? 1.f : -1.f;
        // see https://en.wikipedia.org/wiki/Error_function; one of Abramowitz and Stegun approximations
        float d = (((((0.0000430638f*x + 0.0002765672f)*x + 0.0001520143f)*x +
            0.0092705272f)*x + 0.0422820123f)*x + 0.0705230784f)*x + 1.f;
        d = 1.f/d;        
        d *= d; d *= d;
        d *= d; d *= d;
        float y1 = sx*(1.f - d);
        float err = fabsf(y0 - y1);
        maxerr = std::max(maxerr, err);
    }
    printf("maxerr=%.3g\n", maxerr);
    return 0;
}

note that extracting the sign sx and then putting it back (sx*...) can be done with element-wise v_and() and v_or() intrinsics and the sign mask 0x80000000 - no select is needed here.

Please, try it out.

is there such a dramatic difference really?

I will test on other platforms as well to see whether they can reach the same improvement.

I did not notice erf(-x) = -erf(x) (it is indeed true since erf is an odd function) in the beginning. Let me try your implementation.

this is another implementation that follows yet another Abramowitz & Stegun approximation and also matches vectorized version from PyTorch (https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/cpu/vec/vec512/vec512_float.h#L187-L218):

inline float fast_erf(float x) {
   float sx = x >= 0 ? 1.f : -1.f;
   float t = 1.f/fmaf(fabsf(x), 0.3275911f, 1.f);
   float r = fmaf(1.061405429f, t, -1.453152027f);
   r = fmaf(r, t, 1.421413741f);
   r = fmaf(r, t, -0.284496736f);
   r = fmaf(r, t, 0.254829592f);
   return sx*(1.f - r*t*expf(-x*x));
}

I guess, it should be slower than the previous version that I suggested, but 1) it matches PyTorch and 2) it is more accurate, especially around 0.

@vpisarev With accuracy issue resolved (hopefully), the updated performance results are

#####
# M1
#####
# Paddle
[ PERFSTAT ]    (samples=100   mean=0.13   median=0.13   min=0.11   stddev=0.02 (13.2%))
# Wiki
[ PERFSTAT ]    (samples=100   mean=0.07   median=0.07   min=0.06   stddev=0.01 (11.5%))
# PyTorch
[ PERFSTAT ]    (samples=100   mean=0.10   median=0.09   min=0.08   stddev=0.01 (14.3%))
#####
# i7-12700K
#####
# Paddle
[ PERFSTAT ]    (samples=100   mean=0.05   median=0.05   min=0.05   stddev=0.00 (0.4%))
# Wiki
[ PERFSTAT ]    (samples=100   mean=0.02   median=0.02   min=0.02   stddev=0.00 (1.5%))
# PyTorch
[ PERFSTAT ]    (samples=100   mean=0.04   median=0.04   min=0.04   stddev=0.00 (0.5%))

Note that the method from wiki does not need v_exp.

@fengyuentau, thank you for the detailed experiments! What about accuracy (absolute and relative)? how far are those implementations from std::erf()?

All the dnn accuracy tests are now green. I don't know how to give absolute and relative accuracy though.

ok, I found that PyTorch and Paddle Paddle versions are very close to each other. Paddle Paddle is slightly more accurate, but the difference is small. They both are noticeably more accurate than the fastest 'exp-less' formula, especially around 0, but when you compute GELU, i.e. multiply by 'x*0.5' in the end, the drop in accuracy is not as noticeable.

May I suggest to drop Paddle Paddle version to keep the source more compact? 'wiki' version might be preserved, just in case, but I would use 'PyTorch' approximation as the default option.

also, please rewrite the implementation using scalable universal intrinsics. This way we could get even better performance if we move those kernels to separate dynamically dispatched source files and compile them with AVX2/AVX512/RVV

ok, I found that PyTorch and Paddle Paddle versions are very close to each other. Paddle Paddle is slightly more accurate, but the difference is small. They both are noticeably more accurate than the fastest 'exp-less' formula, especially around 0, but when you compute GELU, i.e. multiply by 'x*0.5' in the end, the drop in accuracy is not as noticeable.

May I suggest to drop Paddle Paddle version to keep the source more compact? 'wiki' version might be preserved, just in case, but I would use 'PyTorch' approximation as the default option.

Thank you for the test. I think I can do it next time comparing difference between accuracy of implementations. Let's keep paddle and wiki version in the git history and keep the pytorch one in the code only.

also, please rewrite the implementation using scalable universal intrinsics. This way we could get even better performance if we move those kernels to separate dynamically dispatched source files and compile them with AVX2/AVX512/RVV

I am not sure whether I do it in the right way by only changing all v_setall_f32 to vx_setall_f32. I don't find a straight example on the usage of scalable intrinsics.

v_erf_approx is only used in gelu. Maybe we do not need to put it in univeral intrinsics.

v_erf_approx is only used in gelu. Maybe we do not need to put it in univeral intrinsics.

Discussed and decided to put v_erf in universal intrinsics.

The optimization totally makes sense!
Jetson tk1 (armv7+neon):

elementwise::Layer_Elementwise::({ 1, 50, 3072 }, "Gelu", OCV/CPU)                                                                               6.855      2.593      2.64 

Core i5-2500K (AVX, no AVX2):

elementwise::Layer_Elementwise::({ 1, 50, 3072 }, "Gelu", OCV/CPU)                                                                               1.256      0.434      2.89   
elementwise::Layer_Elementwise::({ 1, 50, 3072 }, "Gelu", OCV/OCL)                                                                               0.265      0.258      1.03   
elementwise::Layer_Elementwise::({ 1, 50, 3072 }, "Gelu", OCV/OCL_FP16)                                                                          0.261      0.257      1.02  

@fengyuentau please rebase and switch to new v_erf. Manual port to 5.x is required too.

@vpisarev please take a look again.

@fengyuentau, let's delay integration of this PR a bit while @WanliZhong will add vectorized v_erf(). What do you think?

@fengyuentau, let's delay integration of this PR a bit while @WanliZhong will add vectorized v_erf(). What do you think?

It has been done in #25872.

cool!