How to get best AVX2 performance from gfortran?

[Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]]

On 04/18/2018 11:40 PM, Jerry DeLisle wrote:
> On 04/18/2018 07:22 AM, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND 
> APPLICATIONS INC] wrote:
>> Dear GNU Fortran Gurus,
>>
>> I have a question mainly based around AVX2 and flags.
>>
> --- snip ---
> 
>>
>> So, my questions to you gurus are: Is there something I could try 
>> adding to my gfortran options that might help with this discrepancy 
>> between Intel AVX2 and GCC? Or perhaps I need to *remove* something 
>> (some flag kills the vectorizer)?
>>
>> At this point I'm mainly searching for speed for speed's sake so I'm 
>> willing to try anything even if it's thought of as "stupid agressive". 
>> As long as it doesn't crash the model, I'm willing to try and then 
>> check to make sure the world isn't boiling or freezing because I broke 
>> reality.
>>
>> Matt
> 
> Well I have seen some codes pessimized by -O3 so try -O2.  Also 
> generally I do -march=native.
> 
> Also you may try something like this:
> 
> -ffast-math -ftree-vectorize -funroll-loops --param max-unroll-times=4 
> -mprefer-avx128
> 
> Curious about what you find out, generally speakig one has to fine tune 
> to the code being compiled since there are many many factors that can 
> influence the outcome.

An overall update. Jerry's flags with -mno-fma and -O2:

-O2 -march=native -ffast-math -ftree-vectorize -funroll-loops --param 
max-unroll-times=4 -mprefer-avx128 -mno-fma

seem to be the best I've found.:

Original:

    TRACER_2D 273.3723
    REMAPPING 115.3358

Jerry's:

    TRACER_2D 138.4858
    REMAPPING  94.4236

Though compared to Intel:

    TRACER_2D 24.7418
    REMAPPING 27.2644

still a ways to go.

That said, it turns out for now I can't actually *use* these flags. One 
of the requirements for our model is that it regress under layout change 
in MPI. A lot of work was put into getting flags with Intel that did 
this but still vectorized (-fp-model consistent was the key, I think). 
The offending flag in GNU seems to be -ffast-math, but if you remove 
that from Jerry's flags:

Native O2 Delisle NoFMA NoFastMath:

    TRACER_2D 264.1194
    REMAPPING 113.4083

Well, nuts. Looks like -ffast-math was the special flag.

I'll ask for a GCC 8 snapshot to be installed on our cluster to test 
with that at least to see if vectorization/speed is improved, though 
without reproducibility...

Thanks,
Matt


-- 
Matt Thompson, SSAI, Sr Scientific Programmer/Analyst
NASA GSFC,    Global Modeling and Assimilation Office
Code 610.1,  8800 Greenbelt Rd,  Greenbelt,  MD 20771
Phone: 301-614-6712                 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson