[Bug middle-end/125109] New: AMD GPU offloading performance pessimized if also compiling for NVIDIA GPU offloading

[tschwinge at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=125109

            Bug ID: 125109
           Summary: AMD GPU offloading performance pessimized if also
                    compiling for NVIDIA GPU offloading
           Product: gcc
           Version: 16.0
            Status: UNCONFIRMED
          Keywords: offload, openmp
          Severity: normal
          Priority: P3
         Component: middle-end
          Assignee: unassigned at gcc dot gnu.org
          Reporter: tschwinge at gcc dot gnu.org
                CC: amonakov at gcc dot gnu.org, ams at gcc dot gnu.org,
                    jakub at gcc dot gnu.org
  Target Milestone: ---

With the <https://github.com/UoB-HPC/miniBUDE/> benchmark, I'm observing the
following when running its OpenMP offloading variant on a system with AMD
Instinct MI210 GPU, '--iter 1', '--deck data/bm1', with GCC configured for both
AMD and NVIDIA GPU offloading:

  - If compiling for both AMD and NVIDIA GPU offloading (default), I get an
execution time of ~523.3 s.
  - If compiling for AMD GPU offloading only (explicit
'-foffload=amdgcn-amdhsa'), I get a significantly better execution time of
~409.8 s.

The latter matches the execution time of GCC configured only for AMD GPU
offloading.

Per Subversion r242714 (Git commit 6c7509bc070b29293ca9874518b89227ce05361c)
"OpenMP loop cloning for SIMT execution", we have code that (nowadays) in
'gcc/omp-general.cc:omp_max_simt_vf' (that is, in the host compiler) checks
whether nvptx offloading compilation is enabled, and in that case, different
"omp-low" IR is emitted (not inspected in detail) and therefore the AMD GPU
offloading input IR is different, compared to when nvptx offloading compilation
is not enabled.

If I disable this host-side consideration of NVIDIA GPU offloading:

     omp_max_simt_vf (void)
     {
       if (!optimize)
         return 0;
    +#if 0
       if (ENABLE_OFFLOADING)
         for (const char *c = getenv ("OFFLOAD_TARGET_NAMES"); c;)
           {
            if (startswith (c, "nvptx"))
              return 32;
            else if ((c = strchr (c, ':')))
              c++;
           }
    +#endif
       return 0;
     }

..., and again compile for both AMD and NVIDIA GPU offloading, then the
"omp-low" IR and therefore also AMD GPU offloading input IR is again equal to
the '-foffload=amdgcn-amdhsa' case, and I'm back at the good execution time
(~409.8 s).  That is, the (host-side) consideration of NVIDIA GPU offloading is
pessimizing AMD GPU offloading code generation.

Do we need to fix this up in the AMD GPU offloading input IR side, or do we
need more considerate "omp-low" IR emission (such that any "not applicable"
code gets optimized away early in the AMD GPU offloading input IR?), or do we
need better abstractions (IFN?) generally?


That said, there are also other cases of 'getenv ("OFFLOAD_TARGET_NAMES")' in
host-side code (such as commit
r15-4985-g5c9de3df8547682bfb3d484d7d28a27776bf979c "openmp: Tune omp_max_vf for
offload targets": 'gcc/omp-general.cc:omp_max_vf'), which also might need
inspection.