[PATCH][amdgcn] Add runtime ISA check for amdgcn offloading

[Andrew Stubbs]

On 20/01/2020 10:08, Jakub Jelinek wrote:
> On Mon, Jan 20, 2020 at 10:00:09AM +0000, Andrew Stubbs wrote:
>>> @@ -396,6 +396,88 @@ struct gcn_image_desc
>>>     struct global_var_info *global_variables;
>>>   };
>>> +/* This enum mirrors the corresponding LLVM enum's values for all ISAs that we
>>> +   support.
>>> +   See https://llvm.org/docs/AMDGPUUsage.html#amdgpu-ef-amdgpu-mach-table */
>>> +
>>> +typedef enum {
>>> +  EF_AMDGPU_MACH_AMDGCN_GFX801 = 0x028,
>>> +  EF_AMDGPU_MACH_AMDGCN_GFX803 = 0x02a,
>>> +  EF_AMDGPU_MACH_AMDGCN_GFX900 = 0x02c,
>>> +  EF_AMDGPU_MACH_AMDGCN_GFX906 = 0x02f,
>>> +} EF_AMDGPU_MACH;
>>> +
>>> +const static int EF_AMDGPU_MACH_MASK = 0x000000ff;
>>> +typedef EF_AMDGPU_MACH gcn_isa;
>>> +
>>> +const static char* gcn_gfx801_s = "gfx801";
>>> +const static char* gcn_gfx803_s = "gfx803";
>>> +const static char* gcn_gfx900_s = "gfx900";
>>> +const static char* gcn_gfx906_s = "gfx906";
>>> +const static int gcn_isa_name_len = 6;
> 
> Does this mean that GCN has 4 incompatible ISA sets and one can't compile
> for some ISA set that covers them all?  Or is that just in case one wants
> specific optimizations for the hw he has and nothing else?

The HSA/ROCm runtime rejects binaries not built for the exact device 
present.

In practice, binaries built by GCC for GCN3 "fiji" devices would 
probably run on any of the devices we currently support, if only the 
driver would load it. It would not be optimal, but AFAIK the subset of 
the ISA we actually use is compatible.

However, in theory, the meta-data isn't quite the same (for one thing, 
GCN3 allocates registers in increments of 8, where GCN5 uses 16) so some 
programs may misbehave. I have suggested to the ROCm folks that possibly 
this could be fixed in the drivers, but that's not been done yet. I 
suppose that running sub-optimal code on an accelerator device is not a 
priority.

It would be possible for us to a) ensure we always use fully portable 
instruction and meta-data encodings in the compiler, and b) patch the 
ELF flags in libgomp at load time, to achieve some degree of 
forward-portability. We have not attempted this to date. Sooner or later 
AMD would change something in such a way that it couldn't be worked around.

Andrew