Re: [PATCH] [AArch64] support -mfentry feature for arm64

[Andrew Haley <aph at redhat dot com>]

On 18/04/16 18:34, Michael Matz wrote:
> Hi,
> 
> On Mon, 18 Apr 2016, Andrew Haley wrote:
> 
>>>> That may not be safe.  Consider an implementation which looks
>>>> ahead in the instruction stream and decodes the instructions
>>>> speculatively.
>>>
>>> It should go without saying that patching instructions is followed
>>> by whatever means necessary to flush any such caches on a
>>> particular implementation (here after patching the jump, after
>>> patching the rest, and after patching the first insn again,
>>> i.e. three times).
>>
>> That doesn't necessarily help you, though, without an ISB in the reading 
>> thread.
> 
> I don't understand, which reading thread?  We're writing, not reading 
> instructions.  You mean other executing threads? 

Yes.

> I will happily declare any implementation where it's impossible to
> safely patch the instruction stream by flushing the respective
> buffers or other means completely under control of the patching
> machinery, to be broken by design. 

You can declare anything you want, but we have to program for the
architectural specification.

> What failure mode do you envision, exactly?

It's easiest just to quote from the spec:

    How far ahead of the current point of execution instructions are
    fetched from is IMPLEMENTATION DEFINED. Such prefetching can be
    either a fixed or a dynamically varying number of instructions,
    and can follow any or all possible future execution paths. For all
    types of memory:

       The PE might have fetched the instructions from memory at any
       time since the last Context synchronization operation on that
       PE.

       Any instructions fetched in this way might be executed multiple
       times, if this is required by the execution of the program,
       without being re-fetched from memory. In the absence of an ISB,
       there is no limit on the number of times such an instruction
       might be executed without being re-fetched from memory.

    The ARM architecture does not require the hardware to ensure
    coherency between instruction caches and memory, even for
    locations of shared memory.

So, if you write a bunch of instructions (which might have been
pre-fetched) and then rewrite a NOP to jump to those instructions you
need to make sure that the thread which might be running concurrently
does an ISB.

Note also:

    Memory accesses caused by instruction fetches are not required to
    be observed in program order, unless they are separated by an ISB
    or other context synchronization event.

So, if you modify instruction memory in one thread, other threads may
see those changes in a different order from the writing thread.  Sure,
the writing thread executes the cache maintenance instructions on its
side, but you also need to do something on the side which is executing
the instructions.

I have wondered if it might be a good idea to use an inter-processor
interrupt to force a context synchronization event across all PEs.

Andrew.