stack arenas using alloca

[Michael Clark]

On 8/23/24 15:57, Michael Clark wrote:
> On 8/23/24 15:46, Michael Clark wrote:
>> one more thing. it doesn't require PT_GNU_STACK or writable stacks 
>> like GCC nested functions. 🙂 so I think it is safer but it does have 
>> safety issues, mostly related to stack overflows but its going to need 
>> some careful analysis with respect to ROP.
> 
> brain isn't working. of course it needs writable stacks. it doesn't need 
> executable stacks. starting to think about [[allocareturn]] ROP...

if you loaded the return address from the fixed frame in the epilogue 
into a temporary during restore of callee save registers which usually 
unwinds the dynamic offset, you could translate the return instruction 
into an indirect branch. that gives similar ROP protection to functions 
without dynamic alloca returns.

then the biggest problem is stack overflows. we can use alloca anyway 
just we have to call the function twice with some funky convention that 
suppresses array writes for null pointers because we have to call it in 
the parent frame. this just makes it somewhat more useful for functions 
returning lists where you don't want to allocate on the heap.

if you don't have any side effects in loops besides array writes, you 
could have some funky feature where you run the loop once for a size. I 
tend to write a lot of code like that by hand and hate it. just look at 
code that uses the Vulkan API. it is full of client code that does that.

that's sort of where this line of thinking emerged. it makes it harder 
if you want to write to either stack or heap/data though because you 
need a couple of different translations of the function. I would like 
implicitly deduced memory region qualifiers like _StackRegion, 
_DataRegion, _HeapRegion, and _ThreadRegion for TLS. and i'm making the 
assumption the stack is data race free which it is 99.9% of the time. 
then you can implicitly typedef _StackRegion MyType MyType in metadata 
for a translation that has an input type with _StackRegion.

I'd like an OS where stack can only escape to fibres and fibres can only 
run on one OS thread. that way we could allocate page table roots or 
overlay page tables for threads to make stack fault when accessed by 
another thread. grsecurity already does this for Linux kernel threads.

Michael.