Stack smashing detected if the code is compiled with -O1 or with -fomit-frame-pointer. Everything fine, when compiled without optimization or with -fno-omit-frame-pointer Example code follows: [code] #include <stdlib.h> #include <stdio.h> #include <fcntl.h> #include <unistd.h> #include <string.h> typedef struct { int a; int b; int c; int d; } A; A test2(int n) { A ret; printf("Hello n=%d\n",n); ret.a=1; ret.b=2; ret.c=3; return(ret); } #define GTT_SIZE 8 typedef struct {long feld[GTT_SIZE];} GTT; long (*adr)(GTT); void dummy() { long ret; GTT gtt; A t; int i; printf("adr t: %p\n",&t); for(i=0;i<GTT_SIZE;i++) gtt.feld[i]=i; gtt.feld[0]=(long)&t; gtt.feld[1]=5; adr=(long (*)(GTT))test2; ret=adr(gtt); printf("Function returned: 0x%x\n",(unsigned int)ret); if(ret==(long)&t) printf("This is adress of t\n"); printf("Function ret: %d %d %d\n",t.a,t.b,t.c); printf("original stack: \n"); for(i=0;i<GTT_SIZE;i++) { printf("%d : $%x\n",i,(unsigned int)gtt.feld[i]); } // >>>>>>>The ERROR is triggered here !!!!! } int main() { printf("This is main.\n"); dummy(); // <<<<<<< this will never be reached. printf("End of main.\n"); } [/code] Output: (normal case [code] This is main. adr t: 0xbfd76d7c Hallo n=5 Function returned: 0xbfd76d7c This is adress of t Function ret: 1 2 3 original stack: 0 : $bfd76d7c 1 : $5 2 : $2 3 : $3 4 : $4 5 : $5 6 : $6 7 : $7 End of main. [/code] Output if comiled with "gcc -fomit-frame-pointer a.c" [code] This is main. adr t: 0xbfa1f30c Hallo n=5 Function returned: 0xbfa1f30c This is adress of t Function ret: 2 3 0 original stack: 0 : $5 1 : $2 2 : $3 3 : $4 4 : $5 5 : $6 6 : $7 7 : $9fc2c100 *** stack smashing detected ***: ./a.out terminated Abort [/code] The latter should not happen and is considered as a bug in gcc.
This example wants to show, that the optimizer destroys compatibility to the ABI, either with optimizing something in the calling procedure (likely) or in the called function. The same error happens, when test2 is in a .so library and dynamically linked to the main program at runtime. Since the main program does not know the function prototype at compile time it has to rely on calling it in an ABI-compatible way.
So what happening here is you are not calling the function in ABI compatible way at all. So this code is way undefined.
Well, OK, so I have to switch off omit-frame-pointer... Unless I find another more compatible solution how to call functions not knowing at compile time, if they return a struct or not.
Why not look into something like libffi?
Hm, wow, thank you. I did not know it. Maybe rather depend on one more library than have undefined and probably non-portable code....