Bug 80710 - Stack smashing detected in correct code depending on optimization flag
Summary: Stack smashing detected in correct code depending on optimization flag
Status: RESOLVED INVALID
Alias: None
Product: gcc
Classification: Unclassified
Component: middle-end (show other bugs)
Version: 5.4.0
: P3 normal
Target Milestone: ---
Assignee: Not yet assigned to anyone
URL:
Keywords: wrong-code
Depends on:
Blocks:
 
Reported: 2017-05-11 12:42 UTC by dr.markus.hoffmann@gmx.de
Modified: 2017-05-12 12:37 UTC (History)
0 users

See Also:
Host:
Target: i386
Build:
Known to work:
Known to fail:
Last reconfirmed:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description dr.markus.hoffmann@gmx.de 2017-05-11 12:42:21 UTC
Stack smashing detected if the code is compiled with -O1 or with -fomit-frame-pointer. Everything fine, when compiled without optimization or with -fno-omit-frame-pointer

Example code follows:
[code]

#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>

typedef struct {
  int a;
  int b;
  int c;
  int d;
} A;


A test2(int n) {
  A ret;
  printf("Hello n=%d\n",n);
  ret.a=1;
  ret.b=2;
  ret.c=3;
  return(ret);
}

#define GTT_SIZE 8

typedef struct  {long feld[GTT_SIZE];} GTT;
long (*adr)(GTT);


void dummy() {
  long ret;
  GTT gtt;
  A t;
  int i;
  
  printf("adr t: %p\n",&t);
  for(i=0;i<GTT_SIZE;i++) gtt.feld[i]=i;
  gtt.feld[0]=(long)&t;
  gtt.feld[1]=5;
  
  adr=(long (*)(GTT))test2;
  ret=adr(gtt);
  printf("Function returned: 0x%x\n",(unsigned int)ret);
  if(ret==(long)&t) printf("This is adress of t\n");
  printf("Function ret: %d %d %d\n",t.a,t.b,t.c);
  
  printf("original stack: \n");
  for(i=0;i<GTT_SIZE;i++) {
    printf("%d : $%x\n",i,(unsigned int)gtt.feld[i]);
  }
// >>>>>>>The ERROR is triggered here !!!!!
}

int main() {
  printf("This is main.\n");
  dummy();
// <<<<<<< this will never be reached.
  printf("End of main.\n");
}

[/code]

Output: (normal case 
[code]
This is main.
adr t: 0xbfd76d7c
Hallo n=5
Function returned: 0xbfd76d7c
This is adress of t
Function ret: 1 2 3
original stack: 
0 : $bfd76d7c
1 : $5
2 : $2
3 : $3
4 : $4
5 : $5
6 : $6
7 : $7
End of main.

[/code]

Output if comiled with "gcc -fomit-frame-pointer a.c"
[code]
This is main.
adr t: 0xbfa1f30c
Hallo n=5
Function returned: 0xbfa1f30c
This is adress of t
Function ret: 2 3 0
original stack: 
0 : $5
1 : $2
2 : $3
3 : $4
4 : $5
5 : $6
6 : $7
7 : $9fc2c100
*** stack smashing detected ***: ./a.out terminated
Abort
[/code]

The latter should not happen and is considered as a bug in gcc.
Comment 1 dr.markus.hoffmann@gmx.de 2017-05-11 14:50:59 UTC
This example wants to show, that the optimizer destroys compatibility to the ABI, either with optimizing something in the calling procedure (likely) or in the called function. The same error happens, when test2 is in a .so library and dynamically linked to the main program at runtime. Since the main program does not know the function prototype at compile time it has to rely on calling it in an ABI-compatible way.
Comment 2 Andrew Pinski 2017-05-11 14:59:03 UTC
So what happening here is you are not calling the function in ABI compatible way at all.  So this code is way undefined.
Comment 3 dr.markus.hoffmann@gmx.de 2017-05-12 07:33:56 UTC
Well, OK, so I have to switch off omit-frame-pointer... Unless I find another more compatible solution how to call functions not knowing at compile time, if they return a struct or not.
Comment 4 Andrew Pinski 2017-05-12 10:18:03 UTC
Why not look into something like libffi?
Comment 5 dr.markus.hoffmann@gmx.de 2017-05-12 12:37:56 UTC
Hm, wow, thank you. I did not know it. Maybe rather depend on one more library than have undefined and probably non-portable code....