High memory usage compiling large ‘xxd -i’ output

[Xi Ruoyao]

On 2020-04-11 16:31 -0500, relliott--- via Gcc-help wrote:
> Hello,
> 
> Thanks for your reply.  I guess I don’t understand why this requires so much
> memory all at once.  Is it not possible to write to the object file as the
> initializer is parsed?

A compiler doesn't write to object file.  It only writes to assembly file. 
Modern compilers are designed multi-layer so the parser (frontend) should not
output assembly (it's the job of the backend).  And, doing that so will miss
optimizations.  A simple code:

const int b[] = {0, 1, 2};

int foo(void)
{
  int i, x = 0;
  for (i = 0; i < 3; i++)
    x += b[i];
  return x;
}

The body of "foo" will be optimized to "return 3;" at -O2.  If we wrote the
array content to assembly file and discarded the value in memory, this
optimization will be impossible.

"xxd" a binary file and then compile the result with a compiler is just like
"converting a .jpg to .png then back to .jpg".  A smarter way:

  ld -r -b binary some_big_binary_blob.bin -o some_big_binary_blob.o

There will be symbols _binary_some_big_binary_blob_bin_start,
_binary_some_big_binary_blob_end in some_big_binary_blob.o.  Then it's possible
to access the content of the binary blob with:

  extern const char _binary_some_big_binary_blob_bin_start;
  extern const char _binary_some_big_binary_blob_bin_end;

  int main(void)
  {
    const char *begin = &_binary_some_big_binary_blob_bin_start;
    const char *end = &_binary_some_big_binary_blob_bin_end;
    const char *p;
    for (p = begin; p != end; p++)
      play_with(*p);
  }
-- 
Xi Ruoyao <xry111@mengyan1223.wang>
School of Aerospace Science and Technology, Xidian University