[Bug libitm/66453] New: In a deadlock libitm allocates all available RAM until oom is called

[gallir at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66453

            Bug ID: 66453
           Summary: In a deadlock libitm allocates all available RAM until
                    oom is called
           Product: gcc
           Version: 4.9.2
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libitm
          Assignee: unassigned at gcc dot gnu.org
          Reporter: gallir at gmail dot com
  Target Milestone: ---

The following transaction block make libitm to allocate all available memory in
few seconds:

int counter[ARRAY_SIZE];

void *count(void *ptr) {
    long i, max = MAX_COUNT/NUM_THREADS;
    int tid = ((struct tdata *) ptr)->tid;

    __transaction_atomic {
        while (!counter[tid]);
        counter[(tid + 1) % NUM_THREADS] = 1;
    }
    return;

}

Full code with 2 threads: https://gist.github.com/gallir/5df222892b1f633c8275
(tested and reproduced in a Intel i5)


The example is awfully buggy, generates an obvious -binary barrier- deadlock,
but libitm could limit the amount of allocations to prevent that this type of
mistakes brings the server down (the number of allocations is a good hint of a
deadlock, isn't it?)