[Bug tree-optimization/64567] New: missed optimization: redundant test before clearing bit(s)

[rv at rasmusvillemoes dot dk]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64567

            Bug ID: 64567
           Summary: missed optimization: redundant test before clearing
                    bit(s)
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: enhancement
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: rv at rasmusvillemoes dot dk

(Not sure the "Component" is correct). gcc fails to optimize

  if (foo & FLAG)
    foo &= ~FLAG;

into simply unconditionally doing foo &= ~FLAG. Currently, the above code
sequence generates four instructions (mov, and, test, cmov) whereas a single
and would be sufficient. Note that this is a valid transformation regardless of
how many bits are set in FLAG; it doesn't even have to be a compile-time
constant (but must of course be side-effect free).

gcc also doesn't optimize the dual

  if (!(foo & FLAG))
    foo |= FLAG;

into foo |= FLAG;. That transformation is however only valid when FLAG is known
to consist of only a single bit (so would probably require either a
compile-time constant where this can be checked or an expression of the form "1
<< something").

Trivial test case below. I'd expect foo and foo2 to compile to the same code,
and similarly for baz and baz2.

$ cat test.c
#define F1 0x04
#define F2 0x08
int bar(unsigned flags);
int foo(unsigned flags)
{
        if (flags & (F1 | F2))
                flags &= ~(F1 | F2);
        return bar(flags);
}

int foo2(unsigned flags)
{
        flags &= ~(F1 | F2);
        return bar(flags);
}

int baz(unsigned flags)
{
        if (!(flags & F1))
                flags |= F1;
        return bar(flags);
}

int baz2(unsigned flags)
{
        flags |= F1;
        return bar(flags);
}

$ ./gcc-5.0 -Wall -Wextra -O3 -c test.c
$ objdump -d test.o

test.o:     file format elf64-x86-64


Disassembly of section .text:

0000000000000000 <foo>:
   0:   89 f8                   mov    %edi,%eax
   2:   83 e0 f3                and    $0xfffffff3,%eax
   5:   40 f6 c7 0c             test   $0xc,%dil
   9:   0f 45 f8                cmovne %eax,%edi
   c:   e9 00 00 00 00          jmpq   11 <foo+0x11>
  11:   66 66 66 66 66 66 2e    data32 data32 data32 data32 data32 nopw
%cs:0x0(%rax,%rax,1)
  18:   0f 1f 84 00 00 00 00 
  1f:   00 

0000000000000020 <foo2>:
  20:   83 e7 f3                and    $0xfffffff3,%edi
  23:   e9 00 00 00 00          jmpq   28 <foo2+0x8>
  28:   0f 1f 84 00 00 00 00    nopl   0x0(%rax,%rax,1)
  2f:   00 

0000000000000030 <baz>:
  30:   89 f8                   mov    %edi,%eax
  32:   83 c8 04                or     $0x4,%eax
  35:   40 f6 c7 04             test   $0x4,%dil
  39:   0f 44 f8                cmove  %eax,%edi
  3c:   e9 00 00 00 00          jmpq   41 <baz+0x11>
  41:   66 66 66 66 66 66 2e    data32 data32 data32 data32 data32 nopw
%cs:0x0(%rax,%rax,1)
  48:   0f 1f 84 00 00 00 00 
  4f:   00 

0000000000000050 <baz2>:
  50:   83 cf 04                or     $0x4,%edi
  53:   e9 00 00 00 00          jmpq   58 <baz2+0x8>