Bug 104816 - -fcf-protection=branch should generate endbr instead of notrack jumps
Summary: -fcf-protection=branch should generate endbr instead of notrack jumps
Status: WAITING
Alias: None
Product: gcc
Classification: Unclassified
Component: target (show other bugs)
Version: 12.0
: P3 enhancement
Target Milestone: ---
Assignee: Not yet assigned to anyone
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-03-07 11:48 UTC by Joao Moreira
Modified: 2022-05-24 18:12 UTC (History)
6 users (show)

See Also:
Host:
Target: x86_64-linux-gnu
Build:
Known to work:
Known to fail:
Last reconfirmed: 2022-03-07 00:00:00


Attachments
A patch (1.68 KB, patch)
2022-03-11 20:58 UTC, H.J. Lu
Details | Diff
The v2 patch (1.64 KB, patch)
2022-03-13 15:09 UTC, H.J. Lu
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Joao Moreira 2022-03-07 11:48:44 UTC
When -fcf-protection=branch is used, the compiler will generate jump tables where the indirect jump is prefixed with the NOTRACK prefix, so it can jump to non-ENDBR targets. Yet, for NOTRACK prefixes to work, the NOTRACK specific enable bit must be set, what renders the binary broken on any environment where this is not the case. In fact, having NOTRACK disabled was a design choice for the Linux kernel CET support [https://lkml.org/lkml/2022/3/7/1068].

With the above, the compiler should generate jump tables with ENDBRs, for proper correctness. And, if security regarding the additional ENDBRs is a concern, the code can be explicitly compiled with -fno-jump-tables.
Comment 1 Joao Moreira 2022-03-07 12:15:32 UTC
quick reproducer, just in case: https://godbolt.org/z/EaG3rhrnj
Comment 2 Richard Biener 2022-03-07 13:53:20 UTC
Documentation should probably also amended to reflect this limitation (or -fcf-protection=branch should implicitely disable jump-tables).
Comment 3 H.J. Lu 2022-03-07 14:06:43 UTC
(In reply to Richard Biener from comment #2)
> Documentation should probably also amended to reflect this limitation (or
> -fcf-protection=branch should implicitely disable jump-tables).

We should document this limitation and update -fno-jump-tables documentation:

'-fno-jump-tables'
     Do not use jump tables for switch statements even where it would be
     more efficient than other code generation strategies.  This option
     is of use in conjunction with '-fpic' or '-fPIC' for building code
     that forms part of a dynamic linker and cannot reference the
     address of a jump table.  On some targets, jump tables do not
     require a GOT and this option is not needed.
Comment 4 Andrew Cooper 2022-03-07 14:18:33 UTC
I've worked around this in Xen with: https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=9d4a44380d273de22d5753883cbf5581795ff24d and 
https://lore.kernel.org/lkml/YiXpv0q88paPHPqF@hirez.programming.kicks-ass.net/ is pending for Linux.

IMO, it's an error that -fcf-protection=branch is not obeyed for jump tables, and we don't want to end up in a situation where jump tables are unusable with CET.
Comment 5 H.J. Lu 2022-03-07 14:23:00 UTC
(In reply to Andrew Cooper from comment #4)
> I've worked around this in Xen with:
> https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;
> h=9d4a44380d273de22d5753883cbf5581795ff24d and 
> https://lore.kernel.org/lkml/YiXpv0q88paPHPqF@hirez.programming.kicks-ass.
> net/ is pending for Linux.
> 
> IMO, it's an error that -fcf-protection=branch is not obeyed for jump
> tables, and we don't want to end up in a situation where jump tables are
> unusable with CET.

Are you suggesting to add an option to generate jump table with ENDBR?
Comment 6 peterz 2022-03-07 14:27:41 UTC
(In reply to H.J. Lu from comment #5)
> (In reply to Andrew Cooper from comment #4)
> > I've worked around this in Xen with:
> > https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;
> > h=9d4a44380d273de22d5753883cbf5581795ff24d and 
> > https://lore.kernel.org/lkml/YiXpv0q88paPHPqF@hirez.programming.kicks-ass.
> > net/ is pending for Linux.
> > 
> > IMO, it's an error that -fcf-protection=branch is not obeyed for jump
> > tables, and we don't want to end up in a situation where jump tables are
> > unusable with CET.
> 
> Are you suggesting to add an option to generate jump table with ENDBR?

I would suggest having -fcf-protection=branch generate ENDBR for jump-tables and never generate NOTRACK prefix. Then add a mode that allows NOTRACK prefixes, perhaps -fcf-protection=branch,notrack.

IBT without NOTRACK is the strongest form; it would be daft to require additional parameters for that.
Comment 7 Andrew Cooper 2022-03-07 14:38:11 UTC
(In reply to H.J. Lu from comment #5)
> Are you suggesting to add an option to generate jump table with ENDBR?

Jump tables are a legitimate optimisation.  NOTRACK is a weakness in CET protections, and fully hardened userspace (as well as kernels) will want to run with MSR_{U,S}_CET.NOTRACK_EN=0.

There should be some future where jump tables can be used in combination with NOTRACK_EN=0.
Comment 8 H.J. Lu 2022-03-11 20:43:26 UTC
(In reply to Joao Moreira from comment #0)
> When -fcf-protection=branch is used, the compiler will generate jump tables
> where the indirect jump is prefixed with the NOTRACK prefix, so it can jump
> to non-ENDBR targets. Yet, for NOTRACK prefixes to work, the NOTRACK
> specific enable bit must be set, what renders the binary broken on any
> environment where this is not the case. In fact, having NOTRACK disabled was
> a design choice for the Linux kernel CET support
> [https://lkml.org/lkml/2022/3/7/1068].
> 
> With the above, the compiler should generate jump tables with ENDBRs, for
> proper correctness. And, if security regarding the additional ENDBRs is a
> concern, the code can be explicitly compiled with -fno-jump-tables.

There is an undocumented option: -mcet-switch.  It does exactly what you
are looking for.  Currently it is off by default.  We can document it
and turn it on by default.
Comment 9 H.J. Lu 2022-03-11 20:58:55 UTC
Created attachment 52615 [details]
A patch
Comment 10 H.J. Lu 2022-03-13 15:09:42 UTC
Created attachment 52618 [details]
The v2 patch
Comment 11 CVS Commits 2022-05-24 16:06:08 UTC
The master branch has been updated by H.J. Lu <hjl@gcc.gnu.org>:

https://gcc.gnu.org/g:2f4f7de787e5844515d27b2269fc472f95a9916a

commit r13-744-g2f4f7de787e5844515d27b2269fc472f95a9916a
Author: H.J. Lu <hjl.tools@gmail.com>
Date:   Fri Mar 11 12:51:34 2022 -0800

    x86: Document -mcet-switch
    
    When -fcf-protection=branch is used, the compiler will generate jump
    tables for switch statements where the indirect jump is prefixed with
    the NOTRACK prefix, so it can jump to non-ENDBR targets.  Since the
    indirect jump targets are generated by the compiler and stored in
    read-only memory, this does not result in a direct loss of hardening.
    But if the jump table index is attacker-controlled, the indirect jump
    may not be constrained by CET.
    
    Document -mcet-switch to generate jump tables for switch statements with
    ENDBR and skip the NOTRACK prefix for indirect jump.  This option should
    be used when the NOTRACK prefix is disabled.
    
            PR target/104816
            * config/i386/i386.opt: Remove Undocumented.
            * doc/invoke.texi: Document -mcet-switch.
Comment 12 peterz 2022-05-24 18:12:19 UTC
On Tue, May 24, 2022 at 04:06:08PM +0000, cvs-commit at gcc dot gnu.org wrote:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104816
> 
> --- Comment #11 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
> The master branch has been updated by H.J. Lu <hjl@gcc.gnu.org>:
> 
> https://gcc.gnu.org/g:2f4f7de787e5844515d27b2269fc472f95a9916a
> 
> commit r13-744-g2f4f7de787e5844515d27b2269fc472f95a9916a
> Author: H.J. Lu <hjl.tools@gmail.com>
> Date:   Fri Mar 11 12:51:34 2022 -0800
> 
>     x86: Document -mcet-switch
> 
>     When -fcf-protection=branch is used, the compiler will generate jump
>     tables for switch statements where the indirect jump is prefixed with
>     the NOTRACK prefix, so it can jump to non-ENDBR targets.  Since the
>     indirect jump targets are generated by the compiler and stored in
>     read-only memory, this does not result in a direct loss of hardening.
>     But if the jump table index is attacker-controlled, the indirect jump
>     may not be constrained by CET.

Notrack indirect jumps are fully susceptible to speculation attacks.