The patch introduces

scalar_int_mode int_mode;
if (REG_P (x) && is_int_mode (mode, &int_mode)
    && REG_VALUES (REGNO (x)) != NULL
    && (!cselib_current_insn || !DEBUG_INSN_P (cselib_current_insn)))
  {
    rtx copy = shallow_copy_rtx (x);
    scalar_int_mode narrow_mode_iter;
    FOR_EACH_MODE_UNTIL (narrow_mode_iter, int_mode)   // <--------
      {
        PUT_MODE_RAW (copy, narrow_mode_iter);
        cselib_val *v = cselib_lookup (copy, narrow_mode_iter, 0, VOIDmode);
        if (v)
          {
            rtx sub = lowpart_subreg (narrow_mode_iter, e->val_rtx, int_mode);
            if (sub)
              new_elt_loc_list (v, sub);
          }
      }
  }

The failing assert is at the for-loop which is supposed to iterate only over integer modes up to int_mode. I'm not familiar with ia64; is there any machine which I could use for debugging?

The failing assert is

gcc_checking_assert (m_mode != E_VOIDmode);

which is triggered by get_known_wider. Would be interesting to see the initial value of int_mode and if/how FOR_EACH_MODE_UNTIL actually ends up with E_VOIDmode.

I think you have to go with debugging a cross cc1

#3  0x0000000000cbc4a3 in new_cselib_val (hash=6, mode=E_BImode, 
    x=0x7ffff7170a98) at /home/rguenther/src/gcc2/gcc/cselib.cc:1579
1579	      FOR_EACH_MODE_UNTIL (narrow_mode_iter, int_mode)
(gdb) p int_mode
$2 = BImode


but class_narrowest_mode is QImode, so we iterate until OImode which
doesn't have a wider mode (so VOIDmode) but never reach BImode.

Somehow BImode is MODE_INT but declared BOOL_MODE which means we skip it here.
Not sure if that's an error in genmodes - I suspect other places
wouldn't expect to iterate over BImode in FOR_EACH_MODE_UNTIL.

      /* Skip BImode.  FIXME: BImode probably shouldn't be MODE_INT.  */
      if (m->boolean)
        continue;

Thanks for looking into this! Currently I'm out of office and have very limited internet access. I will be back on Tuesday and look right into this. If this is to late feel free to revert my patch. Sorry for the inconvenience!

I did a quick test using

diff --git a/gcc/cselib.cc b/gcc/cselib.cc
index 9b582e5d3d6..2fd0190bc79 100644
--- a/gcc/cselib.cc
+++ b/gcc/cselib.cc
@@ -1571,6 +1571,7 @@ new_cselib_val (unsigned int hash, machine_mode mode, rtx x)
 
   scalar_int_mode int_mode;
   if (REG_P (x) && is_int_mode (mode, &int_mode)
+      && int_mode != BImode
       && REG_VALUES (REGNO (x)) != NULL
       && (!cselib_current_insn || !DEBUG_INSN_P (cselib_current_insn)))
     {
     
which solved the cross ia64 build for me. Maybe there are further integer modes which I didn't consider, i.e., I will have a thorough look at it next week.

I get the same failure on amdgcn building newlib/libm/math/kf_rem_pio2.c

(In reply to Stefan Schulze Frielinghaus from comment #6)
> I did a quick test using
> 
> diff --git a/gcc/cselib.cc b/gcc/cselib.cc
> index 9b582e5d3d6..2fd0190bc79 100644
> --- a/gcc/cselib.cc
> +++ b/gcc/cselib.cc
> @@ -1571,6 +1571,7 @@ new_cselib_val (unsigned int hash, machine_mode mode,
> rtx x)
>  
>    scalar_int_mode int_mode;
>    if (REG_P (x) && is_int_mode (mode, &int_mode)
> +      && int_mode != BImode
>        && REG_VALUES (REGNO (x)) != NULL
>        && (!cselib_current_insn || !DEBUG_INSN_P (cselib_current_insn)))
>      {
>      
> which solved the cross ia64 build for me. Maybe there are further integer
> modes which I didn't consider, i.e., I will have a thorough look at it next
> week.
Looks good, but maybe:

  GET_MODE_SIZE (int_mode) > 1

would be more general.

BImode being MODE_INT is a horrible wart (it should be MODE_PARTIAL_INT
instead).  But unfortunately ia64 is the target that would be most
affected by "fixing" this, and it's not an easy target to test.
So I think in practice nothing will change while ia64 is the in tree.

I can confirm that the patch fixes the amdgcn build.

(In reply to rsandifo@gcc.gnu.org from comment #8)
> Looks good, but maybe:
> 
>   GET_MODE_SIZE (int_mode) > 1
> 
> would be more general.

I very much like the idea of a size guard. Posted a patch:

https://gcc.gnu.org/pipermail/gcc-patches/2022-October/602776.html

The master branch has been updated by Stefan Schulze Frielinghaus <stefansf@gcc.gnu.org>:

https://gcc.gnu.org/g:5fc4d3e1837ea4850aac6460f563913f1d3fc5b8

commit r13-3105-g5fc4d3e1837ea4850aac6460f563913f1d3fc5b8
Author: Stefan Schulze Frielinghaus <stefansf@linux.ibm.com>
Date:   Thu Oct 6 08:43:53 2022 +0200

    cselib: Skip BImode while keeping track of subvalue relations [PR107088]
    
    For BImode get_narrowest_mode evaluates to QImode but BImode < QImode.
    Thus FOR_EACH_MODE_UNTIL never reaches BImode and iterates until OImode
    for which no wider mode exists so we end up with VOIDmode and fail.
    Fixed by adding a size guard so we effectively skip BImode.
    
    gcc/ChangeLog:
    
            PR rtl-optimization/107088
            * cselib.cc (new_cselib_val): Skip BImode while keeping track of
            subvalue relations.

Fixed.