Skip to content
  • Mikael Pettersson's avatar
    math-emu: correct test for downshifting fraction in _FP_FROM_INT() · f8324e20
    Mikael Pettersson authored
    The kernel's math-emu code contains a macro _FP_FROM_INT() which is
    used to convert an integer to a raw normalized floating-point value.
    It does this basically in three steps:
    
    1. Compute the exponent from the number of leading zero bits.
    2. Downshift large fractions to put the MSB in the right position
       for normalized fractions.
    3. Upshift small fractions to put the MSB in the right position.
    
    There is an boundary error in step 2, causing a fraction with its
    MSB exactly one bit above the normalized MSB position to not be
    downshifted.  This results in a non-normalized raw float, which when
    packed becomes a massively inaccurate representation for that input.
    
    The impact of this depends on a number of arch-specific factors,
    but it is known to have broken emulation of FXTOD instructions
    on UltraSPARC III, which was originally reported as GCC bug 44631
    <http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44631>.
    
    Any arch which uses math-emu to emul...
    f8324e20