How do I compare an integer and a floating-point value the right way™?
The builtin comparsion operators give incorrect results in some edge cases, for examp
To compare a FP f
and integer i
for equality:
(Code is representative and uses comparison of float
and long long
as an example)
If f
is a NaN, infinity, or has a fractional part (perhaps use frexp()
), f
is not equal to i
.
float ipart;
// C++
if (frexp(f, &ipart) != 0) return not_equal;
// C
if (frexpf(f, &ipart) != 0) return not_equal;
Convert the numeric limits of i
into exactly representable FP values (powers of 2) near those limits.** Easy to do if we assume FP is not a rare base 10 encoding and range of double
exceeds the range on the i
. Take advantage that integer limits magnitudes are or near Mersenne Number. (Sorry example code is C-ish)
#define FP_INT_MAX_PLUS1 ((LLONG_MAX/2 + 1)*2.0)
#define FP_INT_MIN (LLONG_MIN*1.0)
Compare f
to is limits
if (f >= FP_INT_MAX_PLUS1) return not_equal;
if (f < FP_INT_MIN) return not_equal;
Convert f
to integer and compare
return (long long) f == i;
To compare a FP f
and integer i
for <
, >
, ==
or not comparable:
(Using above limits)
Test f >= lower limit
if (f >= FP_INT_MIN) {
Test f <= upper limit
// reform below to cope with effects of rounding
// if (f <= FP_INT_MAX_PLUS1 - 1)
if (f - FP_INT_MAX_PLUS1 <= -1.0) {
Convert f
to integer/fraction and compare
// at this point `f` is in the range of `i`
long long ipart = (long long) f;
if (ipart < i) return f_less_than_i;
if (ipart > i) return f_more_than_i;
float frac = f - ipart;
if (frac < 0) return f_less_than_i;
if (frac > 0) return f_more_than_i;
return equal;
}
Handle edge cases
else return f_more_than_i;
}
if (f < 0.0) return f_less_than_i;
return not_comparable;
Simplifications possible, yet I wanted to convey the algorithm.
** Additional conditional code needed to cope with non 2's complement integer encoding. It is quite similar to the MAX
code.