Taken from Floating point accuracy problems.

Floating-point numbers cannot precisely represent all real numbers, and that floating-point operations cannot precisely represent true arithmetic operations, leads to many surprising situations. This is related to the finite precision with which computers generally represent numbers.

For example, the non-representability of 0.1 and 0.01 (in binary) means that the result of attempting to square 0.1 is neither 0.01 nor the representable number closest to it. In 24-bit (single precision) representation, 0.1 (decimal) was given previously as:

e = −4

s = 110011001100110011001101, which is 0.100000001490116119384765625 exactly.Squaring this number gives 0.010000000298023226097399174250313080847263336181640625 exactly.

Squaring it with single-precision floating-point hardware (with rounding) gives 0.010000000707805156707763671875 exactly. But the representable number closest to 0.01 is 0.009999999776482582092285156250 exactly.

Categories: Annoying/Interesting