Why is rounding reset when summing?

Developers Ruby API
1

Tell me pls why this code does not work correctly:

a=6.187595819331583.to_mm
a=a.round(1)
p a # result - 157.2
b=12.122847489661602.to_mm
b=b.round(1)
p b # result - 307.9
p a+b # result 465.09999999999997

:nerd_face: a+b=157.2+307.9=465.1
Why is rounding reset when summing?

2

A Floating Point number is an approximation of the value it is trying to represent. The IEEE-754 format which we use to store these numbers is an attempt to represent all numbers between negetive infinity and positive infinity in 64 bits of memory. As a result, for almost all numbers there is a very small error in the value that is stored.

See:
https://en.wikipedia.org/wiki/Double-precision_floating-point_format

For a hands on example try these links:"
https://www.h-schmidt.net/FloatConverter/IEEE754.html
https://www.binaryconvert.com/convert_double.html

This code prints the actual value of your variables a and b

a = 6.187595819331583.to_mm
a = a.round(1)
puts "The value of a is:"
puts a
puts "The actual value of a is:"
puts "%.36f" % a
puts 

b = 12.122847489661602.to_mm
b = b.round(1)
puts "The value of b is:"
puts b 
puts "The actual value of b is:"
puts "%.36f" % b
puts 

puts "the sum is:"
puts a + b
nil
2 Likes
3

Because the resultant value is a new object from the +() method, which does not know that the round() method was called on the two operands (receiver and argument.) And of course, it sees the slight error as explained by sWilliams above.

So, you will need to call round() upon the result as well:

puts (a+b).round(1)
#=> 465.1