Re: Strange problem
Everett Williams wrote:
>Okay, just for fun, let us assume that it works the way you say. There is
>one caveat. The reason why 10 times .1 does not give us back 1 is that
>the float that is created from 4.1 is slightly less than 4.1, and therefore the
>equation gives us back a number that is slightly less than 1...then by the
>manual, it is rounded down to 0. The rounding may not be dependable at
>4.2 or 4.3 or ... The error is in the conversion from base 10 decimal to
>binary floating point. I have not confirmed this, but until I actually tested
>it,
>I would not trust your adding of 1 to the result. Sometimes the conversion
>might give a number just slightly larger than the decimal and then the
>result will be 1 too large. I would love for someone to tell me that it always
>converts to the largest float smaller than the original number. I did not find
>that assurance in the manual. RDS...time to tell us how you do it.
>
Sometimes I am a little slow, but I do finally get there. In a case where the
system truncates rather than rounds, the normal solution is to add .5
except in this case where we are not sure that the conversion is always
less than the original decimal number. Given the answer to that question is
yes, then adding .5 solves the problem. Only a lot of very exacting testing or
an answer from RDS will solve the question for sure.
Everett L.(Rett) Williams
rett at gvtc.com
|
Not Categorized, Please Help
|
|
