1. atom_to_float32 precision
Is there any way to control the precision of the output of atom_to_float32? I
want to take a number that might have the value:
123.450000
And ensure it comes out as:
123.45
Jonas Temple
http://www.innovativesys.net
2. Re: atom_to_float32 precision
Jonas Temple wrote:
> Is there any way to control the precision of the output of atom_to_float32?
> I want to take a number that might have the value:
>
> 123.450000
>
> And ensure it comes out as:
>
> 123.45
I'm not sure exactly what you are trying to achieve,
but it sounds like you are encountering an age-old problem
with floating-point numbers that has been discussed here
many times before. A number like 123.45 can't be
represented *exactly* in binary floating-point form,
no matter how many bits you use. There will always be a
tiny error. This problem can drive you crazy when you are trying
to get money calculations involving pennies to work out
exactly. If you choose 2 decimal places for display (%.2f),
printf() or sprintf() should round it nicely for you, for display purposes.
However it will never be *precisely* 123.45 that is stored
in memory. More likely 123.44999999999999999... or
123.4500000000...000001 or something along those lines.
See:
http://www.openeuphoria.org/cgi-bin/esearch.exu?fromMonth=6&fromYear=1&toMonth=A&toYear=C&postedBy=&keywords=round+decimal+binary+floating-point
Regards,
Rob Craig
Rapid Deployment Software
http://www.RapidEuphoria.com
3. Re: atom_to_float32 precision
Jonas Temple wrote:
>
> Is there any way to control the precision of the output of atom_to_float32?
> I want to take a number that might have the value:
>
> 123.450000
>
> And ensure it comes out as:
>
> 123.45
>
> Jonas Temple
> <a href="http://www.innovativesys.net">http://www.innovativesys.net</a>
function round_with granularity(atom a,atom granularity)
-- rounds a to nearest multiple of granularity
-- also check out w32round() in w32utils, distributed with win32lib 0.70.x
atom rem
if granularity<0.0 then
granularity = -granularity
elsif granularity = 0.0 then
puts(2,"Rounding to a multiple of 0!")
return a
end if
rem=remainder(a,granularity)
if rem+rem>=granularity then
a += granularity
end if
return a-rem
end function
Also, if you need fixed decimal arithmetic, converting everything to integers is
the way, as Rob and a few others have underlined.
Also, modifiying this slightly will give you other rounding mode, according to
your exact needs.
CChris
