1. rounding problem
- Posted by George Walters <gwalters at sc.rr.com> Oct 23, 2006
- 529 views
- Last edited Oct 24, 2006
Using EU 2.4 and Win32 IF I trace and display the value of temp (a variable) the display shows temp = .7149999738 sprintf("%10.4f, temp) displays .7150 which is not correct if its value is .7149999738 On the data base (mysql) the value displayed is .7150. I need to solve this riddle since I'm having invoice totals incorrect. So how does sprintf know that .7150 is the value but yet .7149999738 is displayed on the trace?? I"m confused.
2. Re: rounding problem
- Posted by George Walters <gwalters at sc.rr.com> Oct 23, 2006
- 527 views
- Last edited Oct 24, 2006
Nevermind, I'm stupid
3. Re: rounding problem
- Posted by Ricardo M. Forno <rmforno at tutopia.com> Oct 23, 2006
- 499 views
- Last edited Oct 24, 2006
George Walters wrote: > > > Using EU 2.4 and Win32 > > IF I trace and display the value of temp (a variable) the display shows > temp = .7149999738 > > sprintf("%10.4f, temp) displays .7150 which is not correct if its value is > .7149999738 > > On the data base (mysql) the value displayed is .7150. > > I need to solve this riddle since I'm having invoice totals incorrect. So > how does sprintf know that .7150 is the value but yet .7149999738 is > displayed on the trace?? I"m confused. Actually, 0.7150 is correct for 0.7149999738 since you specified it to be 4 digits (i.e., rounded to 4 digits). If you are using floating numbers for invoicing, you'll found nearly impossible to get exact results. This is a property of floating numbers, the hardware, and the binary and decimal arithmetics; Euphoria isn't guilty. Solution: if you want results with 4 decimals, for example, use integers that are 10,000 times greater than the original numbers. Round or truncate them by means of the float() function [truncate: z = floor(x * 10000); round: floor(x * 10000 + 0.5)]. When printing them, divide them into 10,000, only for the task of showing them, not to further compute results. In other words: if you want exact results using cents, then operate with cents. Regards.
4. Re: rounding problem
- Posted by Chris Burch <chriscrylex at aol.com> Oct 23, 2006
- 501 views
- Last edited Oct 24, 2006
Hi Quick fix * 10000 add 1 floor /10000 Note, you'll never get more accurate than that, unless you use specialised libraries, looks like mysql is doing that for you. Chris http://euallegro.wikispaces.com http://members.aol.com/chriscrylex/euphoria.htm http://uboard.proboards32.com/ http://members.aol.com/chriscrylex/EUSQLite/eusql.html
5. Re: rounding problem
- Posted by Ricardo M. Forno <rmforno at tutopia.com> Oct 23, 2006
- 536 views
- Last edited Oct 24, 2006
Ricardo M. Forno wrote: > > George Walters wrote: > > > > > > Using EU 2.4 and Win32 > > > > IF I trace and display the value of temp (a variable) the display shows > > temp = .7149999738 > > > > sprintf("%10.4f, temp) displays .7150 which is not correct if its value is > > .7149999738 > > > > On the data base (mysql) the value displayed is .7150. > > > > I need to solve this riddle since I'm having invoice totals incorrect. So > > how does sprintf know that .7150 is the value but yet .7149999738 is > > displayed on the trace?? I"m confused. > > Actually, 0.7150 is correct for 0.7149999738 since you specified it to be > 4 digits (i.e., rounded to 4 digits). > If you are using floating numbers for invoicing, you'll found nearly > impossible to get exact results. > This is a property of floating numbers, the hardware, and the binary > and decimal arithmetics; Euphoria isn't guilty. > Solution: if you want results with 4 decimals, for example, use integers > that are 10,000 times greater than the original numbers. Round or truncate > them by means of the float() function [truncate: z = floor(x * 10000); round: > floor(x * 10000 + 0.5)]. When printing them, divide them into 10,000, > only for the task of showing them, not to further compute results. > In other words: if you want exact results using cents, then operate > with cents. > Regards. Oops! I meant the floor() function, not float().
6. Re: rounding problem
- Posted by Chris Bensler <bensler at nt.net> Oct 24, 2006
- 514 views
Ricardo M. Forno wrote: > > Ricardo M. Forno wrote: > > > > George Walters wrote: > > > > > > > > > Using EU 2.4 and Win32 > > > > > > IF I trace and display the value of temp (a variable) the display shows > > > temp = .7149999738 > > > > > > sprintf("%10.4f, temp) displays .7150 which is not correct if its value > > > is > > > .7149999738 > > > > > > On the data base (mysql) the value displayed is .7150. > > > > > > I need to solve this riddle since I'm having invoice totals incorrect. So > > > how does sprintf know that .7150 is the value but yet .7149999738 is > > > displayed on the trace?? I"m confused. > > > > Actually, 0.7150 is correct for 0.7149999738 since you specified it to be > > 4 digits (i.e., rounded to 4 digits). > > If you are using floating numbers for invoicing, you'll found nearly > > impossible to get exact results. > > This is a property of floating numbers, the hardware, and the binary > > and decimal arithmetics; Euphoria isn't guilty. > > Solution: if you want results with 4 decimals, for example, use integers > > that are 10,000 times greater than the original numbers. Round or truncate > > them by means of the float() function [truncate: z = floor(x * 10000); > > round: > > floor(x * 10000 + 0.5)]. When printing them, divide them into 10,000, > > only for the task of showing them, not to further compute results. > > In other words: if you want exact results using cents, then operate > > with cents. > > Regards. > > Oops! I meant the floor() function, not float(). This is called fixed point math. Chris Bensler ~ The difference between ordinary and extraordinary is that little extra ~ http://empire.iwireweb.com - Empire for Euphoria
7. Re: rounding problem
- Posted by Alex Chamberlain <alex.chamberlain at tiscali.co.uk> Oct 24, 2006
- 522 views
Am I right in thinking that working in pence saves memory as well? Thanks, Alex
8. Re: rounding problem
- Posted by George Walters <gwalters at sc.rr.com> Oct 24, 2006
- 506 views
Ricardo M. Forno wrote: > them by means of the float() function [truncate: z = floor(x * 10000); round: > floor(x * 10000 + 0.5)]. When printing them, divide them into 10,000, This is exactly what my round functin does, but produces a different result than sprintf. I would like to know how sprintf does it's rounding. Anyone know?
global function round(atom a, integer b) atom f if b < 0 then return a -- invalid argument, do nothing elsif b = 0 then return floor(a+.5) else f = power(10,b) return floor(a * f +.5 / f) end if end function
9. Re: rounding problem
- Posted by Juergen Luethje <j.lue at gmx.de> Oct 24, 2006
- 513 views
These are the rounding functions that I use:
global type nonnegative_int (object x) if integer(x) then return x >= 0 end if return 0 -- FALSE end type global function round_half_even (object x, nonnegative_int digits) -- rounding according to the IEEE standard -- in: x : (sequence of) number(s) to round -- digits: desired number of decimal places atom p, ret if atom(x) then p = power(10, digits) ret = floor(x*p + 0.5) if remainder(ret, 2) then ret -= 1 end if return ret/p end if for i = 1 to length(x) do x[i] = round_half_even(x[i], digits) end for return x end function global function round_half_up (object x, nonnegative_int digits) -- "commercial rounding" -- in: x : (sequence of) number(s) to round -- digits: desired number of decimal places atom p if atom(x) then p = power(10, digits) if x >= 0 then return floor( x*p + 0.5)/p else return -floor(-x*p + 0.5)/p end if end if for i = 1 to length(x) do x[i] = round_half_up(x[i], digits) end for return x end function -- Demo ? round_half_even( 2.65, 1) ? round_half_up ( 2.65, 1) ? round_half_even( 2.75, 1) ? round_half_up ( 2.75, 1) puts(1, "\n") ? round_half_even(-2.65, 1) ? round_half_up (-2.65, 1) ? round_half_even(-2.75, 1) ? round_half_up (-2.75, 1)
Regards, Juergen
10. Re: rounding problem
- Posted by Ricardo M. Forno <rmforno at tutopia.com> Oct 24, 2006
- 544 views
George Walters wrote: > > Ricardo M. Forno wrote: > > them by means of the float() function [truncate: z = floor(x * 10000); > > round: > > floor(x * 10000 + 0.5)]. When printing them, divide them into 10,000, > > This is exactly what my round functin does, but produces a different result > than > sprintf. I would like to know how sprintf does it's rounding. Anyone know? > > }}} <eucode> > global function round(atom a, integer b) > atom f > > if b < 0 then > return a -- invalid argument, do nothing > elsif b = 0 then > return floor(a+.5) > else > f = power(10,b) > return floor(a * f +.5 / f) > end if > > end function > </eucode> {{{ Please check the sligth change I made to your round() routine. Division by 10,000 should be made *after* rounding. It is named round1(). I also included my RoundCents routine from my General Functions package. Regards.
include genfunc.e -- General Functions global function round(atom a, integer b) atom f if b < 0 then return a -- invalid argument, do nothing elsif b = 0 then return floor(a+.5) else f = power(10,b) return floor(a * f +.5 / f) end if end function global function round1(atom a, integer b) atom f if b < 0 then return a -- invalid argument, do nothing elsif b = 0 then return floor(a+.5) else f = power(10,b) return floor(a * f +.5) / f end if end function atom x for i = 1 to 10 do x = rand(100) / 6.56745321 printf(1, "%10.10f %10.10f %10.4f %10.4f %10.4f\n", {x, round(x, 4), round(x, 4), round1(x, 4), RoundCents(x, 10000)}) end for
11. Re: rounding problem
- Posted by George Walters <gwalters at sc.rr.com> Oct 24, 2006
- 513 views
- Last edited Oct 25, 2006
Hmm, thanks I'll play with that some...
12. Re: rounding problem
- Posted by George Walters <gwalters at sc.rr.com> Oct 24, 2006
- 510 views
- Last edited Oct 25, 2006
Thanks Juergen, I'll spin them around some.
13. Re: rounding problem
- Posted by George Walters <gwalters at sc.rr.com> Oct 25, 2006
- 527 views
I have discovered, I believe, the root of my rounding problem. It lies in the fact that I am using MySql to store the data. I had specified the numeric values like "double(12,4)" or "double(12,2)" whereas MySql will do rounding on what you store. GEEZ!! If I stored 1.250 I would get back from Mysql 1.249999 plus some less significant digits at the end. Changing my data base specification to "double" with no size specification now gives me back exactly what I have stored. See this reference if you're interested. http://dev.mysql.com/doc/refman/5.0/en/numeric-types.html Thanks for all the help. I did have an error in my round function which was pointed out by Ricardo and Juergen. I'm trudging on with more testing now. > > This is exactly what my round functin does, but produces a different result > than > sprintf. I would like to know how sprintf does it's rounding. Anyone know? > > }}} <eucode> > global function round(atom a, integer b) > atom f > > if b < 0 then > return a -- invalid argument, do nothing > elsif b = 0 then > return floor(a+.5) > else > f = power(10,b) > return floor(a * f +.5) / f > end if > > end function > </eucode> {{{