1. rounding issue
- Posted by gwalters Aug 20, 2010
- 977 views
I am having a rounding issue when comparing atoms. Here's the problem and I would like to know how to solve it.
sequence invoice, payments
I read from the database which is double type and add this to the invoice table.
invoice &= dbRec[invoice]
The value stored in invoice is not double eventhough it is double coming from the database.
e.g. the payment is 41204.33 but the invoice is 41104.32812 the components stored in the sequence appear to not be double. In the database it is correctly 41104.33 as reported by mysql.
If it makes any difference I'm still using EU 2.4 and Matt's odbc.
So do sequences store numbers as atoms with about 14-16 digits significance?
Comments appreciated
2. Re: rounding issue
- Posted by mattlewis (admin) Aug 20, 2010
- 984 views
I am having a rounding issue when comparing atoms. Here's the problem and I would like to know how to solve it.
[snip]
So do sequences store numbers as atoms with about 14-16 digits significance?
The short answer is not to store money in a floating point format, because you'll always have rounding issues, unless you're only computing binary fractions. This applies to any programming language.
You should store it as integral values of the smallest denomination you need (probably cents). Then do the conversion when required for display purposes.
Matt
3. Re: rounding issue
- Posted by gwalters Aug 20, 2010
- 995 views
I understand what you're saying Matt. My real question is what does EU put in the sequence when the incoming item is double precision? It appears to be single precision in the sequence above. Do you know the answer to that? For example
sequence temp atom dbl
temp = {} dbl = .11 temp &= dbl
What is in temp, single precision item or double precision?
4. Re: rounding issue
- Posted by mattlewis (admin) Aug 20, 2010
- 1012 views
I understand what you're saying Matt. My real question is what does EU put in the sequence when the incoming item is double precision? It appears to be single precision in the sequence above. Do you know the answer to that? For example
sequence temp atom dbl
temp = {} dbl = .11 temp &= dbl
What is in temp, single precision item or double precision?
Internally, euphoria only stores numbers as integers or doubles.
Matt
