OpenEuphoria

Re: File size limit

• Posted by Fernando Bauer <fmbauer at ho?mail.c?m> Jun 12, 2008
• 790 views

Jason Gade wrote:
> 
[snipped]
> 
> > I have asked for larger file numbers too, but i don't think we will get them
> > in Eu v4. Even if we did, we could not do basic full precision integer math
> > on the numbers over 4gigabytes (there are such libs in the archives tho).
> > 
> > Kat
> 
> Jim Brown is working on this right now. And atoms are unique integers up to
> (2^52) - 1 so I don't know why you can't do full precision integer math on
> them.

The maximum contiguous integer in Eu is:
   2^53 = power(2,53) = 9,007,199,254,740,992

Demonstrations:

1) By decimal representation:

2^53-5= 9007199254740987  -- ok
2^53-4= 9007199254740988  -- ok
2^53-3= 9007199254740989  -- ok
2^53-2= 9007199254740990  -- ok
2^53-1= 9007199254740991  -- ok
2^53+0= 9007199254740992  -- ok
2^53+1= 9007199254740992  -- Problem: it's equal to 2^53
2^53+2= 9007199254740994
2^53+3= 9007199254740996
2^53+4= 9007199254740996
2^53+5= 9007199254740996

2) IEEE754 double-precision (64 bits) bit values:
Value =|s|--exponent-|--------------------fraction------------------------|
2^53-5= 0 10000110011 1111111111111111111111111111111111111111111111111011
2^53-4= 0 10000110011 1111111111111111111111111111111111111111111111111100
2^53-3= 0 10000110011 1111111111111111111111111111111111111111111111111101
2^53-2= 0 10000110011 1111111111111111111111111111111111111111111111111110
2^53-1= 0 10000110011 1111111111111111111111111111111111111111111111111111
2^53+0= 0 10000110100 0000000000000000000000000000000000000000000000000000
2^53+1= 0 10000110100 0000000000000000000000000000000000000000000000000000
2^53+2= 0 10000110100 0000000000000000000000000000000000000000000000000001
2^53+3= 0 10000110100 0000000000000000000000000000000000000000000000000010
2^53+4= 0 10000110100 0000000000000000000000000000000000000000000000000010
2^53+5= 0 10000110100 0000000000000000000000000000000000000000000000000010

3)IEEE754 double-precision normalized numbers:
  v = (-1)^s * 2^e * (1 + fraction * 2^-52)
    = (-1)^s * (2^e + fraction * 2^(e-52))

Note that there is NO problem when v=2^52 (e=52):
  v = (-1)^s * (2^52 + fraction)   where 0<=fraction<=2^52-1

Note that there are problems when 2^53<v<2^54 (e=53):
  v = (-1)^s * (2^53 + fraction * 2)   where 0<=fraction<=2^52-1
since fraction is an integer number, it's impossible to represent 2^53+1
exactly.

2^53-5= +2^52 * (1 + 4503599627370491 * 2^-52)
2^53-4= +2^52 * (1 + 4503599627370492 * 2^-52)
2^53-3= +2^52 * (1 + 4503599627370493 * 2^-52)
2^53-2= +2^52 * (1 + 4503599627370494 * 2^-52)
2^53-1= +2^52 * (1 + 4503599627370495 * 2^-52)
2^53+0= +2^53 * (1 +                0 * 2^-52)
2^53+1= +2^53 * (1 +                0 * 2^-52)
2^53+2= +2^53 * (1 +                1 * 2^-52)
2^53+3= +2^53 * (1 +                2 * 2^-52)
2^53+4= +2^53 * (1 +                2 * 2^-52)
2^53+5= +2^53 * (1 +                2 * 2^-52)

4)By measuring integer distances:
Look at this:
http://www.openeuphoria.org/cgi-bin/esearch.exu?fromMonth=1&fromYear=D&toMonth=6&toYear=D&postedBy=Fernando&keywords=NextInteger

> 
> }}}
<eucode>
> include limits.e -- in the archive
> 
> procedure print_large(atom in)
> 	printf(1, "%17.0f\n", in)
> end procedure
> 
> atom large
> 
> large = power(2, 52) - 1
> ? get_epsilon(large)
> print_large(large - 1)
> print_large(large)
> print_large(large + 1)
> print_large(large + 2)
> 
> large = power(2, 52)
> ? get_epsilon(large)
> print_large(large - 1)
> print_large(large)
> print_large(large + 1)
> print_large(large + 2)
> 
> large = power(2, 52) + 1
> ? get_epsilon(large)
> print_large(large - 1)
> print_large(large)
> print_large(large + 1)
> print_large(large + 2)
> </eucode>
{{{

> 
> Output:
> C:\EU311\projects>exwc test_atom.ex
> 1
>  4503599627370494
>  4503599627370495
>  4503599627370496
>  4503599627370497
> 2
>  4503599627370495
>  4503599627370496
>  4503599627370497
>  4503599627370498
> 2
>  4503599627370496
>  4503599627370497
>  4503599627370498
>  4503599627370499
> 

Look at this line in function get_epsilon():

if exp > 1023 then

I think it should be:

if exp > -1023 then

With that modification, the output is:
0.5
 4503599627370494
 4503599627370495
 4503599627370496
 4503599627370497
1
 4503599627370495
 4503599627370496
 4503599627370497
 4503599627370498
1
 4503599627370496
 4503599627370497
 4503599627370498
 4503599627370499

which shows that (2^52) and (2^52+1) are still contiguous.

[snipped]

Regards,
  Fernando
 
> Note, even though numbers over 2^52 seem to work, I wouldn't count on them
> being
> accurate.
> 
> That should be sufficient for most files, right?
> 
> Or maybe it's just multiplication/division that would be broken?
> 
> --
> A complex system that works is invariably found to have evolved from a simple
> system that works.
> --John Gall's 15th law of Systemantics.
> 
> "Premature optimization is the root of all evil in programming."
> --C.A.R. Hoare
> 
> j.