OpenEuphoria

1. File size limit

• Posted by Archarios <archarios at v?riz?n.net> Jun 11, 2008
• 810 views

Taken from latest Euphoria311.chm - "Currently, files up to 2 Gb in size can be
handled. Beyond that, some file operations may not work correctly. This limit
will likely be increased in the future."
I have read in the forums the max file size is 4Gb and most talk about using
seek().
What is the max file size limit for simply read/writing a file, no seek() or
where() etc. (I'm only including win32lib for sleep)?
Also I'm trying to create a file compare function, any tips?

3. Re: File size limit

• Posted by Matt Lewis <matthewwalkerlewis at gmail.?o?> Jun 11, 2008
• 781 views

Kat wrote:
> 
> 
> You can't seek over 2gbytes, but you can getc() and gets() to any length. So
> i made a procedure to getc() the number of bytes i wanted to seek(), and then
> there i was.
> 
> I have asked for larger file numbers too, but i don't think we will get them
> in Eu v4. Even if we did, we could not do basic full precision integer math
> on the numbers over 4gigabytes (there are such libs in the archives tho).

Larger file numbers are probably pretty easy (I think it's just increasing an
array size and updating a macro).  How many do you want?  I don't recall this
specifically (at least not specifically).

You can do full precision integer math using atoms to much larger sizes
than 2^32.  The limit on precision is 2^53, or:

   9,007,199,254,740,990

That's over 9 quadrillion, or about 9 terabytes (in disk manufacturer style
units).  Granted, this isn't using native machine integers, but probably 
faster than the above quoted libraries.

If you need numbers bigger than that for file manipulation, I'd suggest that
you need a better algorithm.

Matt

5. Re: File size limit

• Posted by Jason Gade <jaygade at ya?oo.com> Jun 11, 2008
• 741 views

Kat wrote:
> 
> Archarios wrote:
> > 
> > Taken from latest Euphoria311.chm - "Currently, files up to 2 Gb in size can
> > be handled. Beyond that, some file operations may not work correctly. This
> > limit
> > will likely be increased in the future."
> > I have read in the forums the max file size is 4Gb and most talk about using
> > seek().
> > What is the max file size limit for simply read/writing a file, no seek() or
> > where() etc. (I'm only including win32lib for sleep)?
> > Also I'm trying to create a file compare function, any tips?
> 
> You can't seek over 2gbytes, but you can getc() and gets() to any length.

Really? I still don't have any >2G files to test but I could swear that you
could seek up to 4G in Euphoria now under Windows and Linux?


> I have asked for larger file numbers too, but i don't think we will get them
> in Eu v4. Even if we did, we could not do basic full precision integer math
> on the numbers over 4gigabytes (there are such libs in the archives tho).
> 
> Kat

Jim Brown is working on this right now. And atoms are unique integers up to
(2^52) - 1 so I don't know why you can't do full precision integer math on them.

include limits.e -- in the archive

procedure print_large(atom in)
	printf(1, "%17.0f\n", in)
end procedure

atom large

large = power(2, 52) - 1
? get_epsilon(large)
print_large(large - 1)
print_large(large)
print_large(large + 1)
print_large(large + 2)

large = power(2, 52)
? get_epsilon(large)
print_large(large - 1)
print_large(large)
print_large(large + 1)
print_large(large + 2)

large = power(2, 52) + 1
? get_epsilon(large)
print_large(large - 1)
print_large(large)
print_large(large + 1)
print_large(large + 2)

Output:
C:\EU311\projects>exwc test_atom.ex
1
 4503599627370494
 4503599627370495
 4503599627370496
 4503599627370497
2
 4503599627370495
 4503599627370496
 4503599627370497
 4503599627370498
2
 4503599627370496
 4503599627370497
 4503599627370498
 4503599627370499

Note, even though numbers over 2^52 seem to work, I wouldn't count on them being
accurate.

That should be sufficient for most files, right?

Or maybe it's just multiplication/division that would be broken?

--
A complex system that works is invariably found to have evolved from a simple
system that works.
--John Gall's 15th law of Systemantics.

"Premature optimization is the root of all evil in programming."
--C.A.R. Hoare

j.

7. Re: File size limit

• Posted by Matt Lewis <matthewwalkerlewis at gma?l?com> Jun 11, 2008
• 734 views

Kat wrote:
> 
> But i thought Eu's switch from integer to floating point atoms caused a
> drop in the number of decimal digits at the same count as the integer
> type would go screwy. No? Seems you are saying no. Yes?

No, it's much higher than that.  Euphoria atoms are either 31-bit signed
integers or double floating point numbers, so there's a lot more room
in an atom for full precision.  If you start using fractions, then some
of that precision will be used there, and you won't be able to go as high.

> So all we need is the wrapping of the 64bit file calls in winxp/vista/nix api?

Yes.  And I think (according to the code I've seen, but I haven't actually
tested) it's all in there right now.

Matt

9. Re: File size limit

• Posted by Kat <KAT12 at co?s?hs.net> Jun 12, 2008
• 772 views

Lucius L. Hilley III wrote:
> 
> Jason Gade wrote:
> > 
> <SNIP>
> 
> > Note, even though numbers over 2^52 seem to work, I wouldn't count on them
> > being
> > accurate.
> > 
> > That should be sufficient for most files, right?
> > 
> > Or maybe it's just multiplication/division that would be broken?
> 
> Read about the normalization and the hidden bit.
> Then you will understand why 2^53 works.

For ATA drives, only 2^48 (6 x 8bits) is allocated for LBA addressing, and is a
horribly huge amount of harddrive for one computer, let alone one drive. If Eu
goes bigtime tho, a commercial data warehouse may have that much on their lan.
But 2^40 is a terabyte, a huge value for one drive, more than i'd want in one
drive, i think. Some day, we may have a dos that can virtualize harddrive
addresses for some reason, and may want that amount of addresses, even tho
there's  not that much actual drive platter. So i personally do not, at this
moment in time, need 2^52 bits worth of integer.

Kat

11. Re: File size limit

• Posted by Jason Gade <jaygade at ?a?oo.com> Jun 12, 2008
• 785 views

Fernando Bauer wrote:
> 
> Jason Gade wrote:
> > 
> [snipped]
> > 
> > > I have asked for larger file numbers too, but i don't think we will get
> > > them
> > > in Eu v4. Even if we did, we could not do basic full precision integer
> > > math
> > > on the numbers over 4gigabytes (there are such libs in the archives tho).
> > > 
> > > Kat
> > 
> > Jim Brown is working on this right now. And atoms are unique integers up to
> > (2^52) - 1 so I don't know why you can't do full precision integer math on
> > them.
> 
> The maximum contiguous integer in Eu is:
>    2^53 = power(2,53) = 9,007,199,254,740,992
> 
> Demonstrations:
> 
> 1) By decimal representation:
> 
> 2^53-5= 9007199254740987  -- ok
> 2^53-4= 9007199254740988  -- ok
> 2^53-3= 9007199254740989  -- ok
> 2^53-2= 9007199254740990  -- ok
> 2^53-1= 9007199254740991  -- ok
> 2^53+0= 9007199254740992  -- ok
> 2^53+1= 9007199254740992  -- Problem: it's equal to 2^53
> 2^53+2= 9007199254740994
> 2^53+3= 9007199254740996
> 2^53+4= 9007199254740996
> 2^53+5= 9007199254740996
> 
> 2) IEEE754 double-precision (64 bits) bit values:
> Value =|s|--exponent-|--------------------fraction------------------------|
> 2^53-5= 0 10000110011 1111111111111111111111111111111111111111111111111011
> 2^53-4= 0 10000110011 1111111111111111111111111111111111111111111111111100
> 2^53-3= 0 10000110011 1111111111111111111111111111111111111111111111111101
> 2^53-2= 0 10000110011 1111111111111111111111111111111111111111111111111110
> 2^53-1= 0 10000110011 1111111111111111111111111111111111111111111111111111
> 2^53+0= 0 10000110100 0000000000000000000000000000000000000000000000000000
> 2^53+1= 0 10000110100 0000000000000000000000000000000000000000000000000000
> 2^53+2= 0 10000110100 0000000000000000000000000000000000000000000000000001
> 2^53+3= 0 10000110100 0000000000000000000000000000000000000000000000000010
> 2^53+4= 0 10000110100 0000000000000000000000000000000000000000000000000010
> 2^53+5= 0 10000110100 0000000000000000000000000000000000000000000000000010
> 
> 3)IEEE754 double-precision normalized numbers:
>   v = (-1)^s * 2^e * (1 + fraction * 2^-52)
>     = (-1)^s * (2^e + fraction * 2^(e-52))
> 
> Note that there is NO problem when v=2^52 (e=52):
>   v = (-1)^s * (2^52 + fraction)   where 0<=fraction<=2^52-1
> 
> Note that there are problems when 2^53<v<2^54 (e=53):
>   v = (-1)^s * (2^53 + fraction * 2)   where 0<=fraction<=2^52-1
> since fraction is an integer number, it's impossible to represent 2^53+1
> exactly.
> 
> 2^53-5= +2^52 * (1 + 4503599627370491 * 2^-52)
> 2^53-4= +2^52 * (1 + 4503599627370492 * 2^-52)
> 2^53-3= +2^52 * (1 + 4503599627370493 * 2^-52)
> 2^53-2= +2^52 * (1 + 4503599627370494 * 2^-52)
> 2^53-1= +2^52 * (1 + 4503599627370495 * 2^-52)
> 2^53+0= +2^53 * (1 +                0 * 2^-52)
> 2^53+1= +2^53 * (1 +                0 * 2^-52)
> 2^53+2= +2^53 * (1 +                1 * 2^-52)
> 2^53+3= +2^53 * (1 +                2 * 2^-52)
> 2^53+4= +2^53 * (1 +                2 * 2^-52)
> 2^53+5= +2^53 * (1 +                2 * 2^-52)
> 
> 4)By measuring integer distances:
> Look at this:
> <a
> href="http://www.openeuphoria.org/cgi-bin/esearch.exu?fromMonth=1&fromYear=D&toMonth=6&toYear=D&postedBy=Fernando&keywords=NextInteger">http://www.openeuphoria.org/cgi-bin/esearch.exu?fromMonth=1&fromYear=D&toMonth=6&toYear=D&postedBy=Fernando&keywords=NextInteger</a>

Hey, thanks Fernando and Lucius. Especially for spelling it out in so much
detail, Fernando. Oh, and I do remember that previous post but I was kind of
skeptical until you proved it just now. Very nice.

I'll update limits.e

--
A complex system that works is invariably found to have evolved from a simple
system that works.
--John Gall's 15th law of Systemantics.

"Premature optimization is the root of all evil in programming."
--C.A.R. Hoare

j.

13. Re: File size limit

• Posted by Kat <KAT12 at coosahs.n??> Jun 12, 2008
• 735 views

Archarios wrote:
> 
> Kat wrote:
> > 
> > Archarios wrote:
> > > 
> > > Taken from latest Euphoria311.chm - "Currently, files up to 2 Gb in size
> > > can
> > > be handled. Beyond that, some file operations may not work correctly. This
> > > limit
> > > will likely be increased in the future."
> > > I have read in the forums the max file size is 4Gb and most talk about
> > > using
> > > seek().
> > > What is the max file size limit for simply read/writing a file, no seek()
> > > or
> > > where() etc. (I'm only including win32lib for sleep)?
> > > Also I'm trying to create a file compare function, any tips?
> > 
> > You can't seek over 2gbytes, but you can getc() and gets() to any length. So
> > i made a procedure to getc() the number of bytes i wanted to seek(), and
> > then
> > there i was.
> > 
> > I have asked for larger file numbers too, but i don't think we will get them
> > in Eu v4. Even if we did, we could not do basic full precision integer math
> > on the numbers over 4gigabytes (there are such libs in the archives tho).
> > 
> > Kat
> What about opening a file for append?
> Thanks for the help!

You need math for append?
When i do
writefile = open(filename,"a")
i do not use any math, so i hope precision isn't a problem here?

Kat

15. Re: File size limit

• Posted by Jason Gade <jaygade at yahoo.?om> Jun 13, 2008
• 758 views

Archarios wrote:
> 
> Note: I am fairly new to this myself so dont quote me on this.(quotes here are
> partial)!!!
> I notice that there is a bit of confusion (including me of course) about max
> file size and I remembered something, and this is what I found.
> "NTFS: File size limited only by size of volume.
> FAT: Maximum file size is 2 GB.
> FAT32: Maximum file size is 4 GB."
>  From: 
>  "ms-its:C:\WINDOWS\Help\misc.chm::/choosing_between_NTFS_FAT_and_FAT32.htm"
> Each file system can only support files of certain sizes.
> usually DOS(now DR-DOS) uses FAT(common) or FAT32.
> Reading from euphoria311.chm(which may be different from docs included with
> euphoria) it states that there is a file size limit of 2Gbs, beyond this there
> may be corruption to the file.

Euphoria 3.11 /should/ be able to access whatever size file the OS supports, the
main problem is being able to seek() into a certain point in the file, I believe.
Windows and Linux versions of Euphoria should have no problem seeking up to 4G
(the max size of an unsigned 32-bit integer).

On Euphoria 4.0, this 4G seek limit will go away on operating systems that
support it.

> Also, there are restrictions on variables and
> constants, such as Integers are only capable of (2^53) - 1(from my
> understanding
> from the forums)

Euphoria has two types - atoms and sequences. A third type, integer, is a
special case of atom and is built-in to the interpreter to take advantage of the
fact that most computers handle 32-bit integers as well as or better than 64-bit
double (floating point) numbers.

For reasons internal to the interpreter, though, a variable declared as integer
has a range of -1,073,741,824 to 1,073,741,823. That is, +/- 1 billion right?

A variable declared as an atom has an integer range of +/- 2^53 -- as long as it
doesn't have any digits to the right of the decimal point. Now, this is bigger
than most hard drives available today, approximately 9 Petabytes -- about 75,000
times the size of the HD in my own personal computer.

> "approximately -1e300 (minus one times 10 to the power 300)
> to +1e300 with 15 decimal digits of accuracy" from euphoria311.chm,

Yes, this is the standard range of a double-precision floating point numbers and
applies to most computer languages.

Actual range: +/- 1.7976931348623157 x 10^308 -- that's some pretty big numbers!

> which means that the storing of a byte position within a file beyond the above
> limits may
> cause problems with the running of a program or even data corrupion.
> Note: I am fairly new to this myself so dont quote me on this.(quotes here are
> partial)!!!
> 
> I was only curious lol

If you send an atom variable to seek instead of an integer variable you should
be able to reliably seek up to 4G without corruption. However, I have not tested
this, I have only looked at the source code which converts the variable to a true
32-bit unsigned integer.

--
A complex system that works is invariably found to have evolved from a simple
system that works.
--John Gall's 15th law of Systemantics.

"Premature optimization is the root of all evil in programming."
--C.A.R. Hoare

j.