OpenEuphoria

Re: Out of memory replies

• Posted by "Carl W." <euphoria at cyreksoft.yorks.com> Jun 10, 2002
• 392 views

Michael wrote:

[...]
> The 4-bytes per data element (atom?) helped me greatly. Does that value
> apply regardless of data type?

No. Integers are always 4 bytes, but atoms are 8 bytes wide. The size of an
object or sequence depends on what's in it. 

> hmmmm........ The curious thing for me also is that the sequence was
> generated within euphoria, being built on-the-fly, then saved to disk.
> Admittedly it did take around 6 hours...

Like you hinted later in your message, you're better off not using print()
and get() to transfer data structures to and from files if you have lots of
information. If you make a few assumptions about the data structure, you
don't need to write all the braces and commas to a file.

e.g. Imagine we have the following data in a sequence:
sequence s
s = {{123456,234567,3456789},{432165,57,670321},{741741,8001,90324}}

If we used print(file, s) to save this to a file, the file's going to be 64
bytes long. But we can assume several things; 1) the data structure is 3x3,
2) Each data element is an integer and (less obviously perhaps) will fit in
3 bytes. This means that with a little clever coding, we could save our data
in a file of 27 bytes. Now I realise this doesn't _seem_ like much of a
saving in this example, but our non-print() using method is actually only
using 42% of the original space on disk. Not a bad saving.

Handily, much of the code we need already comes with euphoria...

include machine.e -- for int_to_bytes() and bytes_to_int()
include get.e     -- for get_bytes()
sequence s
sequence bytes
integer fh -- file handle
s = {{123456,234567,3456789},{432165,57,670321},{741741,8001,90324}}
? s
-- write s to a file
fh = open("save_s.dat","wb")
if fh = -1 then puts(2, "File error\n") abort(1) end if
for x = 1 to length(s) do
    for y = 1 to length(s[x]) do
        bytes = int_to_bytes(s[x][y])
        bytes = bytes[1..3] -- discard most sig. byte (always 0)
        puts(fh, bytes)
    end for
end for
close(fh)

s = repeat(repeat(0,3),3) -- reset s to a blank 3x3 array
? s
-- read s back from a file
fh = open("save_s.dat","rb")
if fh = -1 then puts(2, "File error\n") abort(1) end if
for x = 1 to length(s) do
    for y = 1 to length(s[x]) do
        bytes = repeat(0,4) -- an integer is 4 bytes
        bytes[1..3] = get_bytes(fh, 3)
        s[x][y] = bytes_to_int(bytes)
    end for
end for
close(fh)
? s

HTH,
Carl

PS It's also possible to save _atoms_ less than 1e+38 to disk as 4 bytes,
albeit with a loss of some precision. Look into atom_to_float32() in
machine.e.

--
[ Carl R White -=- aka -=- Cyrek the Illogical ]
[ () E-mail...:     cyrek{}cyreksoft.yorks.com ]
[ /\ URL......: http://www.cyreksoft.yorks.com ]