1. Out of memory replies
- Posted by flint at acay.com.au
Jun 07, 2002
Firstly many thanks to all of the responses to my problem.
The 4-bytes per data element (atom?) helped me greatly. Does that value
apply regardless of data type?
hmmmm........ The curious thing for me also is that the sequence was
generated within euphoria, being built on-the-fly, then saved to disk.
Admittedly it did take around 6 hours, which really wasn't what I was
looking for - should've taken around 20-30 mins to run the other parts of
code, which means 5-5 1/2 hours spent amending the sequence.
Would hard-dimensioning the sequence prior to loading with data speed it up
? I remember basic having hissy fits when arrays were dimensioned part way
thru the code, having to reshuffle data around each time - could this be a
similar thing happening ?
Anyway enough theorision, I'll work on a different file format & see how
that goes.
Cheers and thanks for the help,
Michael
2. Re: Out of memory replies
Michael wrote:
[...]
> The 4-bytes per data element (atom?) helped me greatly. Does that value
> apply regardless of data type?
No. Integers are always 4 bytes, but atoms are 8 bytes wide. The size of an
object or sequence depends on what's in it. 
> hmmmm........ The curious thing for me also is that the sequence was
> generated within euphoria, being built on-the-fly, then saved to disk.
> Admittedly it did take around 6 hours...
Like you hinted later in your message, you're better off not using print()
and get() to transfer data structures to and from files if you have lots of
information. If you make a few assumptions about the data structure, you
don't need to write all the braces and commas to a file.
e.g. Imagine we have the following data in a sequence:
sequence s
s = {{123456,234567,3456789},{432165,57,670321},{741741,8001,90324}}
If we used print(file, s) to save this to a file, the file's going to be 64
bytes long. But we can assume several things; 1) the data structure is 3x3,
2) Each data element is an integer and (less obviously perhaps) will fit in
3 bytes. This means that with a little clever coding, we could save our data
in a file of 27 bytes. Now I realise this doesn't _seem_ like much of a
saving in this example, but our non-print() using method is actually only
using 42% of the original space on disk. Not a bad saving.
Handily, much of the code we need already comes with euphoria...
include machine.e -- for int_to_bytes() and bytes_to_int()
include get.e -- for get_bytes()
sequence s
sequence bytes
integer fh -- file handle
s = {{123456,234567,3456789},{432165,57,670321},{741741,8001,90324}}
? s
-- write s to a file
fh = open("save_s.dat","wb")
if fh = -1 then puts(2, "File error\n") abort(1) end if
for x = 1 to length(s) do
for y = 1 to length(s[x]) do
bytes = int_to_bytes(s[x][y])
bytes = bytes[1..3] -- discard most sig. byte (always 0)
puts(fh, bytes)
end for
end for
close(fh)
s = repeat(repeat(0,3),3) -- reset s to a blank 3x3 array
? s
-- read s back from a file
fh = open("save_s.dat","rb")
if fh = -1 then puts(2, "File error\n") abort(1) end if
for x = 1 to length(s) do
for y = 1 to length(s[x]) do
bytes = repeat(0,4) -- an integer is 4 bytes
bytes[1..3] = get_bytes(fh, 3)
s[x][y] = bytes_to_int(bytes)
end for
end for
close(fh)
? s
HTH,
Carl
PS It's also possible to save _atoms_ less than 1e+38 to disk as 4 bytes,
albeit with a loss of some precision. Look into atom_to_float32() in
machine.e.
--
[ Carl R White -=- aka -=- Cyrek the Illogical ]
[ () E-mail...: cyrek{}cyreksoft.yorks.com ]
[ /\ URL......: http://www.cyreksoft.yorks.com ]
