1. RE: Out of memory
- Posted by irv at take.maxleft.com Jun 06, 2002
- 458 views
flint at acay.com.au wrote: > Hi everyone, > > I'm having trouble loading a sequence from file. The sequence consists > of > 32657 elements, each element having up to 100 sub-elements as well. > Total > file size is 72 meg. When attempting to load with get(fn), it runs for > about 1/2 hour and stops with error message: > > E:\Euphoria\include\get.e:265 in function Get() > Your program has run out of memory. > One moment please ... > > I've tried breaking up the sequence and saving to files separately, then > > loading back in separately. > I can then load each sequence on it's own, but when loading all in & > appending to re-create the one sequence I get the same error. > > I'm using a PII 400 w/226 meg ram - 180 meg free before running the > code. > Is there some limit to total number of elements allowed in a sequence? > Or should I be configuring my system differently ? > > Any help muchly appreciated, as I've been stuck on this for a few days > now. Euphoria uses 32 bits to store each atom or each element of a sequence. That means you need 288 megs of memory or virtual memory to load that file. Even if you had enough memory, it is going to take a long time to load. It's very likely that loading the entire file is not the best approach. What exactly are you trying to do with that file? Irv
2. RE: Out of memory
- Posted by Jacques Guy <jguy at alphalink.com.au> Jun 06, 2002
- 407 views
I had a similar problem recently, not memory (my data file was much smaller) but speed of loading. Using get() the file took 26 seconds to load. I thought this grossly wrong. So I rewrote those parts of my program, without get(), using "for" and "while" loops writing to disk and reading from disk and _never_ using recursion. It worked: the same file loaded in 0.33 seconds. No, not a typing mistake: from 26 seconds to one third of a second. I was not surprised. I had had a similar experience several years earlier, memory that time. I was having a team of programmers implement a database of icons (they used C++). It all went along well when, one day, they sent me a new implementation, which barfed after loading just 37 icons -- and I had 64M of RAM, just about the maximum you could ever hope for in those days. So they got a very puzzled "please explain". After apparently much head scratching the team leader explained "we have been using heavily recursive structures and procedures". That was it, of course. They rewrote it using iteration instead of recursion and I ran out of icons before I ran out of memory again. I haven't had a look at the source code, but I guess that get() is -- almost necessarily -- implemented recursively. It's fine for loading flat or shallow data, but, as experience shows, dreadfully inefficient for deeply embedded sequences, as you obviously have.
3. RE: Out of memory
- Posted by Derek Parnell <Derek.Parnell at SYD.RABOBANK.COM> Jun 06, 2002
- 413 views
This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01C20DB7.972B78A0 charset=iso-8859-1 > -----Original Message----- > From: Jacques Guy [mailto:jguy at alphalink.com.au] > Sent: Friday, 7 June 2002 10:00 > To: EUforum > Subject: RE: Out of memory > > > > I had a similar problem recently, not memory (my data file > was much smaller) but speed of loading. Using get() the > file took 26 seconds to load. I thought this grossly > wrong. So I rewrote those parts of my program, without > get(), using "for" and "while" loops writing to disk > and reading from disk and _never_ using recursion. It > worked: the same file loaded in 0.33 seconds. No, not > a typing mistake: from 26 seconds to one third of a > second. > > I was not surprised. I had had a similar experience > several years earlier, memory that time. I was having > a team of programmers implement a database of icons > (they used C++). It all went along well when, one day, > they sent me a new implementation, which barfed after > loading just 37 icons -- and I had 64M of RAM, just > about the maximum you could ever hope for in those > days. So they got a very puzzled "please explain". > After apparently much head scratching the team leader > explained "we have been using heavily recursive > structures and procedures". That was it, of course. > They rewrote it using iteration instead of recursion > and I ran out of icons before I ran out of memory > again. > > I haven't had a look at the source code, but I > guess that get() is -- almost necessarily -- > implemented recursively. It's fine for loading > flat or shallow data, but, as experience shows, > dreadfully inefficient for deeply embedded > sequences, as you obviously have. > I have to agree with Jacques here. I've found that it is best to only use getc(), gets() and occassionally get_bytes() to read disk files. And I would never recommend get() or print() based on their speed (or lack thereof). --------- Derek. ================================================================== De informatie opgenomen in dit bericht kan vertrouwelijk zijn en is uitsluitend bestemd voor de geadresseerde. Indien u dit bericht onterecht ontvangt wordt u verzocht de inhoud niet te gebruiken en de afzender direct te informeren door het bericht te retourneren. ================================================================== The information contained in this message may be confidential and is intended to be exclusively for the addressee. Should you receive this message unintentionally, please do not use the contents herein and notify the sender immediately by return e-mail. ================================================================== ------_=_NextPart_000_01C20DB7.972B78A0 Content-Type: application/ms-tnef