1. Out of memory error
E-Mail Memorandum Friday, 25 April 1997/14:31/L
Greetings ALL!
Help me please. I do not understand what I am doing wrong, if any thing.
I picked up Euphoria a few days ago to look at. I figured it might be
a good language to use for a freeware tagline manager/thief I am
working on. One of the areas of concern for me is the ability to
handle very large files with out splitting them into sections to work
on then merging them together again. Euphoria seemed to have the
ability to handle this with its advertized use of Expanded/Extended
memory.
I really don't want to use C++ for this project as I want users to be
able to modify the code if they desire. And Basic's memory limits
don't allow for the simple sort and file append functions I want to
use.
I setup my test.in file (1.9M) and using MSDOS ran
ex filesort.ex <test.in>test.out
in the demo/bench dir. I got an out of memory error message. By using
trace and by adding some debugging vars;
LRead -- Number of lines read from file
MemUsed -- Bytes of memory used to store sequence
I found this error takes place during the file reading loop. (see
enclosed ex1.err)
When I ran it in a DOS window with Win3.1 and it read the entire file
but the memory swapping during sort is unacceptable. The program ran
for 35 minutes (okay so I went to lunch during the run but I figured
it would finish and the prompt would be waiting on me when I got back)
before I killed it.
If I reduced the size of the input file to around 1.4M it takes
roughly 203 seconds to sort the data running from DOS. Haven't tried
this in Windows yet.
I have taken the liberty of removing the Buffer and line Var data from
the snips to shorten the post.
====8<----- SNIP memory just prior to run ------->8====
Modules using memory below 1 MB:
Name Total = Conventional + Upper Memory
-------- ---------------- ---------------- ----------------
MSDOS 21,037 (21K) 21,037 (21K) 0 (0K)
HIMEM 1,168 (1K) 1,168 (1K) 0 (0K)
EMM386 3,120 (3K) 3,120 (3K) 0 (0K)
MOUSE 16,080 (16K) 16,080 (16K) 0 (0K)
COMMAND 3,376 (3K) 3,376 (3K) 0 (0K)
SMARTDRV 26,720 (26K) 26,720 (26K) 0 (0K)
TC 5,696 (6K) 5,696 (6K) 0 (0K)
COMMAND 3,216 (3K) 3,216 (3K) 0 (0K)
DISPLAY 8,304 (8K) 0 (0K) 8,304 (8K)
ANSI 4,208 (4K) 0 (0K) 4,208 (4K)
RAMDRIVE 1,264 (1K) 0 (0K) 1,264 (1K)
VIRSTOP 42,208 (41K) 0 (0K) 42,208 (41K)
DOSKEY 4,144 (4K) 0 (0K) 4,144 (4K)
Free 599,408 (585K) 574,688 (561K) 24,720 (24K)
Memory Summary:
Type of Memory Total = Used + Free
---------------- ---------- ---------- ----------
Conventional 655,360 80,672 574,688
Upper 84,848 60,128 24,720
Reserved 393,216 393,216 0
Extended (XMS)* 19,838,096 11,733,136 8,104,960
---------------- ---------- ---------- ----------
Total memory 20,971,520 12,267,152 8,704,368
Total under 1 MB 740,208 140,800 599,408
Total Expanded (EMS) 20,250,624 (19,776K
Free Expanded (EMS)* 8,355,840 (8,160K)
* EMM386 is using XMS memory to simulate EMS memory as needed.
Free EMS memory may change as free XMS memory changes.
Largest executable program size 574,496 (561K)
Largest free upper memory block 13,728 (13K)
MS-DOS is resident in the high memory area.
====8<----- SNIP ex1.err ------->8====
FILESORT.EX:27 in procedure file_sort()
Your program has run out of memory.
One moment please...
LRead = 35892
MemUsed = 1801298
buffer = {data removed to conserve space}
sorted_buffer = <no value>
line = {data removed to conserve space}
... called from FILESORT.EX:51
Global & Local Variables
E:\EUPHORIA\INCLUDE\sort.e:
x = <no value>
The 25 (max) most-recently traced lines:
FILESORT.EX:27 line = gets(0) -- read a line (sequence of chars)
from standard input
FILESORT.EX:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
FILESORT.EX:32 LRead=LRead+1 -- debug code
FILESORT.EX:33 MemUsed=MemUsed+length(line) -- debug code
FILESORT.EX:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
FILESORT.EX:38 end while
FILESORT.EX:27 line = gets(0) -- read a line (sequence of chars)
from standard input
FILESORT.EX:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
FILESORT.EX:32 LRead=LRead+1 -- debug code
FILESORT.EX:33 MemUsed=MemUsed+length(line) -- debug code
FILESORT.EX:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
FILESORT.EX:38 end while
FILESORT.EX:27 line = gets(0) -- read a line (sequence of chars)
from standard input
FILESORT.EX:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
FILESORT.EX:32 LRead=LRead+1 -- debug code
FILESORT.EX:33 MemUsed=MemUsed+length(line) -- debug code
FILESORT.EX:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
FILESORT.EX:38 end while
FILESORT.EX:27 line = gets(0) -- read a line (sequence of chars)
from standard input
FILESORT.EX:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
FILESORT.EX:32 LRead=LRead+1 -- debug code
FILESORT.EX:33 MemUsed=MemUsed+length(line) -- debug code
FILESORT.EX:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
FILESORT.EX:38 end while
FILESORT.EX:27 line = gets(0) -- read a line (sequence of chars)
from standard input
====8<----- SNIP ex2.err ------->8====
filesort.ex:41 in procedure file_sort()
program aborted
LRead = 37619
MemUsed = 1890627
buffer = {data removed to conserve space}
sorted_buffer = <no value>
line = -1
... called from filesort.ex:51
Global & Local Variables
E:\EUPHORIA\INCLUDE\sort.e:
x = <no value>
The 25 (max) most-recently traced lines:
filesort.ex:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
filesort.ex:38 end while
filesort.ex:27 line = gets(0) -- read a line (sequence of chars)
from standard input
filesort.ex:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
filesort.ex:32 LRead=LRead+1 -- debug code
filesort.ex:33 MemUsed=MemUsed+length(line) -- debug code
filesort.ex:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
filesort.ex:38 end while
filesort.ex:27 line = gets(0) -- read a line (sequence of chars)
from standard input
filesort.ex:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
filesort.ex:32 LRead=LRead+1 -- debug code
filesort.ex:33 MemUsed=MemUsed+length(line) -- debug code
filesort.ex:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
filesort.ex:38 end while
filesort.ex:27 line = gets(0) -- read a line (sequence of chars)
from standard input
filesort.ex:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
filesort.ex:32 LRead=LRead+1 -- debug code
filesort.ex:33 MemUsed=MemUsed+length(line) -- debug code
filesort.ex:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
filesort.ex:38 end while
filesort.ex:27 line = gets(0) -- read a line (sequence of chars)
from standard input
filesort.ex:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
filesort.ex:29 exit -- quit the while loop
filesort.ex:40 trace(1)
filesort.ex:41 sorted_buffer = sort(buffer) -- call the sort
routine, it will compare
Thanks in advance for any help you can give to me in solving this
problem.
Cheers!
Larry Poos
-[USMC (Retar{bks}{bks}ired) Havelock, NC]-
e-mail: ldpoos at juno.com
or
Fido: 1:3629/101.6
.!. We got plenty of youth - What we REALLY need is a Fountain of Smart!
--- Terminate 4.00/Pro
2. Out of memory error
- Posted by Robert Craig <robert_craig at COMPUSERVE.COM>
Apr 25, 1997
-
Last edited Apr 26, 1997
Larry D Poos writes:
> When I ran it in a DOS window with Win3.1 and it read the entire file
> but the memory swapping during sort is unacceptable. The program ran
> for 35 minutes (okay so I went to lunch during the run but I figured
> it would finish and the prompt would be waiting on me when I got back)
> before I killed it.
> If I reduced the size of the input file to around 1.4M it takes
> roughly 203 seconds to sort the data running from DOS. Haven't tried
> this in Windows yet.
The filesort.ex demo program (less than 20 lines of code) is pretty fast
at sorting data up to a "reasonable" size. If you want to push the limits
of your machine's memory you should consider the following:
1. Euphoria stores all integer values in 4 bytes. i.e. characters
are stored in 4 bytes (not 1). This might change in the future,
but for now it makes the implementation simple and fast.
2. filesort.ex makes a full copy of the input data, i.e.
sorted_buffer = sort(buffer)
creates a second sorted copy of the input data. If
you were to use something like great_sort of allsorts.ex you
could sort buffer in-place.
3. The disk swapping feature of the DOS extender has its limitations,
as you've discovered. Some algorithms, such as Euphoria's standard
shell sort (sort.e) are not well suited to a least-recently-used swapping
algorithm.If your algorithm sweeps through entire large sequences, doing
just a little bit of work on each element, you will not benefit from LRU.
You'll be frequently accessing the disk and performance can drop by orders
of magnitude. I once experimented a bit with this, figuring that a
"divide and conquer" type of sort, such as quick sort (allsorts.ex) might
be better suited to LRU swapping. I think the answer was yes, but it was
still pretty slow.
4. For Euphoria Gurus Only: If you have lots of duplicate lines of input,
you might try matching each incoming line against lines you've already
read in. Then you can do:
buffer = append(buffer, buffer[i])
instead of:
buffer = append(buffer, line)
whenever the input line and buffer[i] are the same.
In this way Euphoria will only create one copy of that line,
and will point to it from many elements of buffer, saving memory.
You'll need a very fast check for duplicates, or this will
be slow.
Good Luck,
Rob Craig
Rapid Deployment Software
3. Re: Out of memory error
On Fri, 25 Apr 1997, Robert Craig wrote:
> 4. For Euphoria Gurus Only: If you have lots of duplicate lines of input,
Who qualifies as a "Euphoria Guru" and do they have a secret
club\handshake?
Michael Packard
Lord Generic Productions
lgp at exo.com http://exo.com/~lgp
A Crash Course in Game Design and Production
http://exo.com/~lgp/euphoria
4. Out of memory error
Robert Craig mistakenly said:
> 2. filesort.ex makes a full copy of the input data, i.e.
>
> sorted_buffer = sort(buffer)
>
> creates a second sorted copy of the input data. If
> you were to use something like great_sort of allsorts.ex you
> could sort buffer in-place.
Actually Euphoria *won't* copy the data. If buffer is a sequence
of 10000 strings, Euphoria will create a new sorted sequence for sorted_buffer
that uses 10000 pointers (x 4 bytes each = 40000 bytes). After sorting,
each string will be pointed to from one element of buffer and one element
of sorted_buffer. There will only be a single copy in memory of each
string.
As a Euphoria programmer these "pointers" are not visible to you
unless, Michael, you are a "Euphoria Guru", or you are wearing an
excellent pair of 3-D glasses. Euphoria creates the illusion that
buffer and sorted_buffer contain 2 distinct copies of all the data.
Conclusion: An in-place sort would only save a small percentage
of the total memory required here.
Regards,
Rob Craig
Rapid Deployment Software
5. Re: Out of memory error
- Posted by Michael Packard <lgp at EXO.COM>
Apr 25, 1997
-
Last edited Apr 26, 1997
On Sat, 26 Apr 1997, Robert Craig wrote:
>
> As a Euphoria programmer these "pointers" are not visible to you
> unless, Michael, you are a "Euphoria Guru", or you are wearing an
> excellent pair of 3-D glasses. Euphoria creates the illusion that
> buffer and sorted_buffer contain 2 distinct copies of all the data.
>
I actually DO have an excellent pair of 3D Glasses... =)
Michael Packard
Lord Generic Productions
lgp at exo.com http://exo.com/~lgp
A Crash Course in Game Design and Production
http://exo.com/~lgp/euphoria
6. Re: Out of memory error
Hi Larry
This is a new one on me. Euphoria is supposed to create a swap file on your
hard drive when demands for memory are more than what you have available. I
am not sure that it is a problem with main memory because if you look at my
system memory below, you can see I am operating with less memory than you
are, yet have written some programs that involve heavy graphics and a lot
of file I/O. How much FREE hard drive space do you have available on your
hard drive? I find the more free space I have on the hard drive, the more I
can push the memory envelope.
Memory Summary:
Type of Memory Total Used Free
---------------- ----------- ----------- -----------
Conventional 651,264 85,728 565,536
Upper 92,816 92,816 0
Reserved 0 0 0
Extended (XMS) 15,635,824 2,368,880 13,266,944
---------------- ----------- ----------- -----------
Total memory 16,379,904 2,547,424 13,832,480
Total under 1 MB 744,080 178,544 565,536
Total Expanded (EMS) 13,975,552 (13M)
Free Expanded (EMS) 13,975,552 (13M)
Largest executable program size 565,520 (552K)
Largest free upper memory block 0 (0K)
MS-DOS is resident in the high memory area.
Thanks
David Gay
http://www.interlog.com/~moggie/Euphoria
"A Beginner's Guide To Euphoria"
Serving The Euphoria Programming Community
Since April 21st, 1996.
7. Re: Out of memory error
On Sat, 26 Apr 1997 05:08:29 -0400 David Alan Gay writes:
>Hi Larry
>
>This is a new one on me. Euphoria is supposed to create a swap file on
your
>hard drive when demands for memory are more than what you have
available.
I suspect the fault is in the size of the data in the buffer variable,
causing the choke. This is an issue as the program I am writing must be
able to run on machines with many different memory configurations and
sizes.
>I am not sure that it is a problem with main memory because if you look
>at my system memory below, you can see I am operating with less memory
than
>you are, yet have written some programs that involve heavy graphics and
a
>lot of file I/O. How much FREE hard drive space do you have available on
>your hard drive?
About 104Mb on that partition.
> I find the more free space I have on the hard drive, the more I can
push the
> memory envelope.
Robert Craig wrote in a message that characters are stored as integers (4
bytes each, not 1 as in C++) so the 1.9meg of characters in the file uses
7.6meg of memory to store. with only 8.7meg free in this configuration
it is pushing the edge of the memory available. A lot of the people who
will use my Freeware program are running with 4 to 8meg of total memory
and this is an issue.
One of the reasons I was looking at Euphoria was its dynamic memory
management, so I did not have to mess with the memory checking and
allocation issues with very large data files. Oh, well I guess I'll
unzip the C++ compiler and get to work. Maybe I can find some good memory
management code in Snippets to plagiarize.
Have a great day!
Larry D. Poos
-[USMC (Retar{bks}{bks}ired) Havelock, NC]-
e-mail: ldpoos at juno.com
or
Fido: 1:3629/101.6
8. Re: Out of memory error
On Fri, 25 Apr 1997 21:41:28 -0400 Robert Craig writes:
>Larry D Poos writes:
>> When I ran it in a DOS window with Win3.1 and it read the entire file
>The filesort.ex demo program (less than 20 lines of code) is pretty
>fast at sorting data up to a "reasonable" size. If you want to push the
>limits of your machine's memory you should consider the following:
>
> 1. Euphoria stores all integer values in 4 bytes. i.e. characters
> are stored in 4 bytes (not 1). This might change in the future,
> but for now it makes the implementation simple and fast.
Yes that almost maxes out the available memory 1.9M characters = 7.6MB
memory out of 8MB available. This code will have to work on machines with
a minimum of 1MB so a swap file is the only way to handle it.
> you were to use something like great_sort of allsorts.ex you
> could sort buffer in-place.
Yep, did that too. Swapping was still unacceptable in the windows
environment. Still could not get past the read loop without the "out of
memory error" in the DOS environment. I disabled the Ramdrive and opened
up an additional 10MB of memory and got it past the read-loop but even
the great_sort was to slow (18min) on this file. I have a program using a
Radix sort that benched this same file in 1.5 min, now if I can find the
radix sort alogrithms so I can understand how they work then I can write
one for the Euphoria language.
Once I get the code to work as an entity then I can work on the memory
swapping problem as I now feel it is with the extender in DOS mode.
> 3. The disk swapping feature of the DOS extender has its
>limitations, as you've discovered.
I think this is what is causing it to choke with the Buffer variable
being so large. Since I can get past the read loop when I run in a DOS
box under windows. If I understand your docs correctly, when run under
windows Euphoria uses the windows swap protocal and when run under DOS it
uses the extenders swap protocol.
>I once experimented a bit with this, figuring that
>a "divide and conquer" type of sort, such as quick sort (allsorts.ex)
might
>be better suited to LRU swapping. I think the answer was yes, but it was
still pretty slow.
I think your are correct on this, a "divide and conquer" method to handle
the large files is going to be the only way to handle it. The 'trick' is
going to be to develop the fastest possible algorithm to handle the
different file sizes expected to be encountered.
> 4. For Euphoria Gurus Only: If you have lots of duplicate lines of
>input, you might try matching each incoming line against lines you've
>already read in.
> You'll need a very fast check for duplicates, or this will
> be slow.
This speed problem is what I want to overcome in the steal taglines
portion of the program. Selecting the line to steal is not a problem it
is the dup checking that gets slow with large tagline files especially if
you want to use Fuzzy logic dup-checking. I feel that ordering both the
source and destination files prior to the check would speed this up a bit
by allowing a type of indexed look-up in the destination file before the
dup check. At some point though the product has to be combined and
ordered. If it can't read in and ordered as one collection of objects
then it has to be split, operated on and merged, which may cost in speed
as these will require disk I/O vs ram I/O.
Thanks for your insight and help with my difficulty.
Larry D. Poos
-[USMC (Retar{bks}{bks}ired) Havelock, NC]-
e-mail: ldpoos at juno.com
or
Fido: 1:3629/101.6
9. Out of memory error
- Posted by kayhlan at EARTHLINK.NET
Feb 20, 2001
-
Last edited Feb 21, 2001
Hello folks,
I was hoping to find out if anyone of you knowledgeable people has experienced
"out of memory" errors when running large programs in Euphoria. I use allocate
and free() in only two places and in neither case is there an exit out before
the free() is executed. I do quite a bit of manipulation of large sequences,
most averaging a length of 380,000. My program generally starts using about 24k
of memory and within 5 to 6 hours shoots up to the 80-90 range, which then
exceeds he maximum amount of memory and terminates. I have monitored my
databases and there is very little variation in size from start to finish.
If anyone could give me some ideas as to ways to check this I would appreciate
it it immensely. I am running in a Linux system, if that helps at all.
Thank you again, in advance.
Kayhlan A.
>
10. Re: Out of memory error
On 20 Feb 2001, at 23:50, kayhlan at EARTHLINK.NET wrote:
> Hello folks,
>
> I was hoping to find out if anyone of you knowledgeable people has experienced
> "out of memory" errors when running large programs in Euphoria. I use allocate
> and free() in only two places and in neither case is there an exit out before
> the free() is executed. I do quite a bit of manipulation of large sequences,
> most averaging a length of 380,000.
Eu uses 4 bytes for each atom in the sequence, so your 380K sequence is easily
useing 1.5 megabytes. Possibly you aren't freeing it, or Eu doesn't have time to
free it?
Kat
11. Re: Out of memory error
Kat wrote:
> ...
> Eu uses 4 bytes for each atom in the sequence, ...
An atom could be of 8 byte length, and one has to add space 4 bytes for
a pointer at least, that makes 12 bytes per atom in a sequence minimum.
That how i think it might be. Ask Rob about the maximum amount. This was
discussed a long time ago.
Have a nice day, Rolf
12. Re: Out of memory error
Thank you for your responses, but I need to clarify something.....the memory
size I was
quoting was in mb not kb as I stated. The sequences I use are initialized to a
certain length
at the beginning of the program. I don't allocate and since it is a sequence, I
don't see how
I could free(). The size of the sequences is not wavering by much which is why I
am having a
difficult time finding the memory leak.
Thank you again in advance for your help,
Kayhlan
Kat wrote:
> On 20 Feb 2001, at 23:50, kayhlan at EARTHLINK.NET wrote:
>
> > Hello folks,
> >
> > I was hoping to find out if anyone of you knowledgeable people has
> > experienced
> > "out of memory" errors when running large programs in Euphoria. I use
> > allocate
> > and free() in only two places and in neither case is there an exit out
> > before
> > the free() is executed. I do quite a bit of manipulation of large
> > sequences,
> > most averaging a length of 380,000.
>
> Eu uses 4 bytes for each atom in the sequence, so your 380K sequence is easily
> useing 1.5 megabytes. Possibly you aren't freeing it, or Eu doesn't have time
> to free it?
>
> Kat
>
13. Re: Out of memory error
Kayhlan A. writes:
<problems with "out of memory">
Can you send me your program in a form
that I can run it? I have ways of tracking
down memory leaks, if that's what it is.
Remember that sequence elements
need at least 4 bytes each. Also be aware
that large temporary sequences might be created
when you perform an operation on a sequence.
Regards,
Rob Craig
Rapid Deployment Software
http://www.RapidEuphoria.com
14. Re: Out of memory error
- Posted by kinz at peterlink.ru
Feb 21, 2001
Dear Kayhlan:
> The sequences I use are initialized to a certain length
> at the beginning of the program.
> I don't allocate and since it is a sequence,
> I don't see how I could free().
Try please
--------------------
sequence s
s=repeat(1,100)-- Eu allocates
? s
s={} -- Eu frees
?s
-------------------
Regards,
Igor Kachan
kinz at peterlink.ru
