Out of memory error
E-Mail Memorandum Friday, 25 April 1997/14:31/L
Greetings ALL!
Help me please. I do not understand what I am doing wrong, if any thing.
I picked up Euphoria a few days ago to look at. I figured it might be
a good language to use for a freeware tagline manager/thief I am
working on. One of the areas of concern for me is the ability to
handle very large files with out splitting them into sections to work
on then merging them together again. Euphoria seemed to have the
ability to handle this with its advertized use of Expanded/Extended
memory.
I really don't want to use C++ for this project as I want users to be
able to modify the code if they desire. And Basic's memory limits
don't allow for the simple sort and file append functions I want to
use.
I setup my test.in file (1.9M) and using MSDOS ran
ex filesort.ex <test.in>test.out
in the demo/bench dir. I got an out of memory error message. By using
trace and by adding some debugging vars;
LRead -- Number of lines read from file
MemUsed -- Bytes of memory used to store sequence
I found this error takes place during the file reading loop. (see
enclosed ex1.err)
When I ran it in a DOS window with Win3.1 and it read the entire file
but the memory swapping during sort is unacceptable. The program ran
for 35 minutes (okay so I went to lunch during the run but I figured
it would finish and the prompt would be waiting on me when I got back)
before I killed it.
If I reduced the size of the input file to around 1.4M it takes
roughly 203 seconds to sort the data running from DOS. Haven't tried
this in Windows yet.
I have taken the liberty of removing the Buffer and line Var data from
the snips to shorten the post.
====8<----- SNIP memory just prior to run ------->8====
Modules using memory below 1 MB:
Name Total = Conventional + Upper Memory
-------- ---------------- ---------------- ----------------
MSDOS 21,037 (21K) 21,037 (21K) 0 (0K)
HIMEM 1,168 (1K) 1,168 (1K) 0 (0K)
EMM386 3,120 (3K) 3,120 (3K) 0 (0K)
MOUSE 16,080 (16K) 16,080 (16K) 0 (0K)
COMMAND 3,376 (3K) 3,376 (3K) 0 (0K)
SMARTDRV 26,720 (26K) 26,720 (26K) 0 (0K)
TC 5,696 (6K) 5,696 (6K) 0 (0K)
COMMAND 3,216 (3K) 3,216 (3K) 0 (0K)
DISPLAY 8,304 (8K) 0 (0K) 8,304 (8K)
ANSI 4,208 (4K) 0 (0K) 4,208 (4K)
RAMDRIVE 1,264 (1K) 0 (0K) 1,264 (1K)
VIRSTOP 42,208 (41K) 0 (0K) 42,208 (41K)
DOSKEY 4,144 (4K) 0 (0K) 4,144 (4K)
Free 599,408 (585K) 574,688 (561K) 24,720 (24K)
Memory Summary:
Type of Memory Total = Used + Free
---------------- ---------- ---------- ----------
Conventional 655,360 80,672 574,688
Upper 84,848 60,128 24,720
Reserved 393,216 393,216 0
Extended (XMS)* 19,838,096 11,733,136 8,104,960
---------------- ---------- ---------- ----------
Total memory 20,971,520 12,267,152 8,704,368
Total under 1 MB 740,208 140,800 599,408
Total Expanded (EMS) 20,250,624 (19,776K
Free Expanded (EMS)* 8,355,840 (8,160K)
* EMM386 is using XMS memory to simulate EMS memory as needed.
Free EMS memory may change as free XMS memory changes.
Largest executable program size 574,496 (561K)
Largest free upper memory block 13,728 (13K)
MS-DOS is resident in the high memory area.
====8<----- SNIP ex1.err ------->8====
FILESORT.EX:27 in procedure file_sort()
Your program has run out of memory.
One moment please...
LRead = 35892
MemUsed = 1801298
buffer = {data removed to conserve space}
sorted_buffer = <no value>
line = {data removed to conserve space}
... called from FILESORT.EX:51
Global & Local Variables
E:\EUPHORIA\INCLUDE\sort.e:
x = <no value>
The 25 (max) most-recently traced lines:
FILESORT.EX:27 line = gets(0) -- read a line (sequence of chars)
from standard input
FILESORT.EX:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
FILESORT.EX:32 LRead=LRead+1 -- debug code
FILESORT.EX:33 MemUsed=MemUsed+length(line) -- debug code
FILESORT.EX:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
FILESORT.EX:38 end while
FILESORT.EX:27 line = gets(0) -- read a line (sequence of chars)
from standard input
FILESORT.EX:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
FILESORT.EX:32 LRead=LRead+1 -- debug code
FILESORT.EX:33 MemUsed=MemUsed+length(line) -- debug code
FILESORT.EX:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
FILESORT.EX:38 end while
FILESORT.EX:27 line = gets(0) -- read a line (sequence of chars)
from standard input
FILESORT.EX:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
FILESORT.EX:32 LRead=LRead+1 -- debug code
FILESORT.EX:33 MemUsed=MemUsed+length(line) -- debug code
FILESORT.EX:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
FILESORT.EX:38 end while
FILESORT.EX:27 line = gets(0) -- read a line (sequence of chars)
from standard input
FILESORT.EX:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
FILESORT.EX:32 LRead=LRead+1 -- debug code
FILESORT.EX:33 MemUsed=MemUsed+length(line) -- debug code
FILESORT.EX:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
FILESORT.EX:38 end while
FILESORT.EX:27 line = gets(0) -- read a line (sequence of chars)
from standard input
====8<----- SNIP ex2.err ------->8====
filesort.ex:41 in procedure file_sort()
program aborted
LRead = 37619
MemUsed = 1890627
buffer = {data removed to conserve space}
sorted_buffer = <no value>
line = -1
... called from filesort.ex:51
Global & Local Variables
E:\EUPHORIA\INCLUDE\sort.e:
x = <no value>
The 25 (max) most-recently traced lines:
filesort.ex:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
filesort.ex:38 end while
filesort.ex:27 line = gets(0) -- read a line (sequence of chars)
from standard input
filesort.ex:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
filesort.ex:32 LRead=LRead+1 -- debug code
filesort.ex:33 MemUsed=MemUsed+length(line) -- debug code
filesort.ex:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
filesort.ex:38 end while
filesort.ex:27 line = gets(0) -- read a line (sequence of chars)
from standard input
filesort.ex:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
filesort.ex:32 LRead=LRead+1 -- debug code
filesort.ex:33 MemUsed=MemUsed+length(line) -- debug code
filesort.ex:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
filesort.ex:38 end while
filesort.ex:27 line = gets(0) -- read a line (sequence of chars)
from standard input
filesort.ex:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
filesort.ex:32 LRead=LRead+1 -- debug code
filesort.ex:33 MemUsed=MemUsed+length(line) -- debug code
filesort.ex:35 buffer = append(buffer, line) -- add the line to
the buffer of lines
filesort.ex:38 end while
filesort.ex:27 line = gets(0) -- read a line (sequence of chars)
from standard input
filesort.ex:28 if atom(line) then -- gets(0) returns atom -1 on
end of file
filesort.ex:29 exit -- quit the while loop
filesort.ex:40 trace(1)
filesort.ex:41 sorted_buffer = sort(buffer) -- call the sort
routine, it will compare
Thanks in advance for any help you can give to me in solving this
problem.
Cheers!
Larry Poos
-[USMC (Retar{bks}{bks}ired) Havelock, NC]-
e-mail: ldpoos at juno.com
or
Fido: 1:3629/101.6
.!. We got plenty of youth - What we REALLY need is a Fountain of Smart!
--- Terminate 4.00/Pro
|
Not Categorized, Please Help
|
|
