1. Out of memory
Has anyone any ideas why the following programme works OK
with a 1.47 GHz Athlon 1700+ processor, 512MB RAM and 0.75
to 1.5 GB swap file, albeit with memory swapping for 7 mins,
but runs out of memory with a Pentium (R) 4 CPU 4.39 GHz
(Pentium 640 with hyperthreading and 2MB cache), 2GB RAM
and 3 to 5 GB swap file when the loop number is 41441051?
Both have XP as operating system.
The Pentium box runs at maximum 50% CPU usage for a single
thread application. If I run the programme a few times
simultaneously, the Pentium peaks at 100% CPU usage and goes
into memory swapping but each programme terminates at the same
loop number.
Incidentally, with smaller loops (i.e no memory swapping), the
Pentium's only 80% faster than the Athlon for this application.
--run as an .exw file
atom t
sequence s
s = {}
t = time()
for n = 1 to 200000000 do
s = append(s, n)--sequence s needs 800000024 bytes
end for
?time() -t
machine_proc(26,0)
If I create more, say ten, shorter sequences within a single
application instead of one long one, I can approach 2GB before
it quits but I still run out of memory where the total number of
bytes needed for all sequences is well short of total memory
availability.
Is this a Windows or a Euphoria memory management issue, can I
change any settings, or am I pushing the limits a little too
high in this test?
Thanks in advance for any help,
Bob
2. Re: Out of memory
- Posted by Robert Craig <rds at RapidEuphoria.com>
Nov 18, 2005
-
Last edited Nov 19, 2005
Bob Thompson wrote:
> Has anyone any ideas why the following programme works OK
> with a 1.47 GHz Athlon 1700+ processor, 512MB RAM and 0.75
> to 1.5 GB swap file, albeit with memory swapping for 7 mins,
> but runs out of memory with a Pentium (R) 4 CPU 4.39 GHz
> (Pentium 640 with hyperthreading and 2MB cache), 2GB RAM
> and 3 to 5 GB swap file when the loop number is 41441051?
>
> Both have XP as operating system.
>
> The Pentium box runs at maximum 50% CPU usage for a single
> thread application. If I run the programme a few times
> simultaneously, the Pentium peaks at 100% CPU usage and goes
> into memory swapping but each programme terminates at the same
> loop number.
>
> Incidentally, with smaller loops (i.e no memory swapping), the
> Pentium's only 80% faster than the Athlon for this application.
>
> --run as an .exw file
> atom t
> sequence s
>
> s = {}
>
> t = time()
>
> for n = 1 to 200000000 do
> s = append(s, n)--sequence s needs 800000024 bytes
> end for
>
> ?time() -t
>
> machine_proc(26,0)
>
> If I create more, say ten, shorter sequences within a single
> application instead of one long one, I can approach 2GB before
> it quits but I still run out of memory where the total number of
> bytes needed for all sequences is well short of total memory
> availability.
>
> Is this a Windows or a Euphoria memory management issue, can I
> change any settings, or am I pushing the limits a little too
> high in this test?
I don't know what's going on in your two systems,
but keep in mind that a sequence requires contiguous memory.
Every now and then append() must allocate a new space
bigger than what the sequence is using already, and copy
the sequence into that new space, before the old space is freed.
So sometimes you will have slightly more than
double the space allocated. And it could be that you have
enough total space for those two blocks, but it's fragmented
into smaller blocks that aren't big enough.
Using repeat() to create the size that you need,
might work a lot better.
Regards,
Rob Craig
Rapid Deployment Software
http://www.RapidEuphoria.com
3. Re: Out of memory
Thanks Rob,
Tomorrow, I'll pre-allocate the space beforehand and let you know if
that works.
Regards,
Bob
4. Re: Out of memory
Nothing too sinister using pre-allocation, Rob. I achieved
1.64 GB before the programme ran out of memory, i.e.,
atom t
sequence s
constant repeats = 410000000--s needs 1.640000024 GB
s = repeat({}, repeats)
t = time()
for n = 1 to repeats do
s[n] = n
end for
?time() -t
machine_proc(26,0)
Perhaps it's better to use pre-allocation for all large sequences,
even where its final length is unknown, by pre-allocating in blocks
as, I suspect, Euphoria does. Thanks.
As an aside, and maybe of interest to others, I carried out a few
basic trials comparing the performance of the Athlon 1700+ with that
of the Pentium 3.4 HT CPU for a single programme, and for two programmes
running simultaneously, since the Pentium only uses 50% processing
power on a single programme, i.e.
--------------------------------
--saved as main.exw
constant shells = 2
for n = 1 to shells do
system("start test_prog.exw", 2)
end for
--------------------------------
--saved as test_prog.exw
atom t
sequence s
constant repeats = 10000000, rounds = 10
s = repeat({}, repeats)
t = time()
for m = 1 to rounds do
for n = 1 to repeats do
s[n] = n
end for
end for
?time() -t
machine_proc(26,0)
--------------------------------
Average of 4 runs.
For a single programme running,
Athlon 1.85 sec.
Pentium 0.95 sec. (95% faster)
For two programmes running,
Athlon 3.55 and 3.55 sec.
Pentium 1.40 and 1.42 sec. (152% faster)
Expressed differently, the Athlon executed two programmes
simultaneously in 1.92 times the time it took to execute one,
whereas the Pentium HT took 1.48 times longer for two than it
took for one - significant enough to consider non (or loosely
synchronised) background programme activities such as music,
interfaces etc. with communication through memshare or equivalent.
Regards,
Bob
5. Out of memory
- Posted by flint at acay.com.au
Jun 06, 2002
Hi everyone,
I'm having trouble loading a sequence from file. The sequence consists of
32657 elements, each element having up to 100 sub-elements as well. Total
file size is 72 meg. When attempting to load with get(fn), it runs for
about 1/2 hour and stops with error message:
E:\Euphoria\include\get.e:265 in function Get()
Your program has run out of memory.
One moment please ...
I've tried breaking up the sequence and saving to files separately, then
loading back in separately.
I can then load each sequence on it's own, but when loading all in &
appending to re-create the one sequence I get the same error.
I'm using a PII 400 w/226 meg ram - 180 meg free before running the code.
Is there some limit to total number of elements allowed in a sequence?
Or should I be configuring my system differently ?
Any help muchly appreciated, as I've been stuck on this for a few days now.
Michael.
6. Re: Out of memory
Could you be using a EUPHORIA database? It might be more robust in dealing
with data of this quantity...
> I'm having trouble loading a sequence from file. The sequence consists of
> 32657 elements, each element having up to 100 sub-elements as well. Total
> file size is 72 meg. When attempting to load with get(fn), it runs for
> about 1/2 hour and stops with error message:
7. Re: Out of memory
flint at acay.com.au wrote:
> I'm having trouble loading a sequence from file. The sequence consists of
> 32657 elements, each element having up to 100 sub-elements as well. Total
> file size is 72 meg. When attempting to load with get(fn), it runs for
> about 1/2 hour and stops with [an] error message.
> I'm using a PII 400 w/226 meg ram - 180 meg free before running the code.
It all depends how big your individual sub-elements are, and how big *their*
sub-elements are.
If you're working with integer data, a Euphoria sequence will use 4 bytes
for every integer, even if it's character data you're dealing with. That's a
waste of 3 bytes per character.
For most programs this isn't a problem, but it does tend to fill up memory 4
times as fast as you'd expect it to.
Let's say you've got 72Mb of character data; That's going to take up 288Mb
of space (excluding overheads) - and it becomes obvious why you're running
out of memory.
If you can't process your file a piece at a time, which might be a good idea
given its size, you might want to specifically 'allocate()' some memory and
handle the data outside of Euphoria's built-in data manipulation.
> Is there some limit to total number of elements allowed in a sequence?
I reckon you'd probably have trouble somewhere around sequence index
'1073741823', but you'd need well over 4Gb of RAM before you hit that
problem. If I'm wrong and sequence indices are limited by _atoms_ not
integers, you'd need something like 16Pb (sixteen petabytes ~= 18
quadrillion bytes) of RAM before you have problems.
So 'No.' :)
> Or should I be configuring my system differently ?
Handling your data as smaller pieces is probably the way to go here.
Carl
8. Re: Out of memory
Hi Michael,
----------
> Îò: flint at acay.com.au
> Êîìó: EUforum <EUforum at topica.com>
> Òåìà: Out of memory
> Äàòà: ÷åòâåðã, Èþíü 06, 2002 15:35
>
> Hi everyone,
>
> I'm having trouble loading a sequence from file.
> The sequence consists of 32657 elements,
> each element having up to 100 sub-elements as well.
> Total file size is 72 meg. When attempting to load
> with get(fn), it runs for
> about 1/2 hour and stops with error message:
>
> E:\Euphoria\include\get.e:265 in function Get()
> Your program has run out of memory.
> One moment please ...
>
> I've tried breaking up the sequence and saving
> to files separately, then loading back in separately.
> I can then load each sequence on it's own,
> but when loading all in & appending to re-create
> the one sequence I get the same error.
>
> I'm using a PII 400 w/226 meg ram - 180 meg free
> before running the code. Is there some limit to
> total number of elements allowed in a sequence?
> Or should I be configuring my system differently ?
>
> Any help muchly appreciated, as I've been stuck on
> this for a few days now.
>
> Michael.
Try please one more recipe.
If you run EX.EXE then run it on plain DOS32, in MS DOS mode
of your machine, not in DOS box on Windows.
If you run on Windows then use yourPROGRAM.EXW and EXW.EXE.
Good luck.
Regards,
Igor Kachan,
kinz at peterlink.ru
9. Re: Out of memory
- Posted by a.tammer at hetnet.nl
Jun 06, 2002
Anyone any idea, or IO.sys, for WinMe,
to enable Real DOS32-bootup as well?
If so please mail&attach to
a.tammer at hetnet.nl
EUrs,
a@t
10. Re: Out of memory
- Posted by rforno at tutopia.com
Jun 07, 2002
Igor:
You are right, I know by experience, but probably the best approach for him
is to change the problem structure. Not knowing what the problem is, it is
difficult to say how to do that.
Regards.
----- Original Message -----
From: "Igor Kachan" <kinz at peterlink.ru>
To: "EUforum" <EUforum at topica.com>
Subject: Re: Out of memory
Hi Michael,
----------
> Îò: flint at acay.com.au
> Êîìó: EUforum <EUforum at topica.com>
> Òåìà: Out of memory
> Äàòà: ÷åòâåðã, Èþíü 06, 2002 15:35
>
> Hi everyone,
>
> I'm having trouble loading a sequence from file.
> The sequence consists of 32657 elements,
> each element having up to 100 sub-elements as well.
> Total file size is 72 meg. When attempting to load
> with get(fn), it runs for
> about 1/2 hour and stops with error message:
>
> E:\Euphoria\include\get.e:265 in function Get()
> Your program has run out of memory.
> One moment please ...
>
> I've tried breaking up the sequence and saving
> to files separately, then loading back in separately.
> I can then load each sequence on it's own,
> but when loading all in & appending to re-create
> the one sequence I get the same error.
>
> I'm using a PII 400 w/226 meg ram - 180 meg free
> before running the code. Is there some limit to
> total number of elements allowed in a sequence?
> Or should I be configuring my system differently ?
>
> Any help muchly appreciated, as I've been stuck on
> this for a few days now.
>
> Michael.
Try please one more recipe.
If you run EX.EXE then run it on plain DOS32, in MS DOS mode
of your machine, not in DOS box on Windows.
If you run on Windows then use yourPROGRAM.EXW and EXW.EXE.
Good luck.
Regards,
Igor Kachan,
kinz at peterlink.ru
