1. Out of memory
- Posted by Bob Thompson <rthompson at rthompson.karoo.co.uk> Nov 18, 2005
- 635 views
Has anyone any ideas why the following programme works OK with a 1.47 GHz Athlon 1700+ processor, 512MB RAM and 0.75 to 1.5 GB swap file, albeit with memory swapping for 7 mins, but runs out of memory with a Pentium (R) 4 CPU 4.39 GHz (Pentium 640 with hyperthreading and 2MB cache), 2GB RAM and 3 to 5 GB swap file when the loop number is 41441051? Both have XP as operating system. The Pentium box runs at maximum 50% CPU usage for a single thread application. If I run the programme a few times simultaneously, the Pentium peaks at 100% CPU usage and goes into memory swapping but each programme terminates at the same loop number. Incidentally, with smaller loops (i.e no memory swapping), the Pentium's only 80% faster than the Athlon for this application. --run as an .exw file atom t sequence s s = {} t = time() for n = 1 to 200000000 do s = append(s, n)--sequence s needs 800000024 bytes end for ?time() -t machine_proc(26,0) If I create more, say ten, shorter sequences within a single application instead of one long one, I can approach 2GB before it quits but I still run out of memory where the total number of bytes needed for all sequences is well short of total memory availability. Is this a Windows or a Euphoria memory management issue, can I change any settings, or am I pushing the limits a little too high in this test? Thanks in advance for any help, Bob
2. Re: Out of memory
- Posted by Robert Craig <rds at RapidEuphoria.com> Nov 18, 2005
- 604 views
- Last edited Nov 19, 2005
Bob Thompson wrote: > Has anyone any ideas why the following programme works OK > with a 1.47 GHz Athlon 1700+ processor, 512MB RAM and 0.75 > to 1.5 GB swap file, albeit with memory swapping for 7 mins, > but runs out of memory with a Pentium (R) 4 CPU 4.39 GHz > (Pentium 640 with hyperthreading and 2MB cache), 2GB RAM > and 3 to 5 GB swap file when the loop number is 41441051? > > Both have XP as operating system. > > The Pentium box runs at maximum 50% CPU usage for a single > thread application. If I run the programme a few times > simultaneously, the Pentium peaks at 100% CPU usage and goes > into memory swapping but each programme terminates at the same > loop number. > > Incidentally, with smaller loops (i.e no memory swapping), the > Pentium's only 80% faster than the Athlon for this application. > > --run as an .exw file > atom t > sequence s > > s = {} > > t = time() > > for n = 1 to 200000000 do > s = append(s, n)--sequence s needs 800000024 bytes > end for > > ?time() -t > > machine_proc(26,0) > > If I create more, say ten, shorter sequences within a single > application instead of one long one, I can approach 2GB before > it quits but I still run out of memory where the total number of > bytes needed for all sequences is well short of total memory > availability. > > Is this a Windows or a Euphoria memory management issue, can I > change any settings, or am I pushing the limits a little too > high in this test? I don't know what's going on in your two systems, but keep in mind that a sequence requires contiguous memory. Every now and then append() must allocate a new space bigger than what the sequence is using already, and copy the sequence into that new space, before the old space is freed. So sometimes you will have slightly more than double the space allocated. And it could be that you have enough total space for those two blocks, but it's fragmented into smaller blocks that aren't big enough. Using repeat() to create the size that you need, might work a lot better. Regards, Rob Craig Rapid Deployment Software http://www.RapidEuphoria.com
3. Re: Out of memory
- Posted by Bob Thompson <rthompson at rthompson.karoo.co.uk> Nov 18, 2005
- 596 views
- Last edited Nov 19, 2005
Thanks Rob, Tomorrow, I'll pre-allocate the space beforehand and let you know if that works. Regards, Bob
4. Re: Out of memory
- Posted by Bob Thompson <rthompson at rthompson.karoo.co.uk> Nov 20, 2005
- 615 views
Nothing too sinister using pre-allocation, Rob. I achieved 1.64 GB before the programme ran out of memory, i.e., atom t sequence s constant repeats = 410000000--s needs 1.640000024 GB s = repeat({}, repeats) t = time() for n = 1 to repeats do s[n] = n end for ?time() -t machine_proc(26,0) Perhaps it's better to use pre-allocation for all large sequences, even where its final length is unknown, by pre-allocating in blocks as, I suspect, Euphoria does. Thanks. As an aside, and maybe of interest to others, I carried out a few basic trials comparing the performance of the Athlon 1700+ with that of the Pentium 3.4 HT CPU for a single programme, and for two programmes running simultaneously, since the Pentium only uses 50% processing power on a single programme, i.e. -------------------------------- --saved as main.exw constant shells = 2 for n = 1 to shells do system("start test_prog.exw", 2) end for -------------------------------- --saved as test_prog.exw atom t sequence s constant repeats = 10000000, rounds = 10 s = repeat({}, repeats) t = time() for m = 1 to rounds do for n = 1 to repeats do s[n] = n end for end for ?time() -t machine_proc(26,0) -------------------------------- Average of 4 runs. For a single programme running, Athlon 1.85 sec. Pentium 0.95 sec. (95% faster) For two programmes running, Athlon 3.55 and 3.55 sec. Pentium 1.40 and 1.42 sec. (152% faster) Expressed differently, the Athlon executed two programmes simultaneously in 1.92 times the time it took to execute one, whereas the Pentium HT took 1.48 times longer for two than it took for one - significant enough to consider non (or loosely synchronised) background programme activities such as music, interfaces etc. with communication through memshare or equivalent. Regards, Bob
5. Out of memory
- Posted by flint at acay.com.au Jun 06, 2002
- 577 views
Hi everyone, I'm having trouble loading a sequence from file. The sequence consists of 32657 elements, each element having up to 100 sub-elements as well. Total file size is 72 meg. When attempting to load with get(fn), it runs for about 1/2 hour and stops with error message: E:\Euphoria\include\get.e:265 in function Get() Your program has run out of memory. One moment please ... I've tried breaking up the sequence and saving to files separately, then loading back in separately. I can then load each sequence on it's own, but when loading all in & appending to re-create the one sequence I get the same error. I'm using a PII 400 w/226 meg ram - 180 meg free before running the code. Is there some limit to total number of elements allowed in a sequence? Or should I be configuring my system differently ? Any help muchly appreciated, as I've been stuck on this for a few days now. Michael.
6. Re: Out of memory
- Posted by "C. K. Lester" <cklester at yahoo.com> Jun 06, 2002
- 589 views
Could you be using a EUPHORIA database? It might be more robust in dealing with data of this quantity... > I'm having trouble loading a sequence from file. The sequence consists of > 32657 elements, each element having up to 100 sub-elements as well. Total > file size is 72 meg. When attempting to load with get(fn), it runs for > about 1/2 hour and stops with error message:
7. Re: Out of memory
- Posted by "Carl W." <euphoria at cyreksoft.yorks.com> Jun 06, 2002
- 610 views
flint at acay.com.au wrote: > I'm having trouble loading a sequence from file. The sequence consists of > 32657 elements, each element having up to 100 sub-elements as well. Total > file size is 72 meg. When attempting to load with get(fn), it runs for > about 1/2 hour and stops with [an] error message. > I'm using a PII 400 w/226 meg ram - 180 meg free before running the code. It all depends how big your individual sub-elements are, and how big *their* sub-elements are. If you're working with integer data, a Euphoria sequence will use 4 bytes for every integer, even if it's character data you're dealing with. That's a waste of 3 bytes per character. For most programs this isn't a problem, but it does tend to fill up memory 4 times as fast as you'd expect it to. Let's say you've got 72Mb of character data; That's going to take up 288Mb of space (excluding overheads) - and it becomes obvious why you're running out of memory. If you can't process your file a piece at a time, which might be a good idea given its size, you might want to specifically 'allocate()' some memory and handle the data outside of Euphoria's built-in data manipulation. > Is there some limit to total number of elements allowed in a sequence? I reckon you'd probably have trouble somewhere around sequence index '1073741823', but you'd need well over 4Gb of RAM before you hit that problem. If I'm wrong and sequence indices are limited by _atoms_ not integers, you'd need something like 16Pb (sixteen petabytes ~= 18 quadrillion bytes) of RAM before you have problems. So 'No.' :) > Or should I be configuring my system differently ? Handling your data as smaller pieces is probably the way to go here. Carl
8. Re: Out of memory
- Posted by Igor Kachan <kinz at peterlink.ru> Jun 06, 2002
- 606 views
Hi Michael, ---------- > Îò: flint at acay.com.au > Êîìó: EUforum <EUforum at topica.com> > Òåìà: Out of memory > Äàòà: ÷åòâåðã, Èþíü 06, 2002 15:35 > > Hi everyone, > > I'm having trouble loading a sequence from file. > The sequence consists of 32657 elements, > each element having up to 100 sub-elements as well. > Total file size is 72 meg. When attempting to load > with get(fn), it runs for > about 1/2 hour and stops with error message: > > E:\Euphoria\include\get.e:265 in function Get() > Your program has run out of memory. > One moment please ... > > I've tried breaking up the sequence and saving > to files separately, then loading back in separately. > I can then load each sequence on it's own, > but when loading all in & appending to re-create > the one sequence I get the same error. > > I'm using a PII 400 w/226 meg ram - 180 meg free > before running the code. Is there some limit to > total number of elements allowed in a sequence? > Or should I be configuring my system differently ? > > Any help muchly appreciated, as I've been stuck on > this for a few days now. > > Michael. Try please one more recipe. If you run EX.EXE then run it on plain DOS32, in MS DOS mode of your machine, not in DOS box on Windows. If you run on Windows then use yourPROGRAM.EXW and EXW.EXE. Good luck. Regards, Igor Kachan, kinz at peterlink.ru
9. Re: Out of memory
- Posted by a.tammer at hetnet.nl Jun 06, 2002
- 659 views
Anyone any idea, or IO.sys, for WinMe, to enable Real DOS32-bootup as well? If so please mail&attach to a.tammer at hetnet.nl EUrs, a@t
10. Re: Out of memory
- Posted by rforno at tutopia.com Jun 07, 2002
- 610 views
Igor: You are right, I know by experience, but probably the best approach for him is to change the problem structure. Not knowing what the problem is, it is difficult to say how to do that. Regards. ----- Original Message ----- From: "Igor Kachan" <kinz at peterlink.ru> To: "EUforum" <EUforum at topica.com> Subject: Re: Out of memory Hi Michael, ---------- > Îò: flint at acay.com.au > Êîìó: EUforum <EUforum at topica.com> > Òåìà: Out of memory > Äàòà: ÷åòâåðã, Èþíü 06, 2002 15:35 > > Hi everyone, > > I'm having trouble loading a sequence from file. > The sequence consists of 32657 elements, > each element having up to 100 sub-elements as well. > Total file size is 72 meg. When attempting to load > with get(fn), it runs for > about 1/2 hour and stops with error message: > > E:\Euphoria\include\get.e:265 in function Get() > Your program has run out of memory. > One moment please ... > > I've tried breaking up the sequence and saving > to files separately, then loading back in separately. > I can then load each sequence on it's own, > but when loading all in & appending to re-create > the one sequence I get the same error. > > I'm using a PII 400 w/226 meg ram - 180 meg free > before running the code. Is there some limit to > total number of elements allowed in a sequence? > Or should I be configuring my system differently ? > > Any help muchly appreciated, as I've been stuck on > this for a few days now. > > Michael. Try please one more recipe. If you run EX.EXE then run it on plain DOS32, in MS DOS mode of your machine, not in DOS box on Windows. If you run on Windows then use yourPROGRAM.EXW and EXW.EXE. Good luck. Regards, Igor Kachan, kinz at peterlink.ru