1. Large memory utilization
- Posted by Paul Missman <missmanp at ADELPHIA.NET>
Jul 28, 1998
-
Last edited Jul 29, 1998
I'm putting together a P-II machine with 512 megs of memory
for number crunching with large Euphoria sequences. I'm
using "boot to command prompt" mode of Win98, and mapping the
VM to a large partition with the causeway setting.
Is there anything else I need to do to take advantage of the
full 512 meg space for a Euphoria sequence (as much as is
available after DOS and Euphoria take their share)? Do I
need any extra emm386 commands, etc?
I've noticed that there is a hard limit on the VM at 1GB. Is
this imposed by the P-II, or is it a holdover from PMMX days?
If it is a holdover, could it be enlarged for PPro/P-II
platforms?
Thanks in advance,
Paul Missman
2. Re: Large memory utilization
>Date: Tue, 28 Jul 1998 22:01:26 -0400
>Reply-To: Euphoria Programming for MS-DOS
<EUPHORIA at LISTSERV.MUOHIO.EDU>
>From: Paul Missman <missmanp at ADELPHIA.NET>
>Subject: Large memory utilization
>To: EUPHORIA at LISTSERV.MUOHIO.EDU
>
>I'm putting together a P-II machine with 512 megs of memory
>for number crunching with large Euphoria sequences. I'm
>using "boot to command prompt" mode of Win98, and mapping the
>VM to a large partition with the causeway setting.
>
>Is there anything else I need to do to take advantage of the
>full 512 meg space for a Euphoria sequence (as much as is
>available after DOS and Euphoria take their share)? Do I
>need any extra emm386 commands, etc?
>
>I've noticed that there is a hard limit on the VM at 1GB. Is
>this imposed by the P-II, or is it a holdover from PMMX days?
>If it is a holdover, could it be enlarged for PPro/P-II
>platforms?
>
>Thanks in advance,
>
>Paul Missman
>
Just wanna make sure we're on the same subject here: P-II, 512MB RAM? Go
run a server machine with that, and still have room! hmm... fill up
500,000,000 bytes of sequences? ... somewhat impossible, in my view.
hehe,
- "LEVIATHAN"
______________________________________________________
Get Your Private, Free Email at http://www.hotmail.com
3. Re: Large memory utilization
Yes, we are reading from the same page and on the same
planet. :o) I'm really gonna put 512 megs in this
machine, and I'd like to have an 8 Gig virtual address
space. Will Euphoria access all the 512 megs? Is there
anything that can be done about the 1 Gig VM limit, or is
that the VM max on a P-II?
Can somebody help me out?
Thanks,
Paul Missman
4. Re: Large memory utilization
Paul Missman wrote:
>
> Yes, we are reading from the same page and on the same
> planet. :o) I'm really gonna put 512 megs in this
> machine, and I'd like to have an 8 Gig virtual address
> space. Will Euphoria access all the 512 megs? Is there
> anything that can be done about the 1 Gig VM limit, or is
> that the VM max on a P-II?
>
> Can somebody help me out?
Everybody's asleep who knows anything, so I guess I will try to help ;)
According to CauseWay's web site:
http://www.devoresoftware.com/cwwmain.htm
CauseWay uses up to 4 GIG memory.
I'm sure Rob will have more info on this.
On the other hand, send me a GIG or two, and I'll be happy to test it.
Regards,
Irv
http://www.mindspring.com/~mountains -- Euphoria
programs/links/soon-to-be FAQ
5. Re: Large memory utilization
Paul Missman writes:
> Yes, we are reading from the same page and on the same
> planet. :o) I'm really gonna put 512 megs in this
> machine, and I'd like to have an 8 Gig virtual address
> space.
> Will Euphoria access all the 512 megs?
The CauseWay DOS extender claims to support up to
4G - which is full 32-bit addressing, but I don't know
of anyone who has tried to go anywhere near 512 megs.
You are on the "bleeding edge" 
If, for some reason, ex doesn't work, you might try exw
which will let you access as much as any other
32-bit Windows program, and it doesn't use CauseWay.
> Is there anything that can be done about the 1 Gig VM limit,
> or is that the VM max on a P-II?
I don't know. 8G sounds tricky, since you'd need more than
32-bits for addressing. The Pentium can theoretically
address with 48 bits (16-bit segment and 32-bit offset),
but internally Euphoria uses 32-bits for almost all pointers.
Keep in mind that, depending on the access pattern,
virtual memory swapping can be painfully slow. It works
well when you have locality of reference to your data, i.e. you
work intensively with a small portion of your data for quite
a while, before moving on. It works badly when you
sweep through huge sequences from beginning to end,
performing a trivial operation on each element.
With that much data, you should also seriously look at
ways of compressing the data.
Regards,
Rob Craig
Rapid Deployment Software
http://members.aol.com/FilesEu/
6. Re: Large memory utilization
Robert Craig wrote:
>
> Paul Missman writes:
> > Yes, we are reading from the same page and on the same
> > planet. :o) I'm really gonna put 512 megs in this
> > machine, and I'd like to have an 8 Gig virtual address
> > space.
> > Will Euphoria access all the 512 megs?
>
> The CauseWay DOS extender claims to support up to
> 4G - which is full 32-bit addressing, but I don't know
> of anyone who has tried to go anywhere near 512 megs.
> You are on the "bleeding edge"
>
When I first got Euphoria, I initialized a single sequence of
300,000,000 bytes,
just to see if it could be done (also because I had a drive with 500
megs free).
It worked, but not all that fast - disk swapping and all - but it did
work.
Irv
7. Re: Large memory utilization
Thanks for the feedback.
I had tried initializing sequences over 250 meg elements (1GB)
using the repeat routine, and gotten the "out of memory" message.
On the swapping versus small section usage, yes I had noticed
that effect. I was planning on using 100 million element sections
of a 2 billion element array at a time. I can't compress the data
in any way which won't horribly slow my calculations, as it is
binary data, and contains every possible element.
Without access to the 2 billion element array (8GB), I think I can
still get the job done reasonably well by faking the lower bound
address using offsets. I can get away with this since I only need
to access a contiguous 100 million elements at any one time.
I'll try the exw with a test program, and see if I can initialize
more than 1 GB.
Thanks again,
Paul Missman
8. Re: Large memory utilization
-----Original Message-----
De: Paul Missman <missmanp at ADELPHIA.NET>
Para: EUPHORIA at cwisserver1.mcs.muohio.edu
<EUPHORIA at cwisserver1.mcs.muohio.edu>
Fecha: Miércoles 29 de Julio de 1998 05:54 PM
Asunto: Re: Large memory utilization
>Thanks for the feedback.
>
>I had tried initializing sequences over 250 meg elements (1GB)
>using the repeat routine, and gotten the "out of memory" message.
Just an idea:
Poke your data directly to memory and read (peek) into sequences
the currently working data (array as you said). I don't know if you can
allocate a 4 Gig chunck at once but there's ceratinly a way to do it
partialy. If you have to allocate several partial chunks you will need
a sequence conatince the "range" or "index" of the data stored in
each chunck. I can guarantee that that auxiliar sequence will be,
relativly speaking, small.
BTW The DOS version detection routine posted by Irv is 100% Jacques
work. I don't have Win98, someone can tell if it's DOS 8?
Regards,
Daniel Berstein
daber at pair.com
9. Re: Large memory utilization
- Posted by Paul Missman <missmanp at ADELPHIA.NET>
Jul 30, 1998
-
Last edited Jul 31, 1998
As a followup:
Using EXW under Win98, I can get 2GB of VM, so can
allocate a 500 million element Euphoria sequence.
Windows won't make a swapfile bigger than 2GB.
This doubles the amount of VM I could access with
the DOS extender under Euphoria. It also lets me
run the system monitor while the program is running
so I can see the VM getting eaten up by the program.
Thanks for the suggestions,
Paul Missman
10. Re: Large memory utilization
- Posted by Daniel Berstein <daber at PAIR.COM>
Jul 30, 1998
-
Last edited Jul 31, 1998
-----Original Message-----
De: Paul Missman <missmanp at ADELPHIA.NET>
Para: EUPHORIA at cwisserver1.mcs.muohio.edu
<EUPHORIA at cwisserver1.mcs.muohio.edu>
Fecha: Jueves 30 de Julio de 1998 11:47 PM
Asunto: Re: Large memory utilization
>As a followup:
>
>Using EXW under Win98, I can get 2GB of VM, so can
>allocate a 500 million element Euphoria sequence.
>Windows won't make a swapfile bigger than 2GB.
This should change with the arrival of Mercedes CPU (64-bit).
It's like an Alpha (Digital) clone by Intel ;)
How does NT behaves?
Regards,
Daniel Berstein
daber at pair.com
11. Re: Large memory utilization
- Posted by lithex <lithex at INTERGATE.BC.CA>
Jul 30, 1998
-
Last edited Jul 31, 1998
Hey Paul
I'm intrigued.
What are you doing that needs so much data space?
Bye
Martin
12. Re: Large memory utilization
- Posted by Arthur Adamson <euclid at ISOC.NET>
Aug 03, 1998
-
Last edited Aug 04, 1998
The following may or may not be pertinent to this discussion, but
just in case:
I recently posted a file savenat.zip to Robt's Official Euphoria
Page. It contains an include file which permits saving to disk a
one-dimensional large array of integers + zero with substantial saving in
time and space compared to print and get.
Arthur P. Adamson, The Engine Man, euclid at isoc.net
