1. 2.4 Official -- memory stuff
No tweak on the memory allocation? I've got a growing list of
"un-runnable" programs with 2.4. Is this just a Windows ME problem?
You said you had no trouble under XP?
I was just reading about HeapAlloc at MSDN. Here is a possible relevant
note:
----------
Windows Me/98/95: The heap managers are designed for memory blocks
smaller than four megabytes. If you expect your memory blocks to be
larger than one or two megabytes, you can avoid significant performance
degradation by using the VirtualAlloc or VirtualAllocEx function
instead.
----------
It also said that with XP/2000 there is a "low-fragmentation" heap
available...
2. Re: 2.4 Official -- memory stuff
Andy Serpa wrote:
> No tweak on the memory allocation? I've got a growing list of
> "un-runnable" programs with 2.4. Is this just a Windows ME problem?
> You said you had no trouble under XP?
I had no trouble on my 256Mb RAM Windows XP system.
In fact, your program ran quite fast.
I couldn't see any easy tweak to help you out on ME.
It was also very slow on my 64Mb ME system
(even after scaling things down).
Some things you can do:
- reduce the size of the sequences
- make more use of integers rather than floating-point
- create sequences at their final size, rather than
"growing them" with append or concatenation
In general, 2.4 is faster at allocating memory than 2.3.
However your program really mangles the system heap.
On Windows, 2.3 uses Watcom's heap manager, but
to make Euphoria .dll's work reliably I must use the
system heap, so that's what 2.4 does. The system heap
might be using an extra 4 bytes per floating-point number.
I'm not sure.
> I was just reading about HeapAlloc at MSDN. Here is a possible relevant
> note:
>
> ----------
> Windows Me/98/95: The heap managers are designed for memory blocks
> smaller than four megabytes. If you expect your memory blocks to be
> larger than one or two megabytes, you can avoid significant performance
> degradation by using the VirtualAlloc or VirtualAllocEx function
> instead.
Thanks. Yes, ME may not be well designed
for handling large blocks of memory.
> It also said that with XP/2000 there is a "low-fragmentation" heap
> available...
That might help, but it probably wastes memory.
Regards,
Rob Craig
Rapid Deployment Software
http://www.RapidEuphoria.com
3. Re: 2.4 Official -- memory stuff
Andy,
I believe this problem is insoluble in Eu 2.4 under Win 95/98/Me--the
problem is that heaps do not grow beyond their initial size and you can get
out of memory conditions when the heap is gone, even if you have plenty of
physical RAM--the OS can't allocate it. This issue does not occur in Win
NT/2000/XP --these can grow heaps beyond initial size.
I ran into the same problem in a VB 6.0 app on the job which dynamically
created a few hundred controls--always overflowed the heap on Win 98 ran
just fine on Win 2000.
By the way, per Microsoft the Win32 API has functions to dynamically grow
heaps, but they are not implemented in the 9x series (they are defined, but
they do not work), only in the NT series.
If upgrading to Win XP is not an option, I think you will have to do the
whole thing under 2.3 if there is no alternate way of accomplishing your
task--on the other hand, maybe there is a solution in Diamond that wouldn't
need so much saving and restoring. Bytecode is intended to implement
persistence--ordinarily, you would be saving to bytecode on disk before
shutdown and restoring from bytecode at startup--perhaps the program could
do its own memory management for holding the property values during the run.
Please correspond with me privately and I will attempt to help you get
something working under Eu 2.4.
-- Mike Nelson
----- Original Message -----
From: "Andy Serpa" <ac at onehorseshy.com>
To: "EUforum" <EUforum at topica.com>
Sent: Sunday, July 06, 2003 5:48 PM
Subject: RE: 2.4 Official -- memory stuff
>
>
> Robert Craig wrote:
> >
> >
> > Andy Serpa wrote:
> > > No tweak on the memory allocation? I've got a growing list of
> > > "un-runnable" programs with 2.4. Is this just a Windows ME problem?
> > > You said you had no trouble under XP?
> >
> > I had no trouble on my 256Mb RAM Windows XP system.
> > In fact, your program ran quite fast.
> >
> > I couldn't see any easy tweak to help you out on ME.
> > It was also very slow on my 64Mb ME system
> > (even after scaling things down).
> > Some things you can do:
> >
> > - reduce the size of the sequences
> > - make more use of integers rather than floating-point
> > - create sequences at their final size, rather than
> > "growing them" with append or concatenation
> >
> > In general, 2.4 is faster at allocating memory than 2.3.
> > However your program really mangles the system heap.
> > On Windows, 2.3 uses Watcom's heap manager, but
> > to make Euphoria .dll's work reliably I must use the
> > system heap, so that's what 2.4 does. The system heap
> > might be using an extra 4 bytes per floating-point number.
> > I'm not sure.
> >
>
> This has become a more general problem than I thought at first. I had
> to break up one of my programs so the "heavy-lifting" is done with a
> 2.3-translated program instead of 2.4 (other parts make use of the 2.4
> features which I don't want to give up -- like user-defined types being
> called w/ translator).
>
> For instance, here's a problem: I'm using Diamond, and the 2.3 part of
> the program creates a bunch about 150 Diamond entities. (This program
> takes 10x longer to run in 2.4 -- that's after rewriting the whole
> algorithm. Before it would never finish at all.) Each entity has one
> property which will hold a somewhat large sequence (each entity takes
> about 1/3 MB on disk after converting to byte code). The goal would be
> to hold all these entities in RAM at the same time. I've got plenty
> enough RAM to do that. Problem is, even if I create the entities in
> 2.3, I can't even load them in 2.4 (without it taking forever and a
> day)! That's because they're stored on disk as bytecode and after the
> bytecode for an entity is read from disk, the entity needs to be
> restored with the Diamond function restore_entity(). Apparently this
> transformation is enough to "mangle the heap" (unless there really is
> just an outright bug somewhere in 2.4) and if I try to load in the rest
> of the entities each one takes longer & longer. So now if I want to
> have access to all the entities I have to do something like store the
> bytecode for each one in an EDS database or something and never have
> more than one (or a few, anyway) "restored" at a time. As I said
> before, there seems to be a definite threshold for this -- if I load in
> one entity everything is fine. And if I destroy that entity before
> loading in the next one everything is fine. But if I load in 10 or so
> then that's all she wrote. So now I'm in the position where I would
> have to restore each entity *EACH* time I want to use it, which of
> course causes a different kind of slowdown -- what I really want is
> access to all of them at once. There is basically no way around it --
> such a program just doesn't work in 2.4 (on this platform) in any
> practical sense if I can't even get the objects in memory.
>
> Bottom line: it seems that with ME/98/95, any program which uses more
> than a little memory in any sort of flexible way is doomed. Since one
> of the main features of Euphoria is supposed to be flexible use of data,
> garbage collection, etc., the price seems rather high. I am effectively
> limited to programs which use no more than a few MBs of RAM on a machine
> which has 512 MB available! I can't be the only one who is going to
> have a problem with this...
>
>
>
> TOPICA - Start your own email discussion group. FREE!
>
>
4. Re: 2.4 Official -- memory stuff
- Posted by gertie at visionsix.com
Jul 07, 2003
On 6 Jul 2003, at 20:05, Mike Nelson wrote:
>
>
> Andy,
>
> I believe this problem is insoluble in Eu 2.4 under Win 95/98/Me--the
> problem is that heaps do not grow beyond their initial size and you can get
> out
> of memory conditions when the heap is gone, even if you have plenty of
> physical
> RAM--the OS can't allocate it. This issue does not occur in Win NT/2000/XP
> --these can grow heaps beyond initial size.
>
> I ran into the same problem in a VB 6.0 app on the job which dynamically
> created a few hundred controls--always overflowed the heap on Win 98 ran
> just fine on Win 2000.
>
> By the way, per Microsoft the Win32 API has functions to dynamically grow
> heaps, but they are not implemented in the 9x series (they are defined, but
> they
> do not work), only in the NT series.
If this is true, why not replace the heap handler DLLs in win9x with those
from 2k ?
Kat
5. Re: 2.4 Official -- memory stuff
Andy Serpa wrote:
> This has become a more general problem than I thought at first.
> ...
I still have the program you gave me before, but if you
have a new one, please post it (or a stripped down example)
or send it to me.
I'll do some experiments using my 64Mb ME machine.
Maybe I can quantify exactly what it is that
leads to the bad performance.
No one else has reported any performance problems
in 2.4 (alpha, beta or official),
but I did confirm terrible performance on my ME machine
when running your previous program, when the memory used
reached a certain level.
As I recall, your previous program allocated millions
of floating-point numbers in 4 sequences that each grew
from zero-length, one element at a time, until they were
500000 elements long. Each f.p. number needs about 24 bytes,
so, with other variables, I calculated your program would
need 50 Mb of RAM on your (originally) 256Mb machine, and that's
the best case, assuming not too many unused free blocks
in the (over 2 million block) heap.
Regards,
Rob Craig
Rapid Deployment Software
http://www.RapidEuphoria.com
6. Re: 2.4 Official -- memory stuff
On Sun, 06 Jul 2003 23:57:18 -0400, Robert Craig
<rds at RapidEuphoria.com> wrote:
>
>
>Andy Serpa wrote:
>> This has become a more general problem than I thought at first. =20
> > ...
>
>I still have the program you gave me before, but if you
>have a new one, please post it (or a stripped down example)
>or send it to me.
>
>I'll do some experiments using my 64Mb ME machine.
>Maybe I can quantify exactly what it is that
>leads to the bad performance.
I just tried this on 2.4 and compared it with 2.3 on the same kit.
I found that it works fine on 2.3 but not 2.4, and also that it
appears to be ok using ex rather than exw in both 2.3 and 2.4.=20
Dunno if that helps.
Pete
7. Re: 2.4 Official -- memory stuff
On Sun, Jul 06, 2003 at 08:05:13PM -0700, Mike Nelson wrote:
>
>
> Andy,
>
> I believe this problem is insoluble in Eu 2.4 under Win 95/98/Me--the
> problem is that heaps do not grow beyond their initial size and you can get
> out of memory conditions when the heap is gone, even if you have plenty of
> physical RAM--the OS can't allocate it. This issue does not occur in Win
> NT/2000/XP --these can grow heaps beyond initial size.
>
<snip>
Well, how does the Watcom heap manager do it then? If we figure that out then
you could just mimic or emulate that for 2.4/2.5.
jbrown
--
/"\ ASCII ribbon | http://www.geocities.com/jbrown1050/
\ / campain against | Linux User:190064
X HTML in e-mail and | Linux Machine:84163
/*\ news, and unneeded MIME | http://verify.stanford.edu/evote.html
8. Re: 2.4 Official -- memory stuff
On Sun, Jul 06, 2003 at 10:54:38PM -0500, gertie at visionsix.com wrote:
>
>
<snip>
> If this is true, why not replace the heap handler DLLs in win9x with those
> from 2k ?
>
> Kat
>
I don't believe NT dlls (especially low level system ones) are very stable
on 9x OSes.
Wouldn't hurt too bad to try tho ... worse case senerio you crash yer comp and
have to reinstall (but wouldnt you have to do that anyways? :)
jbrown
--
/"\ ASCII ribbon | http://www.geocities.com/jbrown1050/
\ / campain against | Linux User:190064
X HTML in e-mail and | Linux Machine:84163
/*\ news, and unneeded MIME | http://verify.stanford.edu/evote.html
9. Re: 2.4 Official -- memory stuff
On Mon, Jul 07, 2003 at 02:22:10AM -0500, gertie at visionsix.com wrote:
>
>
> On 7 Jul 2003, at 5:30, Andy Serpa wrote:
>
> >
> > > If this is true, why not replace the heap handler DLLs in win9x with
> > > those
> > > from 2k ?
> > >
> >
> > There are some third-party libraries for this sort of thing (SmartHeap
> > is one), but Rob would have to pay a license fee to distribute it with
> > Euphoria..
>
> How does this help win95, since no one can recompile any of the windows
> OSs?
>
> Kat
>
Just find a 3rd party heap manager that is written for Win95 I guess....failing
that you could try to find the source for one of those 3rd party libs and port
it or write your own. But then the intepreter/translator would have to use
those libs explicitly for heap management (instead of the WinNT/XP default
dlls).
jbrown
--
/"\ ASCII ribbon | http://www.geocities.com/jbrown1050/
\ / campain against | Linux User:190064
X HTML in e-mail and | Linux Machine:84163
/*\ news, and unneeded MIME | http://verify.stanford.edu/evote.html
10. Re: 2.4 Official -- memory stuff
- Posted by gertie at visionsix.com
Jul 07, 2003
On 7 Jul 2003, at 15:00, jbrown105 at speedymail.org wrote:
>
>
> On Sun, Jul 06, 2003 at 10:54:38PM -0500, gertie at visionsix.com wrote:
> >
> >
> <snip>
> > If this is true, why not replace the heap handler DLLs in win9x with those
> > from 2k ?
> >
> > Kat
> >
>
> I don't believe NT dlls (especially low level system ones) are very stable
> on 9x OSes.
>
> Wouldn't hurt too bad to try tho ... worse case senerio you crash yer comp and
> have to reinstall (but wouldnt you have to do that anyways? :)
I dunno, i have not reinstalled win95 since 1998, roughly 5 years ago.
Kat
11. Re: 2.4 Official -- memory stuff
- Posted by gertie at visionsix.com
Jul 07, 2003
On 7 Jul 2003, at 15:04, jbrown105 at speedymail.org wrote:
>
>
> On Mon, Jul 07, 2003 at 02:22:10AM -0500, gertie at visionsix.com wrote:
> >
> >
> > On 7 Jul 2003, at 5:30, Andy Serpa wrote:
> >
> > >
> > > > If this is true, why not replace the heap handler DLLs in win9x with
> > > > those
> > > > from 2k ?
> > > >
> > >
> > > There are some third-party libraries for this sort of thing (SmartHeap is
> > > one), but Rob would have to pay a license fee to distribute it with
> > > Euphoria..
> >
> > How does this help win95, since no one can recompile any of the windows
> > OSs?
> >
> > Kat
> >
>
> Just find a 3rd party heap manager that is written for Win95 I
> guess....failing
> that you could try to find the source for one of those 3rd party libs and port
> it or write your own. But then the intepreter/translator would have to use
> those
> libs explicitly for heap management (instead of the WinNT/XP default dlls).
that rather means the 3rd party heap manager would be a plug-in
replacement for the oem dll, or RDS is not going to support windows OSs
below WinNT/XP ?
Kat
12. Re: 2.4 Official -- memory stuff
On Mon, Jul 07, 2003 at 02:09:56PM -0500, gertie at visionsix.com wrote:
<snip>
> that rather means the 3rd party heap manager would be a plug-in
> replacement for the oem dll,
Yes. But only for 9x ... the NT version would use the builtin heap manager
most likely.
> or RDS is not going to support windows OSs
> below WinNT/XP ?
That seems a most unlikely senerio.
>
> Kat
>
jbrown
--
/"\ ASCII ribbon | http://www.geocities.com/jbrown1050/
\ / campain against | Linux User:190064
X HTML in e-mail and | Linux Machine:84163
/*\ news, and unneeded MIME | http://verify.stanford.edu/evote.html
13. Re: 2.4 Official -- memory stuff
Pete Lomax wrote:
> I just tried this on 2.4 and compared it with 2.3 on the same kit.
> I found that it works fine on 2.3 but not 2.4, and also that it
> appears to be ok using ex rather than exw in both 2.3 and 2.4.
> Dunno if that helps.
Euphoria 2.4 for Windows uses the system heap manager.
Euphoria 2.3 for Windows uses Watcom's heap manager (malloc/free).
This change was made to properly support
Euphoria-written .dlls. No changes were made in heap management
for DOS, Linux or FreeBSD. I know that in some
cases the new system heap manager is much faster than Watcom.
The Watcom algorithm, as I understand it, is slower at
allocating (possibly *much slower*), but faster at deallocating.
Today I tried a modified version of allsorts.ex to
sort sequences of various lengths up to several million
integers, or a few million floating-point numbers, using
my old 64Mb ME machine. I tested all the reasonably-fast
sorting algorithms with Euphoria 2.3 and 2.4.
In most cases the timing differences were small, but overall
2.4 was somewhat faster. Most of the fast algorithms
use recursive "divide and conquer" methods that
involve allocating and freeing numerous sequences
of various sizes.
I'll be interested to see Andy Serpa's latest program
so I can compare it to his earlier one. Maybe there's
a common ingredient that bogs down ME's heap manager.
XP (with the same memory) is fine.
Regards,
Rob Craig
Rapid Deployment Software
http://www.RapidEuphoria.com
14. Re: 2.4 Official -- memory stuff
--- Robert Craig <rds at RapidEuphoria.com> wrote:
<snip>
> - create sequences at their final size, rather than
> "growing them" with append or concatenation
Or, you can compromise. I often do this. Grow the sequence by concatenating
in chunks (of 100, 1000, or whatever makes sense for your purposes, or based on
some trial and error to figure out an optimal size). Use an index variable,
and another that tracks the length of the sequence, and whenever index=max, add
another chunk. It's a really nice way to get the benefits of preallocation
without wasting too much memory.
Matt Lewis
__________________________________
Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!
http://sbc.yahoo.com
15. Re: 2.4 Official -- memory stuff
----- Original Message -----
From: "Matt Lewis" <matthewwalkerlewis at yahoo.com>
To: "EUforum" <EUforum at topica.com>
Subject: Re: 2.4 Official -- memory stuff
>
>
> --- Robert Craig <rds at RapidEuphoria.com> wrote:
>
> <snip>
>
> > - create sequences at their final size, rather than
> > "growing them" with append or concatenation
>
> Or, you can compromise. I often do this. Grow the sequence by concatenating
> in chunks (of 100, 1000, or whatever makes sense for your purposes, or based
> on
> some trial and error to figure out an optimal size). Use an index variable,
> and another that tracks the length of the sequence, and whenever index=max,
> add
> another chunk. It's a really nice way to get the benefits of preallocation
> without wasting too much memory.
>
This technique is used in win32lib. In addition to reuse of elements rather than
removing them.
--
Derek
