1. RE: 2.4 Official -- memory stuff
Robert Craig wrote:
>
>
> Andy Serpa wrote:
> > No tweak on the memory allocation? I've got a growing list of
> > "un-runnable" programs with 2.4. Is this just a Windows ME problem?
> > You said you had no trouble under XP?
>
> I had no trouble on my 256Mb RAM Windows XP system.
> In fact, your program ran quite fast.
>
> I couldn't see any easy tweak to help you out on ME.
> It was also very slow on my 64Mb ME system
> (even after scaling things down).
> Some things you can do:
>
> - reduce the size of the sequences
> - make more use of integers rather than floating-point
> - create sequences at their final size, rather than
> "growing them" with append or concatenation
>
> In general, 2.4 is faster at allocating memory than 2.3.
> However your program really mangles the system heap.
> On Windows, 2.3 uses Watcom's heap manager, but
> to make Euphoria .dll's work reliably I must use the
> system heap, so that's what 2.4 does. The system heap
> might be using an extra 4 bytes per floating-point number.
> I'm not sure.
>
This has become a more general problem than I thought at first. I had
to break up one of my programs so the "heavy-lifting" is done with a
2.3-translated program instead of 2.4 (other parts make use of the 2.4
features which I don't want to give up -- like user-defined types being
called w/ translator).
For instance, here's a problem: I'm using Diamond, and the 2.3 part of
the program creates a bunch about 150 Diamond entities. (This program
takes 10x longer to run in 2.4 -- that's after rewriting the whole
algorithm. Before it would never finish at all.) Each entity has one
property which will hold a somewhat large sequence (each entity takes
about 1/3 MB on disk after converting to byte code). The goal would be
to hold all these entities in RAM at the same time. I've got plenty
enough RAM to do that. Problem is, even if I create the entities in
2.3, I can't even load them in 2.4 (without it taking forever and a
day)! That's because they're stored on disk as bytecode and after the
bytecode for an entity is read from disk, the entity needs to be
restored with the Diamond function restore_entity(). Apparently this
transformation is enough to "mangle the heap" (unless there really is
just an outright bug somewhere in 2.4) and if I try to load in the rest
of the entities each one takes longer & longer. So now if I want to
have access to all the entities I have to do something like store the
bytecode for each one in an EDS database or something and never have
more than one (or a few, anyway) "restored" at a time. As I said
before, there seems to be a definite threshold for this -- if I load in
one entity everything is fine. And if I destroy that entity before
loading in the next one everything is fine. But if I load in 10 or so
then that's all she wrote. So now I'm in the position where I would
have to restore each entity *EACH* time I want to use it, which of
course causes a different kind of slowdown -- what I really want is
access to all of them at once. There is basically no way around it --
such a program just doesn't work in 2.4 (on this platform) in any
practical sense if I can't even get the objects in memory.
Bottom line: it seems that with ME/98/95, any program which uses more
than a little memory in any sort of flexible way is doomed. Since one
of the main features of Euphoria is supposed to be flexible use of data,
garbage collection, etc., the price seems rather high. I am effectively
limited to programs which use no more than a few MBs of RAM on a machine
which has 512 MB available! I can't be the only one who is going to
have a problem with this...
2. RE: 2.4 Official -- memory stuff
>
> I believe this problem is insoluble in Eu 2.4 under Win 95/98/Me--the
> problem is that heaps do not grow beyond their initial size and you can
> get
> out of memory conditions when the heap is gone, even if you have plenty
> of
> physical RAM--the OS can't allocate it. This issue does not occur in Win
> NT/2000/XP --these can grow heaps beyond initial size.
>
> I ran into the same problem in a VB 6.0 app on the job which dynamically
> created a few hundred controls--always overflowed the heap on Win 98 ran
> just fine on Win 2000.
>
I was reading something about that, but it doesn't seem to be quite the
same problem. There is no problem filling up tons & tons of memory.
Works just fine. The problem is in the re-use of memory. The moment
you release a significant portion of memory and start filling it up
again is when the problems occur. And notice there is never outright
failure -- it just becomes REALLY slow (but no disk activity). For
instance, in the Diamond example of using restore_entity(), let's say
you load in a large saved entity and assign the byte code to "x":
x = get_entity_from_file("somefile.dat") -- mythical function
Now x is large, but is byte code, so you do:
x = restore_entity(x)
Now x is a Diamond instance handle. But in the process, what was
originally assigned to x is freed, and that's where the problems start
when you start repeating that process over & over. In other words, you
might do this:
x = get_entity_from_file()
y = restore_entity(x)
Since x was never freed, no slow-down occurs. That is the one constant
in all this -- as soon as you release memory, trouble begins. As soon
as you start making new sequences, they have trouble getting made. It
is clearly in the re-allocation, clean up, or deallocation where the
problem lies.
But obviously maintaining all memory from program start-up no matter
what is not exactly an ideal situation, and most often not possible.
Thanks for your offer to work something out regarding Diamond, but it
isn't really Diamond's problem. That was just one example. I have a
whole class of programs where this will be an issue. And yes, I may be
able to come up with clever ways when loading things in so that the
problem doesn't occur (as greatly) in one particular case, but with most
of stuff, recycling of memory is simply fundamental to the algorithm.
It is, as you say, insoluble. I don't want to be jumping through a
bunch of hoops just to load some stuff into memory.
Now then, I *am* planning on getting XP. However, XP (along with the
bits of new hardware I'll need to go with it) is going to cost me approx
$300, and that was money I was planning to spend a few months down the
line rather than this week, plus I'll have to put in a day or so just on
the upgrade process itself when I'd much rather just finish up the
project I'm working on.
The thing I want to make sure of is that it really is solely an OS
issue, and that there really isn't anything that can be done about it
Euphoria-wise (i.e. a bug). It sounds from what Rob is saying that he
is making an educated guess as to what is happening, but he's not quite
sure. Worse-case scenario is of course I spend the $300 to upgrade
everything and the problem still persists. That would royally suck!
3. RE: 2.4 Official -- memory stuff
>
> If this is true, why not replace the heap handler DLLs in win9x with
> those
> from 2k ?
>
There are some third-party libraries for this sort of thing (SmartHeap
is one), but Rob would have to pay a license fee to distribute it with
Euphoria..
4. RE: 2.4 Official -- memory stuff
Robert Craig wrote:
>
>
> Andy Serpa wrote:
> > This has become a more general problem than I thought at first.
> > ...
>
> I still have the program you gave me before, but if you
> have a new one, please post it (or a stripped down example)
> or send it to me.
>
> I'll do some experiments using my 64Mb ME machine.
> Maybe I can quantify exactly what it is that
> leads to the bad performance.
>
> No one else has reported any performance problems
> in 2.4 (alpha, beta or official),
> but I did confirm terrible performance on my ME machine
> when running your previous program, when the memory used
> reached a certain level.
>
> As I recall, your previous program allocated millions
> of floating-point numbers in 4 sequences that each grew
> from zero-length, one element at a time, until they were
> 500000 elements long. Each f.p. number needs about 24 bytes,
> so, with other variables, I calculated your program would
> need 50 Mb of RAM on your (originally) 256Mb machine, and that's
> the best case, assuming not too many unused free blocks
> in the (over 2 million block) heap.
>
That was an extreme example, and one in which the program for all
practically purposes STOPPED (although that's what happened in my real
program, so I guess it isn't that extreme).
I can provide you with a number of examples. The floating point thing
does seem to have something to do with it, but you don't have to have a
ton a f.p.'s, just a few. I'll send you some stuff...
5. RE: 2.4 Official -- memory stuff
>
> No one else has reported any performance problems
> in 2.4 (alpha, beta or official),
> but I did confirm terrible performance on my ME machine
> when running your previous program, when the memory used
> reached a certain level.
>
Also meant to say that in most cases people may not notice this
happening if they aren't doing something that iterates too many times.
It gets progressively slower, but the severity depends on just what
you're doing. If you only iterate something 10 times and it takes 10%
longer each time but only takes a microsecond to begin with, you're not
gonna notice. Iterate it 10000 times and you'll find yourself waiting
around wondering what's taking so long.
When translated, I notice the Borland-translated programs are able to
last longer before they really bog down....
6. RE: 2.4 Official -- memory stuff
- Posted by gertie at visionsix.com
Jul 07, 2003
On 7 Jul 2003, at 5:30, Andy Serpa wrote:
>
>
> > If this is true, why not replace the heap handler DLLs in win9x with
> > those
> > from 2k ?
> >
>
> There are some third-party libraries for this sort of thing (SmartHeap
> is one), but Rob would have to pay a license fee to distribute it with
> Euphoria..
How does this help win95, since no one can recompile any of the windows
OSs?
Kat
7. RE: 2.4 Official -- memory stuff
>
> I'll be interested to see Andy Serpa's latest program
> so I can compare it to his earlier one. Maybe there's
> a common ingredient that bogs down ME's heap manager.
> XP (with the same memory) is fine.
>
I am busy for a day or two here. I should have something on Wednesday
or Thursday. Maybe I can send you one of my actual programs but they
have many dependencies in my personal library so I'll have to sort that
out.
There are two common ingredigrents off the top of my head:
-- "something big" (over some unknown threshold) needs to be freed at
some point
-- and then, new sequences need to be created, probably with at least
some nesting or floating point numbers (or mixing of different types)
In all my programs where this occurs, there is some sort of
transformation going on. Sorting is fine, copying or building up
sequences from existing elements is fine, etc. And the slowness seems
to be occuring in those parts of the program that are (re)allocating
memory, i.e. making sequences (specifically, "growing" them as my
algorithms cannot predict final size or shape needed at the beginning).
All other operations are speedy as usual. 2.4 always starts out faster
than the equivalent in 2.3, but 2.3 stays at a steady rate (after 1st
iteration, which is slowest) while 2.4 gets somewhat slower every single
iteration. 2.4 actually uses slightly LESS memory than 2.3 according to
my RAM indicator, but it jumps up & down while 2.3/Watcom the free
memory only goes down, never up until program exits.
With any luck, it is just some quirk that can be isolated and dealt
with. I seem to have an uncanny knack for uncovering these things...
