1. Memory trouble update
Well,
Because of all the problems I was having with Euphoria and other things,
I went ahead and got XP Pro a few days ago. I was simply being too
unproductive. Anyway, all problems are gone! Everything is running
faster than ever! (I think I also traced those "mystery crashes" to
improper latency settings for my RAM.)
So while I do still have another system here still running Windows ME, I
don't get much chance to use it and won't be able to spend any time
tracking down this problem anymore. But I think I figured out the gist
of it anyway, and using these guidelines anyone should be able to create
slow performance at will with Eu 2.4 running Win ME (& 98, etc I
assume).
My original artificial example was this:
--------------------------------------------
include machine.e
function seq_transform(sequence s)
sequence snew
snew = {}
for i = 1 to length(s) do
if rand(2) = 1 then
snew &= int_to_bytes(s[i])
else
snew &= atom_to_float64(s[i])
end if
end for
return snew
end function
object x, s1, s2, s3, s4
s1 = {}
s2 = {}
s3 = {}
s4 = {}
puts(1,"Creating big sequences...\n")
for i = 1 to 50000 do
s1 &= rand(1073741823) * 100
s2 &= rand(1073741823) * 100
s3 &= rand(1073741823) * 100
s4 &= rand(1073741823) * 100
end for
puts(1,"Processing sequences...\n")
t = time()
for i = 1 to 50 do
x = seq_transform(s1[rand(10000)..20000])
end for
? time() - t
s1 = {}
t = time()
for i = 1 to 50 do
-- uncomment the following line to see just how slow
-- ? i
x = seq_transform(s2[rand(10000)..20000])
end for
? time() - t
s2 = {}
t = time()
for i = 1 to 50 do
x = seq_transform(s3[rand(10000)..20000])
end for
? time() - t
s3 = {}
t = time()
for i = 1 to 50 do
x = seq_transform(s4[rand(10000)..20000])
end for
? time() - t
s4 = {}
puts(1,"\n\nDone.")
t = gets(0)
---------------------------
Now Rob said I was basically fragmenting the heap with this, but what
failed to hit me is just how specifically I was doing that:
for i = 1 to 50000 do
s1 &= rand(1073741823) * 100
s2 &= rand(1073741823) * 100
s3 &= rand(1073741823) * 100
s4 &= rand(1073741823) * 100
end for
It has much less to do with what you do after you free up one of the
sequences then how you built them up in the first place. The really
important part here is not just that they are large, or contain part
integers & part floats (integers outside integer range), but that they
are "grown" in parallel. It adds a little bit to one sequence, then a
little bit to the next, etc. This has the effect (I assume) of breaking
up the memory used for each into lots of non-adjacent bits. So if you
do this instead:
for i = 1 to 50000 do
s1 &= rand(1073741823) * 100
end for
for i = 1 to 50000 do
s2 &= rand(1073741823) * 100
end for
for i = 1 to 50000 do
s3 &= rand(1073741823) * 100
end for
for i = 1 to 50000 do
s4 &= rand(1073741823) * 100
end for
Then the program runs fine because when each sequence is later freed
("s1 = {}") it is freeing (I assume) a big more or less contiguous
block. (Unfortunately, in real programs this is not always possible or
would create a new kind of slowdown.) But when you build them up in
parallel, and then free one of them, you are creating a ton of little
"holes" in memory, and then when you allocate some more it seems to take
forever for the system to do that. The other important thing to note is
that if you build up something again (after freeing a big fragmented
sequence) that is the same shape and size RAM-wise as the freed
sequence, it will work ok. It is when allocating chunks of memory that
don't fit into the "holes" well when the problems occur. I don't think
it has too much to do with integers vs. floats per se, only that they
require a different number of bytes to store.
I did some experiments where after I created fragmentation, I built up
some new sequences using only integers. Worked fine, but if I made it
so that it would add a slice of random length (say 1 to 10 elements of
integers), then the slowdown would occur.
So, to create bad performance, do the following:
-- "Grow" incrementally several large sequences *in parallel*.
-- Free one of them.
-- Grow some new sequences of a different shape & size (ram-wise, mixing
elements requiring different storage needs) than the one you freed.
-- You should get diminished performance building up the new sequences.
To further diminish, iterate the process.
With that, I leave this problem for Rob to deal with, if desired...
