1. RE: 2.4 problems
>
> I hope you can narrow it down. I don't know of any other
> performance bugs like this in 2.4 beta, except for one that
> can only happen if you use a Euphoria-written .dll
> (reported by Euman, that bug has been fixed for the official release).
>
> Be careful that you are running 2.4 beta, not alpha.
>
Definitely beta. I can tell you it must take some pretty unique
circumstances to cause the problem. I'm using the compress() routine
with my Berkeley DB wrapper to convert Euphoria objects to bytes for
storage, just as in EDS. I made a database of over 1 GB in size and
every single record in there was compress()'ed with no problem (all with
2.4). However, none of those records are particularly large.
But the other day I found a need to store some records that in their
original form as Euphoria sequences run from a few hundred elements to
as many as 200,000 elements, each element being a 32-bit integer (most
being 31-bits or less, but not all). After storing a few dozen such
records (quickly) the program slows down to a crawl (suddenly, not a
gradual thing) once it gets to a particular record. At first I thought
it was the database system, but I found that the freeze up occurs in the
compress() call, before the db insert call occurs.Furthermore, it
doesn't seem to have anything to do with the particular record in
question because if I do them in a different order it still freezes up
after doing the same approxiamate amount of work, but freezes up on a
different record.
I checked for memory leaks and didn't find any, and my "RAM bar" shows
plenty of memory available. (And in fact the program is looping, using
the same variables over & over to build up the records and compress()
them before insertion into the db.)
Now, if I try to replicate the problem outside of my database program
(which is huge and complex), no luck so far....
2. RE: 2.4 problems
Andy Serpa wrote:
>
>
> > I hope you can narrow it down.
I have narrowed it down! I don't have a simple example yet (in code),
but I have found that the problem has to due with garbage collection and
the re-use of memory (I'm guessing).
Explanation:
My program was filling up several very large sequences, taking up many
MBs in RAM. There were 8 such large sequences, all created at the same
time, but with different contents. (Like 8 big hash buckets).
At this point I was taking the sequences one-at-a-time and creating
database records based on them, and inserting them into the database.
This process involves further splitting of each large sequence into
smaller records, which are compress()ed along the way. The important
thing was that after finishing with each of the "big 8", I would clear
it out by doing this:
seq1 = {}
and then move on to the next one (seq2).
Once it got to the middle of the 4th big sequence and started is when
the big slow-down would occur.
What I have now discovered is that if I comment out all the
seq1 = {}
seq2 = {}
etc.
lines the program zips along quite speedily to the end with no problem.
I knowing nothing about the EU internals, but this leads to the obvious
guess:
The seq1 = {} etc lines are making big chunks of RAM again available to
the program, but somewhere along the line in the garbage collection
and/or re-use of that memory, the process becomes incredibly slow. So
re-allocation of memory is where I'd be looking for problems.
That help?
3. RE: 2.4 problems
Andy Serpa wrote:
>
>
> Andy Serpa wrote:
> >
> >
> > > I hope you can narrow it down.
>
> I have narrowed it down! I don't have a simple example yet (in code)
Now I do. The following rather strange program seems to do the trick:
---------------------------------------------------------------------
include machine.e
function seq_transform(sequence s)
sequence snew
snew = {}
for i = 1 to length(s) do
if rand(2) = 1 then
snew &= int_to_bytes(s[i])
else
snew &= atom_to_float64(s[i])
end if
end for
return snew
end function
object t, x, s1, s2, s3, s4
s1 = {}
s2 = {}
s3 = {}
s4 = {}
puts(1,"Creating big sequences...\n")
for i = 1 to 500000 do
s1 &= rand(1073741823) * 100
s2 &= rand(1073741823) * 100
s3 &= rand(1073741823) * 100
s4 &= rand(1073741823) * 100
end for
puts(1,"Processing sequences...\n")
t = time()
for i = 1 to 50 do
x = seq_transform(s1[rand(10000)..20000])
end for
? time() - t
--s1 = {}
t = time()
for i = 1 to 50 do
-- uncomment the following line to see just how slow
-- ? i
x = seq_transform(s2[rand(10000)..20000])
end for
? time() - t
--s2 = {}
t = time()
for i = 1 to 50 do
x = seq_transform(s3[rand(10000)..20000])
end for
? time() - t
--s3 = {}
t = time()
for i = 1 to 50 do
x = seq_transform(s4[rand(10000)..20000])
end for
? time() - t
--s4 = {}
puts(1,"\n\nDone.")
t = gets(0)
---------------------------------------------------------------------
The above is the fast version. Timings are about 1.4 seconds per
sequence on my system. Now comment out the "s1 = {}" etc lines and it
grinds to a near halt when processing the second sequence. (Uncomment
the "? i" to see it get slower & slower.)
Other things to notice:
-- The "seq_transform" function seems to be important. (This is taking
the place of compress() for purposes of this example.) If you alter
this function so it uses "int_to_bytes" only or "atom_to_float64" only
instead of mixing & matching the two, the slow-down does not occur.
-- The slow-down also does not occur if you keep the integers in the
31-bit range. Take out the " * 100 "'s and that becomes evident.
4. RE: 2.4 problems
> Now comment out the "s1 = {}" etc lines.
Correction: comment "in" those lines to slow it down...
5. RE: 2.4 problems
Pete Lomax wrote:
>
>
> On Mon, 9 Jun 2003 16:18:42 +0000, Andy Serpa <ac at onehorseshy.com>
> wrote:
>
> <snip>
> >for i = 1 to 500000 do
> ^ i had to kill a zero on that ;-( {P233, 48MB, Eu 2.4 beta}
> >--s1 = {}
> ^^^ bizarre!!!
> Correct me if I'm wrong, but that is a top-level statement executed
> exactly ONCE (when uncommented 
)
> Bizzare, because variable s1 is not referenced *at all* after that
> point (I'm measuring just up to the --s2 = {} line, no further).
>
> Andy, Rob, just that *one* line uncomment made it run like a dog (TEN
> times slower: 130 seconds vs 13 seconds) on my PC.
>
> Bizzare, because code *NOT* referencing that variable, like I just
> said, runs ten times slower!!
>
> Andy: I think you have definitely found something 
.
>
Yeah, I think it is because the memory that s1 was using is now
available for Eu to use again, and the problem is in the re-use of the
memory, not the variable. We'll see what Rob finds out.
My system is 256MB Windows ME, by the way...
6. RE: 2.4 problems
Robert Craig wrote:
>
>
> Andy Serpa wrote:
> > The following rather strange program seems to do the trick:
>
> Thanks for the example.
> It runs fine on my 256MB XP system,
> but on my 64Mb ME system it stalls for an
> extremely long time on: s3 = {}
>
> I'll try to figure out what's happening.
> Probably the storage allocation/deallocation
> algorithm needs to be tweaked to handle this
> extreme case.
>
Oh, and if it wasn't clear, on my system the hangup occurs during the x
= seq_transform() loop -- each iteration gets slower & slower (after
about 3 it is basically stopped for a very very long time).
The "deallocation" lines themselves "s1 = {}" etc don't take any time...
