RE: 2.4 problems
- Posted by Andy Serpa <ac at onehorseshy.com> Jun 09, 2003
- 410 views
Andy Serpa wrote: > > > > I hope you can narrow it down. I have narrowed it down! I don't have a simple example yet (in code), but I have found that the problem has to due with garbage collection and the re-use of memory (I'm guessing). Explanation: My program was filling up several very large sequences, taking up many MBs in RAM. There were 8 such large sequences, all created at the same time, but with different contents. (Like 8 big hash buckets). At this point I was taking the sequences one-at-a-time and creating database records based on them, and inserting them into the database. This process involves further splitting of each large sequence into smaller records, which are compress()ed along the way. The important thing was that after finishing with each of the "big 8", I would clear it out by doing this: seq1 = {} and then move on to the next one (seq2). Once it got to the middle of the 4th big sequence and started is when the big slow-down would occur. What I have now discovered is that if I comment out all the seq1 = {} seq2 = {} etc. lines the program zips along quite speedily to the end with no problem. I knowing nothing about the EU internals, but this leads to the obvious guess: The seq1 = {} etc lines are making big chunks of RAM again available to the program, but somewhere along the line in the garbage collection and/or re-use of that memory, the process becomes incredibly slow. So re-allocation of memory is where I'd be looking for problems. That help?