OpenEuphoria

2. Re: memory use increasing while running, till...

• Posted by useless_ Dec 27, 2012
• 1425 views

If Eu doesn't return memory to the OS while it's running, then how is the reported memory use falling drastically while it's running? Is windoze caching the writes and reporting that cache as part of Eu?

useless

4. Re: memory use increasing while running, till...

• Posted by useless_ Dec 27, 2012
• 1420 views

And it just reset:



If Eu does not give memory back to the OS, how is this happening?

useless

6. Re: memory use increasing while running, till...

• Posted by mattlewis (admin) Dec 27, 2012
• 1372 views

On Windows, some memory allocations of sequences are cached, but there are limits. We only keep up to 2000 allocations cached, and each allocation must be 1024 bytes or less. Memory for atoms stored as doubles is allocated in chunks, and I don't think those chunks are ever actually freed back to the OS.

Matt

8. Re: memory use increasing while running, till...

• Posted by useless_ Dec 28, 2012
• 1358 views

I got the app down to a demo of the bug, it's 8 lines, 184 bytes of Eu program, munges a 2-char sequence, and eats 30 megabytes of ram per second doing it.

If i had a clue anyone on Euphorum cared, i might post the code demonstrating the bug.

useless

10. Re: memory use increasing while running, till...

• Posted by CoJaBo2 Dec 28, 2012
• 1332 views

Frankly, it just seems rude and selfish to play "hide and seek" with testcases just because nobody immediately jumps at the chance to help you for free.

Nobody here monitors the forums as their full-time job. Its likely the people who could help had not even seen that code.

12. Re: memory use increasing while running, till...

• Posted by DerekParnell (admin) Dec 28, 2012
• 1294 views

I don't hang about in #euphoria (or anywhere in chat actually), so I didn't see the discussion or the demo code.

I doubt that.

Have you decided that only people in the chat room could help? Are you sure that no-one else can, for example me? We do live in different time zones and I do have other things to do beside monitor forums etc ...

14. Re: memory use increasing while running, till...

• Posted by DerekParnell (admin) Dec 28, 2012
• 1284 views

That's not how I saw it.

I'm making an assumption that the bug is not desired and that you'd like it to be resolved. Thus I kinda thought that you were trying to be part of the solution to help remove the mistake in Euphoria's implementation. I too, was attempting to be part of that solution. By submitting your demo code, I thought you were trying to help someone else get to the next stage of diagnosis or resolution, so it's not so much that you were personally asking for help with your application, but more that you were asking for help from whom ever to get Euphoria fixed.

16. Re: memory use increasing while running, till...

• Posted by useless_ Dec 28, 2012
• 1256 views

I am not sure you want to know of the bug. No one has asked for the code to demo the bug. The way i see it, i am a pita if i do or if i don't post it. I don't know what to do. So i am looking for a very narrow set of words in the right order that, to me, will absolve me of blame if i post the code. Cojabo encouraged me to post it, but i see that as Jeremy encouraging me to be on the dev list: it ends with me siting in a corner under a blanket.

useless

18. Re: memory use increasing while running, till...

• Posted by DerekParnell (admin) Dec 28, 2012
• 1240 views

I don't understand - blame for what? Finding a bug is not something to be blamed for.

Anyhow, I've got your demo code now and I have narrowed down the scope of where to look.

If you remove the call to reverse(), the problem doesn't happen. But it's not that simple. If one leaves the call to reverse() in, but changes match_replace() like this ...

--haystack = replace(haystack, replacement, posn, posn + needle_len) 
haystack = haystack[1 .. posn-1] & replacement & haystack[posn + needle_len + 1 .. $] 

the problem goes away too.

So, the problem lies with the interaction of the reverse() and replace() calls. Without looking at the IL code yet, it looks like a dereference is being skipped for some reason.

Here is Kat's demo code that she emailed to me (sorry Kat if you wanted this code to be kept a secret but you're email didn't specify that)...

-- quickly eats up all available ram -- 
  
include std/sequence.e -- reverse 
include std/search.e -- match_replace 
  
sequence data1, data2, data3 
  
data1 = "<_" 
  
while 1 do 
  
 data2 = reverse(data1) 
  
 data3 = match_replace('_',data2,' ') -- underscore,data2,space 
  
end while 

20. Re: memory use increasing while running, till...

• Posted by mattlewis (admin) Dec 28, 2012
• 1208 views

The problem manifests translated, too. The memory being leaked is allocated inside of sequence.e reverse:

 
	for lowr = pFrom to lLimit do 
		t[uppr] = target[lowr] -- << this line! 
		t[lowr] = target[uppr] 
		uppr -= 1 
	end for 

It happens when we copy the sequence due to a non-unique reference count. From a valgrind session on a translated version:

==4997== 7,543,928 bytes in 134,713 blocks are definitely lost in loss record 192 of 192 
==4997==    at 0x4C2B3F8: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) 
==4997==    by 0x40E44C: EMalloc (be_alloc.c:1065) 
==4997==    by 0x40DEAE: NewS1 (be_alloc.c:830) 
==4997==    by 0x40E0AD: SequenceCopy (be_alloc.c:894) 
==4997==    by 0x405494: _2reverse (sequence.c:121) 
==4997==    by 0x40482F: main (main-.c:310) 

Next step is to figure out why it's not being freed...

Matt

22. Re: memory use increasing while running, till...

• Posted by mattlewis (admin) Dec 28, 2012
• 1190 views

No, it's definitely a problem with replace(). See ticket:830.

Matt

24. Re: memory use increasing while running, till...

• Posted by mattlewis (admin) Dec 29, 2012
• 1131 views

It's replace() that leaks the memory allocated by that line. The solution isn't obvious to me, so it's going to take some work to figure it out (reference counting bugs usually do).

I'd say that there probably are more leaks and bugs in the code, since we keep finding them. This bug has been there since 4.0.0.

Matt

26. Re: memory use increasing while running, till...

• Posted by useless_ Dec 29, 2012
• 1121 views

That doesn't sound right, that memory used in one procedure still exists to be used by another, when that second procedure may never be called. IIRC, i ran reverse() alone in the while loop and it wasn't a memory hog by itself, ditto running match_replace() by itself.

useless

28. Re: memory use increasing while running, till...

• Posted by mattlewis (admin) Dec 29, 2012
• 1092 views

The memory I'm referring to is the memory that's used to store a sequence. So, to your euphoria code, it's just a sequence. To the back end, it's a chunk of memory that's been allocated, to be freed up when the sequence's reference count drops to zero.

The implementation of replace() is somewhat complex, based on the data being replaced and the data replacing the old data. Add to that the complications of doing things in place vs making copies, and things get very complex. So basically, you've found a case where we're not handling the reference counts correctly in the bowels of replace().

You can find the implementation in be_runtime.c:Replace().

Matt