OpenEuphoria

1. Memory usage on Win32: Euphoria usage vs. what Task Manager displays

• Posted by CladStone Jul 25, 2009
• 1124 views

Maybe I'm just worried/curious about stuff that doesn't matter, but in the interest of "running a tight ship", I'm trying to figure out why what I see in Windows Task Manager doesn't seem to match up with what I would expect, so I'm wondering if someone could explain to me some basics of how Euphoria uses and releases memory, and how I can "force" it to take it or give it back, etc.

I realize that part of the issue may have to do with the way Task Manager reports memory usage, so any insights on that would help too.

A few examples:

1) I have some code where I explicitly allocate 32 MB of memory, but it doesn't show up in Task Manager. Does Task Manager not show you allocated memory, but rather only what is actually being used of that allocated memory?

2) I have another bit of code that loads a file and does some work with it. When I'm done, I want to be able to free up the extra space, so I say: mybigvar = {}. The first such statement produces NO effect whatsoever in Task Manager. The second such statement (e.g. myotherbigvar = {}) produces a small decrease (a couple hundred KB) which is well under the size of either of these variables.

3) Same example as #2, but with memory, not sequences. The file data itself was in a memory buffer, which I free, but doing so produces NO corresponding effect in Task Manager, though I'd expect to see a drop of about 500 KB.

3. Re: Memory usage on Win32: Euphoria usage vs. what Task Manager displays

• Posted by LarryMiller Jul 25, 2009
• 1089 views

I should add that internal memory management in Windows is VERY complex. The principles are not well known, even among computer professionals. I am by no means an expert, but I'm trying.

The Task Manager display in Windows can be difficult to interpret. Many of the numbers do not mean what is commonly believed. To really understand them you must understand how the system works, and that knowledge is not easily obtained.

5. Re: Memory usage on Win32: Euphoria usage vs. what Task Manager displays

• Posted by CoJaBo Jul 25, 2009
• 1112 views

Well I'm no expert on Windows memory allocation either, but it is likely that the Eu interpreter or maybe Windows itself is holding on to that memory for possible future use. If that is the case, I would expect that if you fill some other var with a lot of data it would reuse that memory, and you would see little or no increase in the VM size.

7. Re: Memory usage on Win32: Euphoria usage vs. what Task Manager displays

• Posted by AndySerpa Jul 30, 2009
• 994 views
• Last edited Feb 03, 2011

There also may be a bug in 3.1 Euphoria -- it definitely changed how it releases memory under Windows between 2.5 and 3.0 (or maybe it was 3.0 and 3.1, can't remember). It used to be you could use a huge chunk of memory processing something, and then release it by setting the variables to null (i.e. var = {}). This stopped working in v3.1. It then held on to the memory, and you could re-use it (without it growing further, so it wasn't a total memory leak), but if you weren't going to reuse, you could never get the application to give it up. I had a number of pre-3.1 programs that did some big processing like this when they started up, but then released it. I had to rewrite them for 3.1 and break them up into two programs chained together because they were long-running programs and they'd continue hogging system resources forever after they were finished with those resources. I used to be able to run several of these programs simulataneously, but with 3.1 it would practically crash the system.

I reported this several times, but couldn't get anyone to pay attention, only getting the response that memory handling hadn't changed and it was "complicated". But it can be easily shown running the same code that v2.5 will release the memory and v3.1 won't. Haven't tried v4.0 yet.

9. Re: Memory usage on Win32: Euphoria usage vs. what Task Manager displays

• Posted by mattlewis (admin) Jul 30, 2009
• 995 views

My recollection was that prior to 3.0, or so (I can't recall exactly, either), euphoria simply used the standard malloc() from watcom. This never released anything back to the OS. At this point, euphoria uses Windows' allocation routines. In both cases, euphoria uses an internal cache to speed up reuse of memory.

Matt

11. Re: Memory usage on Win32: Euphoria usage vs. what Task Manager displays

• Posted by AndySerpa Jul 30, 2009
• 1015 views
• Last edited Feb 03, 2011

Gee, it seems like the opposite of what you guys are saying. In version 2.5, I can fill up a giant sequence that uses a ton of memory, release it (var = {}), and then that memory is once again available to other apps -- you can see this with a "free RAM" type program or in the task manager. In 3.1 (3.1.1 under Windows XP is what I'm using) and that same program keeps and holds the memory forever, PREVENTING me from using another copy of the same program simultaneously. Like I said, I had to rewrite some programs I'd been using for years on v2.5 because I wanted to switch to v3.1. Those are real-world consequences. I'm not imagining it.

I posted a simple example of this on the old forum, which I will now recycle here. Let's check it again.

clear_screen() 
 
object x 
 
x = {} 
for i = 1 to 50 do -- adjust if it grabs too much memory for your system 
	? i 
	for j = 1 to 500 do 
		x &= rand(repeat(1000,1000))/rand(repeat(1000,1000)) 
	end for 
end for 
 
x = gets(0) 
? x 
x = gets(0) 

This program eats up a bunch of memory. Running it on v2.5, when it gets to 50, the task manager reports it using 684,188K. Now I hit enter (once), which replaces the entire "x" variable with a one-element sequence. You can actually watch the memory being released in blocks in the task manager all the way down to 4146K, basically nothing. It frees it all.

Ok, now same test with v3.1:

counter at 50: 684,164K

hit enter: 586,308K

About 90MB freed. Same exact results I got when I originally posted it. That memory is FREE in v2.5 for use in other apps. Not in v3.1.

In v3.1 it seems like it is releasing one block of memory only (which happens instantly). In v2.5 you can watch the memory being freed in stages; it takes 5 seconds or so to free it all.

However, like I said, it is not an outright leak. If you add a second memory-grabbing loop, like so:

x = {} 
for i = 1 to 50 do -- adjust if it grabs too much memory for your system 
	? i 
	for j = 1 to 500 do 
		x &= rand(repeat(1000,1000))/rand(repeat(1000,1000)) 
	end for 
end for 
 
x = {} 
for i = 1 to 50 do -- adjust if it grabs too much memory for your system 
	? i 
	for j = 1 to 500 do 
		x &= rand(repeat(1000,1000))/rand(repeat(1000,1000)) 
	end for 
end for 
 
x = gets(0) 
x = gets(0) 

... then the amount of used memory stays stable after the first loop. So it does manage to reuse that memory. It is at least available to itself.

Anyone care to report what happens in v4.0?

13. Re: Memory usage on Win32: Euphoria usage vs. what Task Manager displays

• Posted by AndySerpa Jul 30, 2009
• 999 views

Well, looks like I will have to try v4. Using an atom instead of an empty sequence doesn't help in v3.1. You say when you set to an atom AND there are no other references to the sequence. In what situation would there be another reference to the sequence? I know there would be another reference if there was another explicit reference (another variable) to it:

x = <big sequence> 
y = x 
x = 0 
 
-- y still references big sequence, so memory is not released 
 

But are there situations where I set all explicit references to atoms and there would still be an internal reference hanging around?

15. Re: Memory usage on Win32: Euphoria usage vs. what Task Manager displays

• Posted by AndySerpa Jul 30, 2009
• 1000 views
• Last edited Jul 31, 2009

sequence AA = <some huge value> 
 
libProc(AA) 
 
AA = 0 -- Frees RAM unless 'libProc' took a copy of it. 

Yes, good. I'm pretty careful where I go leaving my 100s-of-MB-large variables, so it should be fine.

Thanks for checking it out.

17. Re: Memory usage on Win32: Euphoria usage vs. what Task Manager displays

• Posted by DerekParnell (admin) Jul 31, 2009
• 991 views

I'm using Windows XP too.

I stand by my assertion that memory released by reducing the size a sequence is made available for reuse. If the amount of memory released is larger than the system page size, that memory can be reused by other applications.

I'm using two measurements to confirm this. Using the Windows system monitor, I'm looking at the Available Physical Memory (APM) and the "Mem Usage" (MU) for the eui.exe process.

As memory is allocated and assigned to the 'x' object, APM is decreasing and MU is increasing. When the 'x' object is assigned 0, APM is increases to (almost) the initial value, and MU decreases to (almost) its initial value. The residual amount I think is associated with the minimum allocation size for RAM (usually 4K or system page size).

Also, the time difference between concatenating and pre-allocating is negliable.

Here is my pre-allocation version

  
clear_screen()  
integer c 
object ch 
object x  
sequence t 
atom e 
c = 1 
puts(1, "Begin?\n") 
ch = gets(0) 
 
t = repeat(0,1000) 
x = repeat(0, 50*500)  
puts(1, "Pre-allocation done\n") 
ch = gets(0) 
puts(1, "Allocating ...\n") 
 
e = time() 
 
for i = 1 to 50 do -- adjust if it grabs too much memory for your system  
	? i  
	for j = 1 to 500 do  
 		for q = 1 to 1000 do 
 			t[q] = rand(1000) / rand(1000) 
 		end for 
		x[c] = t 
		c += 1 
	end for  
end for  
 
? time() - e  
puts(1, "End of allocation\n") 
ch = gets(0)  
x = 0 
puts(1, "X deferenced\n") 
ch = gets(0) 

I don't think so.

What are you using to measure this? I'm not seeing this effect at all.

19. Re: Memory usage on Win32: Euphoria usage vs. what Task Manager displays

• Posted by DerekParnell (admin) Jul 31, 2009
• 995 views

This might be the cause of our different results.

eui.exe is the Version 4 Euphoria executable.

exw.exe is the version 3 Euphoria executable.

21. Re: Memory usage on Win32: Euphoria usage vs. what Task Manager displays

• Posted by mattlewis (admin) Jul 31, 2009
• 1045 views

x = <big sequence> 
y = x 
x = 0 
 
-- y still references big sequence, so memory is not released 
 

It might be possible. If there was a temporary variable used at top level that referenced the sequence, it might not be released. v4 has changed a lot of things about the way references are handled.

One change is that we're trying to release references created by temps as soon as possible. Previously, they were often only released at the end of their routine, which was basically never, if the routine was actually the top level.

Matt

23. Re: Memory usage on Win32: Euphoria usage vs. what Task Manager displays

• Posted by DerekParnell (admin) Jul 31, 2009
• 1007 views

Yep, using mananged_mem exhibits the effects that Andy reported. One needs to compile with System Memory in order for it to release memory.

25. Re: Memory usage on Win32: Euphoria usage vs. what Task Manager displays

• Posted by AndySerpa Jul 31, 2009
• 1015 views
• Last edited Feb 03, 2011

Yeah, says managed memory. I didn't even notice that.

So, the question is -- if that's the way it is gonna be, is there any reason I couldn't use a "system memory" version? Does it mess up the way it interacts with .dlls or anything?

27. Re: Memory usage on Win32: Euphoria usage vs. what Task Manager displays

• Posted by AndySerpa Jul 31, 2009
• 1034 views
• Last edited Feb 03, 2011

Well, if there was a change, it must have been at v3.0, because that is when I started having this problem. Like I said, I reported on this a couple of times over a year ago. Looking at the code in be_alloc.c, which I do not understand, I can nevertheless tell that the code for Windows is much more complex than for other platforms. Any chance there is actually a subtle bug in there -- an unintended change? Why, for instance, in my example when I set x = {} immediately get back 90MB? What's special about 90MB? Is it releasing one block of something and then stopping when it shouldn't?

If there is no bug, would a simple programmatic option be possible that could control that amount of caching? (e. "with cache <min/normal/large>") Which would not affect the ability to use all the memory you need, but would allow (with option "min") to release as much as possible (at the expense of alloc/dealloc speed if that's the cost) when finished in order to keep the memory usage of a process no bigger than need be for what is it currently doing? (Option normal could be normal/default and large could actually pre-allocate extra memory to keep things speedy. Or some such scheme. Possible? Smart?) Or maybe an explicit command that could be put anywhere? (e.g. "free_cache()")

Or does it actually need to be compiled differently?

It doesn't come up that often, but when it does it turns your program into a real resource hog, even when it is finished with the resources. I know Euphoria has always prided itself on automatic garbage collection, but it is holding on to the trash.