1. free seems not work
- Posted by PtitChaud Feb 22, 2011
- 1786 views
Hello,
i run the Euphoria 4 code below on vista and the result, noticed into the ex.err file, is
D:\Programme\euphoria\euphoria\include\std\machine.e:81 in function allocate()
Your program has run out of memory.
One moment please...
n = 10000
cleanup = 0
iaddr = <no value>
eaddr = <no value>
the 2 Giga memory line is reached, after the 3rd call to the UnCycleComplet procedure. Where is my mystake ? or the free statement doesn't release back the memory to the system ? or any idea ?
thanks for Euphoria :):):)
include std/os.e -- include std/console.e -- include std/machine.e -- -- -- constant -- NOMBRE_CYCLE =100, -- NOMBRE_CYCLE2 =200, -- BOUCLE =1000 -- -- -- procedure WaitKeyMsg(sequence msg) -- integer rc_key -- printf(1,"%s\r\n",{msg}) -- rc_key = wait_key () -- printf(1,"[%d]",{rc_key}) -- end procedure -- -- -- procedure allocate_bcl(sequence ptr_liste) -- for j=1 to BOUCLE do -- ptr_liste[j]=allocate(10000) -- end for -- end procedure -- -- -- procedure free_bcl(sequence ptr_liste) -- for j=1 to BOUCLE do -- free(ptr_liste[j]) -- end for -- end procedure -- -- -- procedure UnCycleComplet() -- sequence ptr_liste -- ptr_liste = repeat(0,BOUCLE) -- for i=1 to NOMBRE_CYCLE do -- allocate_bcl(ptr_liste) -- free_bcl(ptr_liste) -- end for -- end procedure -- -- -- WaitKeyMsg("Appuye sur une touche pour commencer....") -- for k=1 to NOMBRE_CYCLE2 do -- printf(1,"[%3d]",{k}) -- UnCycleComplet() -- end for -- WaitKeyMsg("Appuye sur une touche pour terminer....") --
2. Re: free seems not work
- Posted by mattlewis (admin) Feb 22, 2011
- 1736 views
Hello,
i run the Euphoria 4 code below on vista and the result, noticed into the ex.err file, is
D:\Programme\euphoria\euphoria\include\std\machine.e:81 in function allocate()
Your program has run out of memory.
One moment please...
n = 10000
cleanup = 0
iaddr = <no value>
eaddr = <no value>
the 2 Giga memory line is reached, after the 3rd call to the UnCycleComplet procedure. Where is my mystake ? or the free statement doesn't release back the memory to the system ? or any idea ?
The problem is that ptr_liste in UnCycleComplet() is not updated by the call to allocate_bcl(). You could change your code to (only showing the parts that changed):
function allocate_bcl(sequence ptr_liste) for j=1 to BOUCLE do ptr_liste[j]=allocate(10000) end for return ptr_liste end function procedure UnCycleComplet() sequence ptr_liste ptr_liste = repeat(0,BOUCLE) for i=1 to NOMBRE_CYCLE do ptr_liste = allocate_bcl(ptr_liste) free_bcl(ptr_liste) end for end procedure
Matt
3. Re: free seems not work
- Posted by AndySerpa Feb 22, 2011
- 1732 views
The problem is that you declare the sequence ptr_liste in the procedure UnCycleComplet. It is LOCAL to that procedure. Then you pass it to the allocator, which makes its OWN LOCAL COPY. After allocation, the ptr_liste in UnCycleComplet is still just a string of zeros, which is what you are then passing to the free function. You can fix it by making your allocator a function that passes the sequence back instead of a procedure:
function allocate_bcl(sequence ptr_liste) -- for j=1 to BOUCLE do -- ptr_liste[j]=allocate(10000) -- end for return ptr_liste -- end function
and then also:
procedure UnCycleComplet() -- sequence ptr_liste -- ptr_liste = repeat(0,BOUCLE) -- for i=1 to NOMBRE_CYCLE do -- ptr_liste = allocate_bcl(ptr_liste) -- free_bcl(ptr_liste) -- end for -- end procedure
Understand the difference?
4. Re: free seems not work
- Posted by PtitChaud Feb 22, 2011
- 1769 views
Sorry, very sorry. I've changed my code without good test. OK with this new code :
function allocate_bcl(sequence ptr_liste) -- for j=1 to BOUCLE do -- ptr_liste[j]=allocate(10000) -- end for -- return ptr_liste -- end function -- -- -- procedure UnCycleComplet() -- sequence ptr_liste -- ptr_liste = repeat(0,BOUCLE) -- for i=1 to NOMBRE_CYCLE do -- ptr_liste=allocate_bcl(ptr_liste) -- free_bcl(ptr_liste) -- end for -- end procedure --
i observe that with wmmap and processexplorer programmes of sysinternals :
before the first call of UnCycleComplet(), the programme use 8MB of private memory and the heap memory type is about 5MB
after the last call of UnCycleComplet(), the programme use 15MB of private memory
and the virtual memory is 32MB.
with vmmap.exe, i notice that : private size of heap memory type is about 12MB.
Is it the normal behavior of the free statement ?
5. Re: free seems not work
- Posted by petelomax Feb 23, 2011
- 1640 views
with vmmap.exe, i notice that : private size of heap memory type is about 12MB.
Is it the normal behavior of the free statement ?
Your inner code:
ptr_liste=allocate_bcl(ptr_liste) -- free_bcl(ptr_liste) --
is "allocating and freeing" about 10MB. I cannot be absolutely certain about 10K blocks, but small blocks are not released "back to the OS" but kept in a "pool" for later use, which can significantly improve performance. Occasionally someone gripes about this, and perhaps there should be a "freepools()" builtin to shut them up, but I rather doubt it would actually help anyone. 12MB would tally with those blocks being kept in a pool, and the same memory is definately being re-used over and over, otherwise the main routine (UnCycleComplet) would chew through about 1GB and the outer about 200GB, which tallies with your original post.
Pete
6. Re: free seems not work
- Posted by PtitChaud Feb 23, 2011
- 1617 views
with vmmap.exe, i notice that : private size of heap memory type is about 12MB.
Is it the normal behavior of the free statement ?
Your inner code:
ptr_liste=allocate_bcl(ptr_liste) -- free_bcl(ptr_liste) --
is "allocating and freeing" about 10MB. I cannot be absolutely certain about 10K blocks, but small blocks are not released "back to the OS" but kept in a "pool" for later use, which can significantly improve performance. Occasionally someone gripes about this, and perhaps there should be a "freepools()" builtin to shut them up, but I rather doubt it would actually help anyone. 12MB would tally with those blocks being kept in a pool, and the same memory is definately being re-used over and over, otherwise the main routine (UnCycleComplet) would chew through about 1GB and the outer about 200GB, which tallies with your original post.
Pete
Thank's Pete,
I'm working on the growing up memory in my big application. I hope i have not stupid mistakes like done yesterday.
So to search and find, i'm trying logical piece of code from my application, like odbc functions or spécific protocole functions and so on. Actually, i found nothing but the heap memory trouble. This type of memory grows up, slowly or quickly, depends on input parameters of my application.
with the previous code, we start with 5MB on the private heap and after the last use, the private heap is nearly 12MB. The différence is 7MB, for one sequence with 1000 atoms, which is very small, compared with the use in my application. In my application, very many big sequences are used, and very often changes of length are done on the sequences.
In 2008, the use of memory was stable. Now i must stop the application after 2 days of run.
May be i've bugs in my code, may be trouble with the use of différents library writed with Eu311 or with Eu40, may be all is ok but the need of private heap memory is more than 2 Gigabytes.
I will be happy for some tips to help me in my search.
7. Re: free seems not work
- Posted by AndySerpa Feb 23, 2011
- 1586 views
I cannot be absolutely certain about 10K blocks, but small blocks are not released "back to the OS" but kept in a "pool" for later use, which can significantly improve performance. Occasionally someone gripes about this, and perhaps there should be a "freepools()" builtin to shut them up, but I rather doubt it would actually help anyone. 12MB would tally with those blocks being kept in a pool, and the same memory is definately being re-used over and over, otherwise the main routine (UnCycleComplet) would chew through about 1GB and the outer about 200GB, which tallies with your original post.
I have "griped" about Eu not releasing memory back to the OS, but only very large amounts. If you have a program that eats up a ton of memory with some initialization routine, but then never needs it again, Eu will hold onto it forever (mostly -- it will release maybe 25% of it) even if you release the variables holding it (i.e. ref count = 0). I'm talking 100s or 1000s of megabytes not available to other programs and un-needed by the Eu program holding onto it. I thought this would no longer be an issue with Eu4.0 cause you can compile with "system managed memory", but I tried that and it seemed to have no effect. But I've moved to Windows 7 64-bit since I last had this discussion (used to use XP 32-bit), so maybe the Windows memory management is the difference -- it does seem to do a better job swapping out the Eu memory hogging program now if another program needs the RAM, but I don't see why the Eu program needs to hold onto it at all in such huge amounts.
See this thread: http://openeuphoria.org/forum/107609.wc
(I chime in around post#7 and show some examples of what I'm talking about later in the discussion). I tried to compile the version of the beta that Derek was using since he said at that time his version compiled with system memory showed no such effects, but I have been unable to figure out how to compile that old version from scratch.
8. Re: free seems not work
- Posted by mattlewis (admin) Feb 23, 2011
- 1639 views
I have "griped" about Eu not releasing memory back to the OS, but only very large amounts. If you have a program that eats up a ton of memory with some initialization routine, but then never needs it again, Eu will hold onto it forever (mostly -- it will release maybe 25% of it) even if you release the variables holding it (i.e. ref count = 0). I'm talking 100s or 1000s of megabytes not available to other programs and un-needed by the Eu program holding onto it. I thought this would no longer be an issue with Eu4.0 cause you can compile with "system managed memory", but I tried that and it seemed to have no effect. But I've moved to Windows 7 64-bit since I last had this discussion (used to use XP 32-bit), so maybe the Windows memory management is the difference -- it does seem to do a better job swapping out the Eu memory hogging program now if another program needs the RAM, but I don't see why the Eu program needs to hold onto it at all in such huge amounts.
Looking at the code, we appear to use HeapAlloc and HeapFree for memory allocation and freeing.
The managed memory stuff hangs onto sequence allocations, not user allocations.
Matt
9. Re: free seems not work
- Posted by AndySerpa Feb 23, 2011
- 1596 views
The managed memory stuff hangs onto sequence allocations, not user allocations.
Which means?
10. Re: free seems not work
- Posted by useless Feb 23, 2011
- 1573 views
I have "griped" about Eu not releasing memory back to the OS, but only very large amounts. If you have a program that eats up a ton of memory with some initialization routine, but then never needs it again, Eu will hold onto it forever (mostly -- it will release maybe 25% of it) even if you release the variables holding it (i.e. ref count = 0). I'm talking 100s or 1000s of megabytes not available to other programs and un-needed by the Eu program holding onto it. I thought this would no longer be an issue with Eu4.0 cause you can compile with "system managed memory", but I tried that and it seemed to have no effect. But I've moved to Windows 7 64-bit since I last had this discussion (used to use XP 32-bit), so maybe the Windows memory management is the difference -- it does seem to do a better job swapping out the Eu memory hogging program now if another program needs the RAM, but I don't see why the Eu program needs to hold onto it at all in such huge amounts.
I also have asked for a way to free memory used once and not needed again. I figure it just won't happen in Euphoria, using 32bits for each 7bit ascii character is the norm. A few people have found ways around this (C strings outside Eu, compacting multiple "sequence strings" chars into 32bit "integer" bytes, etc), but it's not included in Euphoria releases. As for apps that use gobs of memory, find a way to run them as external programs (i do not mean Eu tasks), and then kill them when done. There's a number of "shared memory" include files in the archives for moving data about between the programs. I doubt that will ever become a built-in for an Eu release either. You can also break up data and keep it on a harddrive, i do this often for multi-gigabyte files. I also use multiple computers at times on a gigabit lan to get around these limitations.
useless
11. Re: free seems not work
- Posted by AndySerpa Feb 23, 2011
- 1560 views
I have "griped" about Eu not releasing memory back to the OS, but only very large amounts. If you have a program that eats up a ton of memory with some initialization routine, but then never needs it again, Eu will hold onto it forever (mostly -- it will release maybe 25% of it) even if you release the variables holding it (i.e. ref count = 0). I'm talking 100s or 1000s of megabytes not available to other programs and un-needed by the Eu program holding onto it. I thought this would no longer be an issue with Eu4.0 cause you can compile with "system managed memory", but I tried that and it seemed to have no effect. But I've moved to Windows 7 64-bit since I last had this discussion (used to use XP 32-bit), so maybe the Windows memory management is the difference -- it does seem to do a better job swapping out the Eu memory hogging program now if another program needs the RAM, but I don't see why the Eu program needs to hold onto it at all in such huge amounts.
I also have asked for a way to free memory used once and not needed again. I figure it just won't happen in Euphoria, using 32bits for each 7bit ascii character is the norm. A few people have found ways around this (C strings outside Eu, compacting multiple "sequence strings" chars into 32bit "integer" bytes, etc), but it's not included in Euphoria releases. As for apps that use gobs of memory, find a way to run them as external programs (i do not mean Eu tasks), and then kill them when done. There's a number of "shared memory" include files in the archives for moving data about between the programs. I doubt that will ever become a built-in for an Eu release either. You can also break up data and keep it on a harddrive, i do this often for multi-gigabyte files. I also use multiple computers at times on a gigabit lan to get around these limitations.
I have no problem with the way Eu store values and using what it needs to WHILE IT IS USING IT, but with big memory-hogging operations it is a problem when it won't release the memory no matter what. Once upon a time (way back in v2.5) it actually did release it. I had to rewrite several programs when Eu v3 came along that became no longer usable. So using techniques as you describe -- such as breaking the program in two so I could have it do its processing and then kill that part of it, chaining it to a new process, all of which of course is a big pain. I was hoping that compiling with system managed memory would do the trick, but it seems just the same -- is that working correctly?
12. Re: free seems not work
- Posted by LarryMiller Feb 23, 2011
- 1562 views
The managed memory stuff hangs onto sequence allocations, not user allocations.
Which means?
I am not the authority on this so Matt Lewis can correct me if I am wrong. But this is how I understand it.
If Euphoria allocates memory for a sequence it will not return it to the OS when it is finished with it. Memory allocations are frequent in Euphoria and doing it this way avoids the overhead of calling the OS functions.
However, when an application calls free() to release memory it IS returned to the OS.
Note that all of the above references to memory above are to the processes virtual address space, not RAM. The OS always maintains control over how much RAM an application will get and how long it will keep it. Applications have no control and only limited influence over this. If a large amount of RAM was assigned to an application for sequences and it is no longer in active use the OS is free to reassign it to other processes. This is a common occurance in a modern OS and causes no problems.
13. Re: free seems not work
- Posted by mattlewis (admin) Feb 23, 2011
- 1595 views
The managed memory stuff hangs onto sequence allocations, not user allocations.
Which means?
Actually, managed memory seems to not free user allocations, either. Sorry, it's been a while since I've looked at the Windows memory management stuff.
Looking closer, the caching of memory appears to follow the following:
- Cache up to 2000 individual memory allocations
- Only cache allocations up to 1024 bytes
So, if you try to free (or if the interpreter frees the memory for a sequence) that's less than 1K, and assuming there aren't already too many saved, then instead of actually freeing the memory, euphoria puts it on a list, so that later, the memory can be used for some other purpose. This avoids the overhead of actually allocating new memory. However, the amounts involved here are relatively small.
For larger allocations (whether sequence data, or user allocated) then, we call HeapFree. Based on the documentation, I can't tell what windows might do with that memory. Comparing the wording used for the Heap vs Virtual memory management functions, however, I suspect that the pages of memory are still part of the heap, and not decommitted, meaning that from the system's perspective, that memory is still allocated.
I think this is good for speed, but obviously does not release memory. I could be way off, though. Perhaps we could test changing over to the Virtual memory management functions, where it's possible to fully decommit the memory using VirtualFree.
Matt
14. Re: free seems not work
- Posted by mattlewis (admin) Feb 23, 2011
- 1552 views
I also have asked for a way to free memory used once and not needed again. I figure it just won't happen in Euphoria, using 32bits for each 7bit ascii character is the norm.
It gets worse! We use 64 bits with 64-bit euphoria (similar to, e.g., erlang). 
I think that the switch to the Heap functions was meant to return memory to the OS, but I think now that it's still committed to the process, as far as the OS is concerned. So while it would get recycled in future allocations, it would always appear allocated to other processes.
Matt
15. Re: free seems not work
- Posted by jimcbrown (admin) Feb 23, 2011
- 1558 views
I also have asked for a way to free memory used once and not needed again. I figure it just won't happen in Euphoria
This is already the case if you use Unix.
16. Re: free seems not work
- Posted by PtitChaud Feb 23, 2011
- 1558 views
Thank's for your answer,
today with the same code, i've done this 2 tests:
-1- a loop with 10 iterations of 10MB, allocated and freed
At the start, the private memory was 8MB and at the end the same. With WMMAP.EXE, i saw that after the free_bcl() procedure, all the allocated memory was released.
-2- a loop with 100*1024 iterations of 1024bytes, allocated and freed.
The amount of the allocated memory is the same than in the first test.
At the start, the private memory was nearly the same than in the first test : 8MB, but at the end, the private memory was 29MB
Ok, 21MB is perhaps necessary to internal Euphoria functions to improve the performance, reuse memory ...and not a bug
So, when i use ODBC, the result of the querie is stored into a sequence with elements. The first element contains the name of the each column and the next elements contain the values. After, with a loop, each element (each row) stored in the result sequence is stored into another very big sequence with an other loop to store one by one each value. Many usage of temporaly sequences and atoms and with différent length. So in this case, would it be better to use allocated user memory instead of temporaly sequences and avoid the memory needed by internal euphoria functions ? what are the good practices ?
Thank's for Euphoria and any help to how to do to find why private memory grow up.
17. Re: free seems not work
- Posted by AndySerpa Feb 23, 2011
- 1551 views
Actually, managed memory seems to not free user allocations, either. Sorry, it's been a while since I've looked at the Windows memory management stuff.
So is there still a difference between "Euphoria managed memory" and "system managed memory" (a choice at compilation)? I'm not seeing a difference between memory-hoggingness between the two, or should I chalk that up to something Win7 64bit is doing that XP 32bit didn't? I did try it on an virtual XP 32bit with VirtualBox, but even the old version of Eu didn't behave the same as it did on my old non-virtual XP setups, so I assume the virtualization is changing things there somehow.
It isn't as much an issue as it used to be simply because I have more memory nowadays -- back when this first came up I was trying to do these big operations in 2Gb of RAM or less (512MB probably in some cases) and now I've got 8GB to work with on my current machine. Also I could probably run the same programs under Linux (VirtualBox) and it sounds like it might behave better. Haven't tried that.
18. Re: free seems not work
- Posted by mattlewis (admin) Feb 23, 2011
- 1538 views
Actually, managed memory seems to not free user allocations, either. Sorry, it's been a while since I've looked at the Windows memory management stuff.
So is there still a difference between "Euphoria managed memory" and "system managed memory" (a choice at compilation)? I'm not seeing a difference between memory-hoggingness between the two, or should I chalk that up to something Win7 64bit is doing that XP 32bit didn't? I did try it on an virtual XP 32bit with VirtualBox, but even the old version of Eu didn't behave the same as it did on my old non-virtual XP setups, so I assume the virtualization is changing things there somehow.
It's practically guaranteed that the memory managers of XP and Win7 are different. There's probably a Raymond Chen blog out there somewhere that explains this all.
Matt
19. Re: free seems not work
- Posted by LarryMiller Feb 23, 2011
- 1555 views
32 bit processes require more address space and more RAM on a 64 bit OS than on a 32 bit. I am not sure of the details but each 32 bit process has several DLL's mapped into it's address space that are not needed on a 32 bit OS. The RAM used is shown for each process but that is rather misleading because there is only one copy in memory.
I think the best source of this information is the Microsoft publication "Windows Internals", currently in the 5th edition. It is authored by Mark Russinovich and David Solomon.
