1. heap memory and long duration application type
- Posted by PtitChaud Feb 25, 2011
- 1463 views
Hello,
this post follows the first free seems not work.
I've run the code below to do 4 tests, with Eu40 on Vista.
include std/os.e -- include std/console.e -- include std/machine.e -- include std/dll.e -- include std/math.e -- constant -- NOMBRE_CYCLE =1, -- NOMBRE_CYCLE2 =400, -- BOUCLE =10*1024*10, -- BUFFER =2048, -- DELAY_CYCLE =3, -- DELAY_CYCLE2 =3, -- DLL_ALLOC_FREE =0, -- BUFFER_FIXE =0 -- -- -- procedure abort_msg(sequence msg) -- printf(2,"Fatal error [%s]\r\n",{msg}) -- abort((wait_key()*0)+1) -- retourne toujours 1 end procedure -- -- -- constant kernel32 = open_dll("kernel32.dll") -- if not kernel32 then -- abort_msg("kernel32.dll") -- end if -- -- link dll functions -- constant -- iLocalAlloc =define_c_func(kernel32,"LocalAlloc", {C_UINT, C_UINT}, C_INT ), -- iLocalFree =define_c_func(kernel32,"LocalFree", {C_INT}, C_INT ) -- if find(-1,{iLocalAlloc,iLocalFree}) then -- abort_msg("One of the kernel32.dll function was not found.") -- end if -- -- --<>--------------------------------------------------------------- -- -- -- --<>--------------------------------------------------------------- function LocalAlloc(atom uFlags,atom uBytes) -- return c_func(iLocalAlloc,{uFlags,uBytes}) -- end function -- -- --<>--------------------------------------------------------------- -- -- -- --<>--------------------------------------------------------------- function LocalFree(atom hMem) -- return c_func(iLocalFree,{hMem}) -- end function -- -- -- -- -- -- -- procedure WaitKeyMsg(sequence msg) -- integer rc_key -- printf(1,"%s\r\n",{msg}) -- rc_key = wait_key () -- printf(1,"[%d]",{rc_key}) -- end procedure -- -- -- function allocate_bcl(sequence ptr_liste) -- integer a -- for j=1 to BOUCLE do -- if BUFFER_FIXE then -- a=BUFFER -- else -- a=mod(j,BUFFER)+1 -- end if -- if DLL_ALLOC_FREE then -- ptr_liste[j]=LocalAlloc(0,a) -- else -- ptr_liste[j]=allocate(a) -- end if -- end for -- return ptr_liste -- end function -- -- -- procedure free_bcl(sequence ptr_liste) -- atom ok -- for j=1 to BOUCLE do -- if DLL_ALLOC_FREE then -- ok=LocalFree(ptr_liste[j]) -- else -- free(ptr_liste[j]) -- end if -- end for -- end procedure -- -- -- procedure UnCycleComplet() -- sequence ptr_liste -- ptr_liste = repeat(0,BOUCLE) -- for i=1 to NOMBRE_CYCLE do -- ptr_liste=allocate_bcl(ptr_liste) -- sleep(DELAY_CYCLE) -- free_bcl(ptr_liste) -- sleep(DELAY_CYCLE) -- end for -- end procedure -- -- -- WaitKeyMsg("Appuye sur une touche pour commencer....") -- -- -- for k=1 to NOMBRE_CYCLE2 do -- printf(1,"[%3d]",{k}) -- UnCycleComplet() -- sleep(DELAY_CYCLE2) -- end for -- -- -- WaitKeyMsg("Appuye sur une touche pour terminer....") -- -- --
In this code, the sleep statement is only needed to for a good mesurement by processexplorer tool.
For the 4 tests, i've modified the value of these constants:
- BOUCLE : number of time we do an allocation of memory of BUFFER
- BUFFER : amount of memory to be acquired in one time
- DLL_ALLOC_FREE : if TRUE value, use of external allocate/free functions, else use Euphoria functions
- BUFFER_FIXE : if FALSE value, the BUFFER value depend on mod function result but the total size allocated after the loop BOUCLE will be nearly the same as if TRUE value.
| Test number | BOUCLE | BUFFER | DLL_ALLOC_FREE | BUFFER_FIXE |
|---|---|---|---|---|
| 1 | 10 | 2048*1024*10 | 1 | 1 |
| 2 | 10 | 2048*1024*10 | 0 | 1 |
| 3 | 10*1024*10 | 2048 | 0 | 0 |
| 4 | 10*1024*10 | 2048 | 1 | 0 |
For each test, at several step, i've capture the size of the Private memory from the virtual memory used and the virtual size of the used memory
Before the first loop of memory allocation process
| Test number | PRIVATE | VIRTUAL SIZE |
|---|---|---|
| 1 | 008 324K | 042 424K |
| 2 | 008 324K | 042 424K |
| 3 | 008 380K | 042 224K |
| 4 | 008 380K | 042 224K |
During the memory allocation process:
| Test number | PRIVATE MIN | PRIVATE MAX | VIRTUAL SIZE |
|---|---|---|---|
| - | |
|
|
| 1 | 008 324K | 213 568K | 042 224K |
| 1 | 008 324K | 213 568K | 042 224K |
| 1 | 008 324K | 213 568K | 042 224K |
| 1 | 008 324K | 213 568K | 042 224K |
| - | |
|
|
| 2 | 008 324K | 213 568K | 042 224K |
| 2 | 008 324K | 213 568K | 042 224K |
| 2 | 008 324K | 213 568K | 042 224K |
| 2 | 008 324K | 213 568K | 042 224K |
| - | |
|
|
| 3 | 081 000K | 134 300K | 212 500K |
| 3 | 103 600K | 134 300K | 228 700K |
| 3 | 121 000K | 139 000K | 245 000K |
| 3 | 124 000K | 139 120K | 261 112K |
| - | |
|
|
| 4 | 039 692K | 118 284K | 163 960K |
| 4 | 042 908K | 118 348K | 163 960K |
| 4 | 038 256K | 118 400K | 163 960K |
| 4 | 041 672K | 118 400K | 163 960K |
After the last loop of memory allocation process
| Test number | PRIVATE | VIRTUAL SIZE |
|---|---|---|
| 1 | 008 324K | 042 424K |
| 2 | 008 324K | 042 424K |
| 3 | 127 584K | 261 112K |
| 4 | 040 744K | 163 960K |
For test 1 and test 2, 200 MB allocated and freed. So, no problem to say : Euphoria allocate and free functions do its job. Small sequence in these two tests. Allocated block of memory, too big for keeping in the internal Euphoria pool ?
For test 3 and test 4, 100 MB allocated and freed, with différent sizes of memory block, from 1 byte to 2048 bytes. All pointers of allocated memory blocks stored in a sequence of 100*1024 atoms.
With VMMAP.exe of http://www.sysinternals.com :
At the beginning:
| Test number | SIZE OF HEAP | SIZE OF PRIVATE HEAP |
|---|---|---|
| 3 | 008 388K | 005 444K |
| 4 | 008 388K | 005 444K |
At the end:
| Test number | SIZE OF HEAP | SIZE OF PRIVATE HEAP |
|---|---|---|
| 3 | 227 076K | 125 544K |
| 4 | 129 924K | 040 200K |
test 3 :
For an application with long duration, allocated memory seems to be not released and the size of the virtual memory grows up
test 4 :
For an application with long duration, private allocated memory needed to manage the sequence of 100*1024 atoms is about 40 000K but doesn't grow up.
What do you think about that ? bug in the internal manager of the heap memory ?
With a total size of about 2 GB allocated, what's happen for a long duration application type ?
2. Re: heap memory and long duration application type
- Posted by PtitChaud Feb 25, 2011
- 1408 views
I've just done the same test number 3 with Euphoria 311 and i obtain:
Before the first loop of memory allocation process
| Test number | PRIVATE | VIRTUAL SIZE |
|---|---|---|
| 3 | 000 980K | 033 144K |
After the last loop of memory allocation process
| Test number | PRIVATE | VIRTUAL SIZE |
|---|---|---|
| 3 | 030 998K | 144 632K |
With VMMAP.exe of http://www.sysinternals.com :
At the beginning:
| Test number | SIZE OF HEAP | SIZE OF PRIVATE HEAP |
|---|---|---|
| 3 | 001 288K | 000 444K |
At the end:
| Test number | SIZE OF HEAP | SIZE OF PRIVATE HEAP |
|---|---|---|
| 3 | 112 776K | 030 236K |
and the performance graph of the private bytes history shows a nice comb, depending on the duration of sleep function, like test 4. Peak of private bytes : 111 984K
3. Re: heap memory and long duration application type
- Posted by mattlewis (admin) Feb 25, 2011
- 1340 views
Hello,
this post follows the first free seems not work.
I've run the code below to do 4 tests, with Eu40 on Vista.
For comparison, I changed the program to use the normal euphoria allocate and free, and ran it on Linux with massif, a heap profiler. Here's the graph I got (I changed it to just run for 10 cycles):
MB
305.9^ #
| # : : : @
| # @ : :: : : : @
| # @ : : : : : : @
| # @ : : : : : : @ ::
| # @ : : : ::: : : :@ ::
| # @ :: : : : : :: : :@ ::
| ::# @ :: :: ::: : : :: ::: :@ ::
| : # ::@ :: :: : : : : :: : : :@ :::
| : # : @ :: :: : : :: : :: : : :@ :::
| : # : @ :: :: : : :: : :::: : : :::@ :::
| : # : @ ::: :: : ::: :: : : :: :: : : :@ :::
| : # : @ ::: ::: :: :: :: : : :: :: : : :@ :::
| : # : @ ::: ::: : :: :: :: : : :: :: : : :@ ::::
| ::: # :: @ ::: ::: : :: :: @:: : : :: :: : :: :@ ::::
| : : # :: @ ::: ::: : :: :: @:: : :: :: :: : :: :@ ::::
| : : # :: @ ::::: ::: : :: :: @:: : :: :: ::: : :: :@ ::::
| : : # :: @ :::::: @::: :@:: :: @:: : :: :: ::: : :: :@ ::::
| : : # :: @:: :::: @::: :@:: :: @:: : :: :: ::: : :: :@ ::::
| : : #:::: @:: :::: @::: :@:: :: @:: :::: :: ::: :::: :@ ::::@
0 +----------------------------------------------------------------------->Gi
0 2.737
The X-axis is the number of instructions executed.
Matt
4. Re: heap memory and long duration application type
- Posted by PtitChaud Feb 25, 2011
- 1370 views
I'm not sure to read correctly your graph. After ten cycles, what is the size of the heap memory ?
5. Re: heap memory and long duration application type
- Posted by mattlewis (admin) Feb 25, 2011
- 1336 views
I'm not sure to read correctly your graph. After ten cycles, what is the size of the heap memory ?
It seems to cycle between about 300MB and down to very little. If anything, the trend appears to be for less rather than more memory, though it's not a very long run.
I ran it for only two cycles, to get a better view of how the individual cycles look:
MB
305.9^ #
| # :
| :# @:
| ::# :@:
| :::# ::@::
| :::#:: :::@::
| :::::#: ::::@:::
| :::::#: ::::@:::
| :::::#: :::::@:::
| :::::::#: @:::::@:::
| : :::::#: : :@:::::@:::
| :: :::::#: : ::@:::::@::::
| :::: :::::#: : :::@:::::@::::
| : :: :::::#: : :::@:::::@:::::
| :: :: :::::#: : ::::@:::::@:::::
| :: :: :::::#: : :::::@:::::@:::::
| :::: :: :::::#: :: ::::::@:::::@:::::
| :: :: :: :::::#: :: :@:::::@:::::@:::::
| :: :: :: :::::#: :: ::@:::::@:::::@::::::
| ::: :: :: :::::#: :: :::@:::::@:::::@::::::
0 +----------------------------------------------------------------------->Mi
0 814.5
Matt
6. Re: heap memory and long duration application type
- Posted by PtitChaud Feb 25, 2011
- 1351 views
I ran it for only two cycles, to get a better view of how the individual cycles look:
With Test3, on Linux, how the virtual memory grows up ?
