1. DLL memory leaks
- Posted by mattlewis (admin) Feb 05, 2014
- 1578 views
Forked from Re: 4.1 multiple assign sequence items?
Basically, mattlewis hand editted a simple translated dll's C source to fix the leaks. But then he looked at what it'd take to fix the translator to do the same auomagically, and it seems he got hung up by the fact that the std library allocated some memory w/o ever freeing it, among other things. So this hasn't been entirely solved yet. (Like the rest of us, mattlewis is a very busy person, and even within Euphoria, there are other pressing tasks at hand.)
This squares with my memory. I'm pretty sure I fixed at least some memory issues. But I started going down rat holes and I think I got lost without solution to several problems. In particular, hg:euphoria/rev/5e8ee6853eb3.
Matt
2. Re: DLL memory leaks
- Posted by jimcbrown (admin) Feb 05, 2014
- 1592 views
it seems he got hung up by the fact that the std library allocated some memory w/o ever freeing it,
And i suggested doing the same for DLLs, allocate a hunk, perhaps under programmer control, of a size large enough to accomodate the largest DLL to be loading and unloading, then never free it. Just keep re-using it. In 2012.
I am stunned.
The problem is that different dlls, or even the same dll loaded, unloaded, then reloaded, do not communicate each other's memory system. There's no easy way to tell a DLL, "Last time you allocated this memory block and never freed it before being unloaded, so use it again."
While in principle it can be done, I suspect that in practice it'll be a lot easier to just find a way to free the memory.
3. Re: DLL memory leaks
- Posted by mattlewis (admin) Feb 05, 2014
- 1568 views
it seems he got hung up by the fact that the std library allocated some memory w/o ever freeing it,
And i suggested doing the same for DLLs, allocate a hunk, perhaps under programmer control, of a size large enough to accomodate the largest DLL to be loading and unloading, then never free it. Just keep re-using it. In 2012.
I am stunned.
The problem is that different dlls, or even the same dll loaded, unloaded, then reloaded, do not communicate each other's memory system. There's no easy way to tell a DLL, "Last time you allocated this memory block and never freed it before being unloaded, so use it again."
While in principle it can be done, I suspect that in practice it'll be a lot easier to just find a way to free the memory.
There are different issues involved. First, there is the overhead in the back end of euphoria for managing a dll. Second, there is the memory allocated by the dll itself. I was working on the memory that the dll would allocate for itself.
Matt
4. Re: DLL memory leaks
- Posted by andi49 Feb 05, 2014
- 1531 views
Hallo
I tried to follow this thread.
I tried to understand what's going on.
But i didn't get it.
i found the following in the referenced links...
It's a quote from jimcbrown.
I'm not so sure that this is the full story now. I tried cutting out std/dll.e entirely here, using only the operating system library (plus a small helper c lib to deal with the E_SEQUENCE transition), and on startup it uses 14MB but then only uses 2MB more (16MB total) for 10k runs. However, for 90k runs (which takes 27 seconds), it uses 75MB total. http://openeuphoria.org/pastey/155.wc I even tried cutting out the calls to the helper lib, in case repeated c_func() calls or the creation of new sequences was the culprit, but I still saw 75MB after 90k runs. I guess even using dlclose() doesn't do the job.
Why should a program that uses a dll 90000 times in 27seconds release this dll always after using it?
Maybe I'am a stupid but, really, i do not understand this.
(Please have patience, I'am not a programing expert)
Please help.
Andreas
5. Re: DLL memory leaks
- Posted by jimcbrown (admin) Feb 05, 2014
- 1544 views
Hallo
I tried to follow this thread.
I tried to understand what's going on.
But i didn't get it.
i found the following in the referenced links...
It's a quote from jimcbrown.
I'm not so sure that this is the full story now. I tried cutting out std/dll.e entirely here, using only the operating system library (plus a small helper c lib to deal with the E_SEQUENCE transition), and on startup it uses 14MB but then only uses 2MB more (16MB total) for 10k runs. However, for 90k runs (which takes 27 seconds), it uses 75MB total. http://openeuphoria.org/pastey/155.wc I even tried cutting out the calls to the helper lib, in case repeated c_func() calls or the creation of new sequences was the culprit, but I still saw 75MB after 90k runs. I guess even using dlclose() doesn't do the job.
Why should a program that uses a dll 90000 times in 27seconds release this dll always after using it?
Maybe I'am a stupid but, really, i do not understand this.
(Please have patience, I'am not a programing expert)
Please help.
Andreas
You are not stupid. Using a shared library 90k times in 27 seconds and releasing it always after using it, that's what is stupid. ;)
That was an example test case for the worst case scenario, in order to demonstrate the bug. While it'd be unusual to unload and reload the same shared object so many times, or within shuch a short time frame, there are many applications out there that are written in a modular fashion, with functionality implemented in helper shared libraries. It makes sense that the application would only load the librares on demand, on an as-needed basis, and perhaps it should unload them when it realises that it wouldn't need that particular helper library again (to use resources such as memory more efficently). But this bug (that a library won't always unload itself cleanly) affects the more common use case scenarios as well.
6. Re: DLL memory leaks
- Posted by andi49 Feb 05, 2014
- 1545 views
Okay, i see the problem.
But, a dll is made, becouse many differnt programs can use it, so you can't be sure if an other program still uses this dll.
Forcing an unload is not possible in this case.
If the dll is only used by one program, then you have some other opportunities. You can move the dll to an .exe and use some kind of IPC.
If the loaded .exe exits all memory is cleanded up by the OperatingSystem.
On the other hand why should a "End User Programming with Hierarchical Objects for Robust Interpreted Applications" Language should handle this?
Maybe I'am to 'easyminded' but i prefer a 4.1x with memstructs over a 4.1.x with 'dll unloading'.
Just my 2 cent.
Andreas
7. Re: DLL memory leaks
- Posted by jimcbrown (admin) Feb 05, 2014
- 1523 views
Okay, i see the problem.
But, a dll is made, becouse many differnt programs can use it, so you can't be sure if an other program still uses this dll.
Forcing an unload is not possible in this case.
Sort of. You can't force the OS to unload the shared library from all processes, however you can remove it from a particular process's memory address space - which in turn frees up memory for that process.
If the dll is only used by one program, then you have some other opportunities. You can move the dll to an .exe and use some kind of IPC.
If the loaded .exe exits all memory is cleanded up by the OperatingSystem.
Agreed. However, this can be more work than using a shared library, and may be ill-suited to certain types of functionality (e.g. where one has to pass around pointers in memory deal with inter-thread stuff or something).
On the other hand why should a "End User Programming with Hierarchical Objects for Robust Interpreted Applications" Language should handle this?
It shouldn't. The shared library unloading is dealt with in the operating system and in libdl.so
But memory leaks are a no no.
Maybe I'am to 'easyminded' but i prefer a 4.1x with memstructs over a 4.1.x with 'dll unloading'.
Just my 2 cent.
Andreas
Seconded.
