1. FreeLibrary()
- Posted by useless_ Sep 28, 2012
- 2152 views
FreeLibrary() seems to be a windose call, and it's equated to FreeModule in winbase.h. Seems to be equated to a few other terms also. But it's not made availale for us lowly mortals to call? How can we commoners, we dregs, access FreeLibrary in Eu?
useless
2. Re: FreeLibrary()
- Posted by mattlewis (admin) Sep 28, 2012
- 2161 views
FreeLibrary() seems to be a windose call, and it's equated to FreeModule in winbase.h. Seems to be equated to a few other terms also. But it's not made availale for us lowly mortals to call? How can we commoners, we dregs, access FreeLibrary in Eu?
Here's a simple wrapper for FreeLibrary:
include std/dll.e constant KERNEL32 = open_dll( "kernel32.dll" ), FREELIBRARY = define_c_func( KERNEL32, "FreeLibrary", { C_POINTER }, C_INT ) public function free_library( atom dll ) return c_func( FREELIBRARY, { dll } ) end function
This should work with dlls opened with open_dll(), as we simply return the value returned by LoadLibrary.
Matt
3. Re: FreeLibrary()
- Posted by ghaberek (admin) Sep 28, 2012
- 2127 views
include std/dll.e constant KERNEL32 = open_dll( "kernel32.dll" ), FREELIBRARY = define_c_func( KERNEL32, "FreeLibrary", { C_POINTER }, C_INT ) public function free_library( atom dll ) return c_func( FREELIBRARY, { dll } ) end function
This should work with dlls opened with open_dll(), as we simply return the value returned by LoadLibrary.
I'm sure it's a rare use case, but I can see this being quote useful for a larger program that had to dynamically load modules in and out of memory.
And so...
- can we add this to the standard library?
- would it just be a call to dlclose() on UNIX side?
- should it be called close_dll() instead?
-Greg
4. Re: FreeLibrary()
- Posted by jimcbrown (admin) Sep 28, 2012
- 2107 views
include std/dll.e constant KERNEL32 = open_dll( "kernel32.dll" ), FREELIBRARY = define_c_func( KERNEL32, "FreeLibrary", { C_POINTER }, C_INT ) public function free_library( atom dll ) return c_func( FREELIBRARY, { dll } ) end function
This should work with dlls opened with open_dll(), as we simply return the value returned by LoadLibrary.
I'm sure it's a rare use case, but I can see this being quote useful for a larger program that had to dynamically load modules in and out of memory.
And so...
I agree - it's a rare case.
- would it just be a call to dlclose() on UNIX side?
Yes.
- should it be called close_dll() instead?
-Greg
Yes.
- can we add this to the standard library?
Yes!
5. Re: FreeLibrary()
- Posted by useless_ Sep 28, 2012
- 2046 views
I ran a tiny app for loading and unloading a dll 10k times, to time it, and found a memory leak. The app is at http://openeuphoria.org/pastey/153.wc , there's a short pause at the end of the first loop to catch the memory use total, and then the any_key() at the end to see the final memory use. First run uses ~10 megabytes, but use grows to ~100 megabytes after 10k runs. It's keeping ~9k of ram for each loop thru the load-unload loop. The dll source is at http://openeuphoria.org/pastey/154.wc .
useless
6. Re: FreeLibrary()
- Posted by jimcbrown (admin) Sep 28, 2012
- 2059 views
I ran a tiny app for loading and unloading a dll 10k times, to time it, and found a memory leak. The app is at http://openeuphoria.org/pastey/153.wc , there's a short pause at the end of the first loop to catch the memory use total, and then the any_key() at the end to see the final memory use. First run uses ~10 megabytes, but use grows to ~100 megabytes after 10k runs. It's keeping ~9k of ram for each loop thru the load-unload loop. The dll source is at http://openeuphoria.org/pastey/154.wc .
eukat
Part of this is due to the internals of define_c_func/proc. Everytime define_c_func/proc is called, a wrapper is made of the raw function pointer along with extra info (like the parameters and return type), and then a "C routine id" is returned.
Since you use the w32api to unload the shared object, the backend/runtime doesn't know that the shared object has been unloaded and keeps those wrappers around, wasting memory. AFAIK there's no way to free that memory short of ending the entire program.
9 kilobytes for a single define_c_func() call seems excessive, though. I was expecting more like 10 bytes or something (which times a million would only take up under 10 megabytes). Hopefully Matt can shed some light on this.
7. Re: FreeLibrary()
- Posted by ghaberek (admin) Sep 28, 2012
- 2012 views
Part of this is due to the internals of define_c_func/proc. Everytime define_c_func/proc is called, a wrapper is made of the raw function pointer along with extra info (like the parameters and return type), and then a "C routine id" is returned.
Since you use the w32api to unload the shared object, the backend/runtime doesn't know that the shared object has been unloaded and keeps those wrappers around, wasting memory. AFAIK there's no way to free that memory short of ending the entire program.
9 kilobytes for a single define_c_func() call seems excessive, though. I was expecting more like 10 bytes or something (which times a million would only take up under 10 megabytes). Hopefully Matt can shed some light on this.
So, the backend could probably track those resources and then release them with a call to close_dll(). That would probably be good...
-Greg
8. Re: FreeLibrary()
- Posted by useless_ Sep 28, 2012
- 2014 views
I ran a tiny app for loading and unloading a dll 10k times, to time it, and found a memory leak. The app is at http://openeuphoria.org/pastey/153.wc , there's a short pause at the end of the first loop to catch the memory use total, and then the any_key() at the end to see the final memory use. First run uses ~10 megabytes, but use grows to ~100 megabytes after 10k runs. It's keeping ~9k of ram for each loop thru the load-unload loop. The dll source is at http://openeuphoria.org/pastey/154.wc .
eukat
Part of this is due to the internals of define_c_func/proc. Everytime define_c_func/proc is called, a wrapper is made of the raw function pointer along with extra info (like the parameters and return type), and then a "C routine id" is returned.
Since you use the w32api to unload the shared object,
I am open to using other methods, if you know of one, show me it working, please.
the backend/runtime doesn't know that the shared object has been unloaded and keeps those wrappers around, wasting memory. AFAIK there's no way to free that memory short of ending the entire program.
If the back end creates those wrappers on demand, then the back end should have a named pointer to them, so an unload operation by that name should make that memory available for the next load(open_dll) operation. In this case, i am performing the loop on a DLL of the same name, it's not even like i am changing name, or DLL size, or number of functions in the DLL, or etc etc..
9 kilobytes for a single define_c_func() call seems excessive, though. I was expecting more like 10 bytes or something (which times a million would only take up under 10 megabytes). Hopefully Matt can shed some light on this.
I agree. Granted the OS prolly cached the whole operation, since i didn't change up the DLL i loaded, but swapping in-out 500 DLLs per second is impressive, i think.
useless
9. Re: FreeLibrary()
- Posted by jimcbrown (admin) Sep 28, 2012
- 2022 views
I am open to using other methods, if you know of one, show me it working, please.
The only other method I know of is to modify the backend to do what you and Greg describe, below. Obviously, I can't show this working without actually implementing it.
the backend/runtime doesn't know that the shared object has been unloaded and keeps those wrappers around, wasting memory. AFAIK there's no way to free that memory short of ending the entire program.
If the back end creates those wrappers on demand, then the back end should have a named pointer to them, so an unload operation by that name should make that memory available for the next load(open_dll) operation. In this case, i am performing the loop on a DLL of the same name, it's not even like i am changing name, or DLL size, or number of functions in the DLL, or etc etc..
10. Re: FreeLibrary()
- Posted by jimcbrown (admin) Sep 28, 2012
- 1999 views
So, the backend could probably track those resources and then release them with a call to close_dll(). That would probably be good...
-Greg
I agree. We definitely need a proper close_dll(). Right now, the only way to workaround this would be to wrap dlopen()/dlsym()/dlclose() and directly use the operating system calls to manage everything, bypassing the stdlib layer.
11. Re: FreeLibrary()
- Posted by jimcbrown (admin) Sep 29, 2012
- 2108 views
Since you use the w32api to unload the shared object, the backend/runtime doesn't know that the shared object has been unloaded and keeps those wrappers around, wasting memory. AFAIK there's no way to free that memory short of ending the entire program.
9 kilobytes for a single define_c_func() call seems excessive, though. I was expecting more like 10 bytes or something (which times a million would only take up under 10 megabytes). Hopefully Matt can shed some light on this.
I'm not so sure that this is the full story now.
I tried cutting out std/dll.e entirely here, using only the operating system library (plus a small helper c lib to deal with the E_SEQUENCE transition), and on startup it uses 14MB but then only uses 2MB more (16MB total) for 10k runs.
However, for 90k runs (which takes 27 seconds), it uses 75MB total.
http://openeuphoria.org/pastey/155.wc
I even tried cutting out the calls to the helper lib, in case repeated c_func() calls or the creation of new sequences was the culprit, but I still saw 75MB after 90k runs.
I guess even using dlclose() doesn't do the job.
12. Re: FreeLibrary()
- Posted by katsmeow Nov 01, 2021
- 871 views
The need to load and unload code while running has come up again. I was wondering if this memory issue was ever solved in OE, or if Phix resolved it?
The pastey links are still good for the test code that demo's the leaking.
Kat
13. Re: FreeLibrary()
- Posted by jimcbrown (admin) Nov 02, 2021
- 859 views
The need to load and unload code while running has come up again. I was wondering if this memory issue was ever solved in OE, or if Phix resolved it?
The pastey links are still good for the test code that demo's the leaking.
Kat
Afraid not. It only happened on windoze, and while I wasn't able to drive this to ground my working theory is that we exposed some kind of memory leak related to loadlibrary in the C stdlib as provided with MinGW.
It's possible that later versions of MinGW fixed the leak and we wouldn't have the issue anymore, but I no longer have a windoze system to check this.
I haven't tested Phix, but I suspect that the issue would not be present on Phix due to the above.
