1. mem leaks win32 call_back
- Posted by raseu Jan 22, 2010
- 1519 views
first, apologies if i'm posting to the wrong place
version : Euphoria Interpreter 4.0.0 beta 2 (r2670) for Windows
using the following code under win32
causes memory leaks when viewed under Process tab
of Windows Task Manager.
include dll.e function callback_routine() return 0 end function procedure test_callback(sequence rname, integer loopcount) atom headfunc, writefunc headfunc = routine_id(rname) writefunc = routine_id(rname) headfunc = call_back({ '+', headfunc }) writefunc = call_back({ '+', writefunc }) printf (1, "loop = %4d\n", { loopcount }) end procedure --// main integer i = 1 while (1) do test_callback("callback_routine", i) i += 1 end while
the memory leaks can be traced to call_back( ...) function call.
after taking a look at the sources (.c and dll.e) it does appear
that call_back is allocating memory for function pointer handles
under windows.
is there any way to reclaim the memory allocated?
i am almost complete with writing an LGP (linear genetic programming)
library, that uses call_back() extensively (loads eui dlls as plugins,
then fixes up function pointers)
TIA
2. Re: mem leaks win32 call_back
- Posted by mattlewis (admin) Jan 23, 2010
- 1445 views
first, apologies if i'm posting to the wrong place
No, this is a good place to post this sort of thing.
after taking a look at the sources (.c and dll.e) it does appear
that call_back is allocating memory for function pointer handles
under windows.
is there any way to reclaim the memory allocated?
i am almost complete with writing an LGP (linear genetic programming)
library, that uses call_back() extensively (loads eui dlls as plugins,
then fixes up function pointers)
Yes, each call back allocates some memory, and as I think you've noticed, under windows, there are actually two allocations. Ideally, the callback wrapper in dll.e would be a thin wrapper around the MACHINE_FUNC call. There is actually a ticket open to do this (ticket:111).
At a minimum, though, this leak should be plugged. To do so, try adding a call to free(addr) before call_back() returns. Let us know if this works, and we can patch the code temporarily, until the real issue can be dealt with. The current implementation of call_back() allocates in blocks, and there's no way to reclaim individual callback allocations.
Matt
3. Re: mem leaks win32 call_back
- Posted by SDPringle Jan 24, 2010
- 1353 views
free() wont work. One is expected to call call_back() once for each function you want to get called by a C or assembler routine once. I never expected anyone to want to release the memory from this. You need to call free_code().
4. Re: mem leaks win32 call_back
- Posted by jimcbrown (admin) Jan 24, 2010
- 1352 views
free() wont work. One is expected to call call_back() once for each function you want to get called by a C or assembler routine once. I never expected anyone to want to release the memory from this. You need to call free_code().
Isn't this a problem thats also in the C backend anyways? And worse, as we can't free memory allocated from the C backend...?
5. Re: mem leaks win32 call_back
- Posted by raseu Jan 24, 2010
- 1399 views
for the test program i posted earlier,
call_back calls allocate_protect with a
sequence length of 80 (size + BORDER_SPACE * 2)
i retested with the following additions
to the test code
include std/machine.e
and
free_code(headfunc, 80) free_code(writefunc, 80)
after creating call_back(s)
with an additional free(addr) inside std/dll.e
before return page_offset + page_addr
same issue with memory leaks
any pointers (no pun intended) in the right direction
would be of great help
6. Re: mem leaks win32 call_back
- Posted by SDPringle Feb 04, 2010
- 1267 views
I have to retract what I said. You cannot use free_code(). You cannot free memory returned from call_back. CallBacks are there to let C routines call EUPHORIA routines. Call back sets up a place in EUPHORIA you want to let C call. It should be top level and in only a few places. This is what GUI libraries in EUPHORIA do for you. Why would you need to call this more than some constant number of times?
Shawn Pringle
7. Re: mem leaks win32 call_back
- Posted by raseu Feb 06, 2010
- 1187 views
1) ok, the call_back function is meant
to provide external routines the ability to call into euphoria,
but i would like the ability to dynamically setup/teardwon
call back interfaces on the fly as with other languages (namely 'C').
yes, i could resdesign my apps to use a more static/less flexible/fixed
function lookup table, but that's not what i want.
2) more importantly, if at some time in the future euphoria will
be available as an embedded language then the ability to provide
setup\teardown of embedded scripts that provides call backs surely
would need some method of allocating/deallocating function pointers
8. Re: mem leaks win32 call_back
- Posted by mattlewis (admin) Feb 06, 2010
- 1239 views
1) ok, the call_back function is meant
to provide external routines the ability to call into euphoria,
but i would like the ability to dynamically setup/teardwon
call back interfaces on the fly as with other languages (namely 'C').
yes, i could resdesign my apps to use a more static/less flexible/fixed
function lookup table, but that's not what i want.
Could you explain a little more about what you're doing?
2) more importantly, if at some time in the future euphoria will
be available as an embedded language then the ability to provide
setup\teardown of embedded scripts that provides call backs surely
would need some method of allocating/deallocating function pointers
I don't see why this is true. I've never done embedded development, but could you explain the advantage of doing this dynamically? On thing that would concern me about freeing the memory used by a callback is that you've given the address to some other bit of code. How do you know that it won't call the callback again?
You could write an asm stub that takes the parameters from the caller, add some additional information (like a routine_id) and calls a routine that dispatches the calls. Look in the archives for fptr.e to get started. It was originally created to be able to get callbacks that take more parameters than euphoria allows. In this case, you'd have only one real callback, but potentially lots of others that you would be able to allocate or free however you wanted.
Matt
9. Re: mem leaks win32 call_back
- Posted by DerekParnell (admin) Feb 06, 2010
- 1222 views
1) ok, the call_back function is meant
to provide external routines the ability to call into euphoria,
but i would like the ability to dynamically setup/teardwon
call back interfaces on the fly ...
Is this because you don't know which of your Euphoria routines is going to be called until the time you invoke the external function?
2) more importantly, if at some time in the future euphoria will
be available as an embedded language then the ability to provide
setup\teardown of embedded scripts that provides call backs surely
would need some method of allocating/deallocating function pointers
I think that you will find that this would be automated along with Garbage Collection for the released embedded script.
10. Re: mem leaks win32 call_back
- Posted by jimcbrown (admin) Feb 06, 2010
- 1205 views
You could write an asm stub that takes the parameters from the caller, add some additional information (like a routine_id) and calls a routine that dispatches the calls. Look in the archives for fptr.e to get started. It was originally created to be able to get callbacks that take more parameters than euphoria allows. In this case, you'd have only one real callback, but potentially lots of others that you would be able to allocate or free however you wanted.
If there is a good reason to do this, then this should be a builtin capability - or at least fptr.e should be part of the stdlib.
11. Re: mem leaks win32 call_back
- Posted by mattlewis (admin) Feb 06, 2010
- 1180 views
You could write an asm stub that takes the parameters from the caller, add some additional information (like a routine_id) and calls a routine that dispatches the calls. Look in the archives for fptr.e to get started. It was originally created to be able to get callbacks that take more parameters than euphoria allows. In this case, you'd have only one real callback, but potentially lots of others that you would be able to allocate or free however you wanted.
If there is a good reason to do this, then this should be a builtin capability - or at least fptr.e should be part of the stdlib.
Maybe. It's pretty rare, though it's basically what I did with the callbacks for eu.ex. I've also used fptr.e to use somewhat exotic calling conventions, like MS's thiscall. There were also some standard callbacks required in COM programming that required more than the then-allowed number of parameters.
Matt
12. Re: mem leaks win32 call_back
- Posted by jimcbrown (admin) Feb 06, 2010
- 1180 views
Maybe. It's pretty rare, though it's basically what I did with the callbacks for eu.ex. I've also used fptr.e to use somewhat exotic calling conventions, like MS's thiscall. There were also some standard callbacks required in COM programming that required more than the then-allowed number of parameters.
Matt
Adding the ability to free callbacks natively would then simplify eu.ex at least. And for the embedded euphoria, it would make things easier to implement if we can keep the same code used by the backend...
13. Re: mem leaks win32 call_back
- Posted by mattlewis (admin) Feb 06, 2010
- 1176 views
Maybe. It's pretty rare, though it's basically what I did with the callbacks for eu.ex. I've also used fptr.e to use somewhat exotic calling conventions, like MS's thiscall. There were also some standard callbacks required in COM programming that required more than the then-allowed number of parameters.
Matt
Adding the ability to free callbacks natively would then simplify eu.ex at least. And for the embedded euphoria, it would make things easier to implement if we can keep the same code used by the backend...
I don't follow you about eu.ex. It uses the same technique (just transplanted from fptr.e into execute.e) as fptr.e. Take a look at cb_std and cb_cdecl to see what I was talking about as far as adding an additional parameter for dispatching.
And I don't see what use freeing callbacks would be in eu.ex. When would you free them? Why?
Matt
14. Re: mem leaks win32 call_back
- Posted by jimcbrown (admin) Feb 06, 2010
- 1253 views
I don't follow you about eu.ex. It uses the same technique (just transplanted from fptr.e into execute.e) as fptr.e. Take a look at cb_std and cb_cdecl to see what I was talking about as far as adding an additional parameter for dispatching.
Hmm, you're right, it wouldn't really help in this case.
And I don't see what use freeing callbacks would be in eu.ex. When would you free them? Why?
For the same reasons for freeing callbacks in general.
15. Re: mem leaks win32 call_back
- Posted by mattlewis (admin) Feb 06, 2010
- 1238 views
And I don't see what use freeing callbacks would be in eu.ex. When would you free them? Why?
For the same reasons for freeing callbacks in general.
OK, so for callbacks in general: When would you free them? Why?
Sorry, maybe I missed it, or just not getting it. This is an honest question about how you would decide to free a callback. In what sort of situation would this be useful? If there's a good reason for it, it's definitely something we could do.
I haven't looked closely at the changes Shawn recently made, but I suppose there's no reason why we couldn't recycle the memory used for callbacks. But it would complicate the code, add places for us to write bugs, etc, and I just can't justify it, because it seems like something that's likely unsafe, and doesn't really do all that much for you. I'd be happy to be shown to be wrong here, though, because maybe there is something cool that can be done with it.
Matt
16. Re: mem leaks win32 call_back
- Posted by jimcbrown (admin) Feb 06, 2010
- 1156 views
OK, so for callbacks in general: When would you free them? Why?
Sorry, maybe I missed it, or just not getting it. This is an honest question about how you would decide to free a callback. In what sort of situation would this be useful? If there's a good reason for it, it's definitely something we could do.
But it would complicate the code, add places for us to write bugs, etc, and I just can't justify it, because it seems like something that's likely unsafe, and doesn't really do all that much for you. I'd be happy to be shown to be wrong here, though, because maybe there is something cool that can be done with it.
Well, technically, we are leaking memory with callbacks ... even if we only make a callback to be used once, and called once, say for initialization of a library at program startup, the memory stays used until the process ends. I do not find the argument that freeing a callback might be unsafe to be very compelling, as the same applies to freeing memory in general - and in fact to using define_c and c_proc/c_func.
Projects in other languages which have thousands of functions do exist, and being able to generate callbacks arbituarily (without having to worry about memory usage in worst-case scenarios) would be quite handy.
Also, as raseu points out, the current system limits the design of end user code. He or she has to redesign his code to use a table to remember routine ids and if a callback already exists for it (although this could also be fixed by having the backend store and remember the callback, so there is only one callback ever created for one routine id).
Another reason is portability.. moving away from ftpr.e (which in any case seems like overkill just for the ability to free callbacks) means no need to rewrite machine code for a different arch.
The above directly contradicts the next reason, however, which is to allow for directly modifying the callback machine code. Once the end user has copied the machine code over and manipulated it, it would be handy to be able to free the original when it is no longer needed.
I haven't looked closely at the changes Shawn recently made, but I suppose there's no reason why we couldn't recycle the memory used for callbacks.
From his comments, the impression I got was that we had to choose between two options. So it would not be possible to free callbacks without reverting some of his changes..
17. Re: mem leaks win32 call_back
- Posted by mattlewis (admin) Feb 07, 2010
- 1138 views
Well, technically, we are leaking memory with callbacks ... even if we only make a callback to be used once, and called once, say for initialization of a library at program startup, the memory stays used until the process ends. I do not find the argument that freeing a callback might be unsafe to be very compelling, as the same applies to freeing memory in general - and in fact to using define_c and c_proc/c_func.
I guess something that's guaranteed to only be called once, like for initialization is a good example of something that could be freed safely. I was thinking of what I believe is the main use of a callback, which is used in conjunction with GUI libraries as ways for the OS/window manager/etc to send messages to the user's program. About the only time it's safe to free that sort of callback is at the end of your program. Maybe some timer callbacks (assuming it even gets its own callback) are throw aways.
But maybe raseu is doing something completely different. That's what I'm interested in finding out. I guess I view this leaked memory in a similar manner to having messages sitting around in RAM. In fact, strings stored in constants probably take up way more memory in a typical program than the callbacks. Especially than callbacks for things like initialization.
Projects in other languages which have thousands of functions do exist, and being able to generate callbacks arbituarily (without having to worry about memory usage in worst-case scenarios) would be quite handy.
Maybe. An example would be much more persuasive.
Also, as raseu points out, the current system limits the design of end user code. He or she has to redesign his code to use a table to remember routine ids and if a callback already exists for it (although this could also be fixed by having the backend store and remember the callback, so there is only one callback ever created for one routine id).
Yes, that's probably the obvious next step. But I still think we need a better case made for doing something like this.
I haven't looked closely at the changes Shawn recently made, but I suppose there's no reason why we couldn't recycle the memory used for callbacks.
From his comments, the impression I got was that we had to choose between two options. So it would not be possible to free callbacks without reverting some of his changes..
It depends on the platform. On windows, we're allocating chunks of memory (page size, I think) due to the way we have to protect the memory. I suspect we should probably be doing that on other platforms, too. So in that case, we've got a custom malloc process where we could, in theory, add an ability to recycle old callbacks that are no longer required.
This all just seems like so much premature optimization though.
Matt
18. Re: mem leaks win32 call_back
- Posted by jimcbrown (admin) Feb 07, 2010
- 1123 views
On windows, we're allocating chunks of memory (page size, I think) due to the way we have to protect the memory. I suspect we should probably be doing that on other platforms, too. So in that case, we've got a custom malloc process where we could, in theory, add an ability to recycle old callbacks that are no longer required.
This all just seems like so much premature optimization though.
Right. Under the old system, it'd take 1024 callbacks just to use 4MB of ram... now it'll take 43008 callbacks to get that high.
Since we could never free callbacks before, this is really a new feature, and discussion should be pushed back to 4.0.1 or even 4.1
19. Re: mem leaks win32 call_back
- Posted by raseu Feb 08, 2010
- 1065 views
one of the projects i have is a wrapper for
the CURL libraries where i have been converting
some of my older 'C' code that creates/downloads/destroys connections
for url/ftp downloads as and when needed.
as part of the connection process i create callbacks
to buffer data.
porting the 'C' code to the euphoria equivalent yields
the following
... self[headfunc] = routine_id("strip_cookies") self[writefunc] = routine_id("strip_data") self[headfunc] = call_back({ '+', self[headfunc] }) self[writefunc] = call_back({ '+', self[writefunc] }) curl:setopt(self[handle], CURLOPT_HEADERFUNCTION, self[headfunc]) curl:setopt(self[handle], CURLOPT_WRITEFUNCTION, self[writefunc]) etc ...
which, if created/used/destroyed 'on the fly', will run into the mem leak
issues as outlined at the top of this thread
i was hoping to maintain 'like for like' code and flow control
between the two language versions as an aid for testing/debugging
anyways, not a show stopper.
for now, i'll just have to redesign some aspects of the code
would be ideal to have the have as much user control as possible
over allocated memory at some stage though.
thanks for the updates
btw, me is a he