1. Euphoria dlls with multitasking
- Posted by mattlewis (admin) Oct 02, 2009
- 1712 views
Forked from Re: submitting code
I actually find it interesting since I couldn't find the referenced discussions on the dev list.
The background for this issue is that trying to use any of the multitasking API in euphoria code translated into a dll results in a compile error. Jeremy had wondered if there were an easy way to make this work. The main reason why it doesn't is the design of multitasking. In fact, there are two designs.
First, there is one way of dealing with tasks in interpreted code. In many ways, this is a lot easier to handle, since the backend of euphoria handles all of the call stack, etc. With translated code, of course, we're using the standard stack space supplied by the OS. Currently, these two systems operate completely separately.
We actually have an open issue (though low priority) regarding this as it affects the front end of the interpreter. The front end sort of cheats, in that it's translated code, but it uses the standard interpreted backend for its runtime code, as opposed to the normal translated backend that the translator links to your translated euphoria code. In most cases, it doesn't really make a difference, since sprintf() is sprintf() is sprintf(). However, as noted above, the implementation of tasking is not so simple, and is switched out via ifdefs in the C code (see be_task.c).
So, in order to be able to integrate a dll with interpreted euphoria code, we need a way for the two implementations to coexist and work together. Of course, it might be a translated application that's using the euphoria dll, but that's a little easier, since we'd only be using a single tasking mode.
Of course, there's some additional administrative stuff that would have to happen to be able to do this. When you loaded a euphoria dll, there'd need to be some sort of initialization/handshake that happened in order to let the dll know where to call task_yield(), or whatever.
There are probably other implementation issues that we'd encounter implementing this, but we've already found that we probably need to rewrite the translated tasking system anyways. Translated tasks fail on recent linuxes. I think it's trying to prevent against things like stack smashing attacks, which I think is what a translated task looks like to the kernel.
There are lots of reasons why this is desirable. Let's start with one of the initial reasons for translating euphoria code into a dll: making your code run faster. Suppose you did this with the standard library. It's presumed to be well tested, etc, so it might be a good candidate for speeding up your code. There are a few places where multitasking is used (or can be used) to improve performance. One such place is within http.e.
In the future, there are some plans to integrate translated and interpreted code so that it might even be transparent to the programmer. Excluding that code from using tasks could make the feature a lot less useful.
Matt
2. Re: Euphoria dlls with multitasking
- Posted by jaygade Oct 03, 2009
- 1622 views
- Last edited Oct 04, 2009
Thanks for the explanation, Matt. I'd seen a bit of the discussion on tasking itself here on the forum but I was looking more for the source of the conflict between Kat and others and not finding it.
But with regards to the actual topic that's pretty interesting. I kind of understood how the tasking worked within the interpreter, but (even though I haven't read the source for the actual details) the implementation in translated code seems very complex.
Question: why does the front end "cheat"? Does that not make it a bit slower? I have noticed pretty long load times for even simple Euphoria programs anymore.
Anyway, thanks for the explanation.
3. Re: Euphoria dlls with multitasking
- Posted by mattlewis (admin) Oct 03, 2009
- 1613 views
- Last edited Oct 04, 2009
Question: why does the front end "cheat"? Does that not make it a bit slower? I have noticed pretty long load times for even simple Euphoria programs anymore.
"Cheat" is maybe the wrong word. Obviously, the front end is translated euphoria code. If you look at the source code for euphoria, in be_execute.c, there is a giant switch statement that is the heart of running interpreted code. It does an awful lot of stuff in there.
However, there is a lot of other stuff that is done via function calls, like opening a file, any IO, copying a sequence for a copy on write operation, etc. These other functions are also used by the translator, though there are some differences, especially for certain things like tasks. Others are identical, such as copying a sequence, since a sequence is a sequence is a sequence.
In any case, the front end for the interpreter uses the same runtime functions that your interpreted code uses. And again, for most things, this is fine, because there really is no difference. As we've expanded the front end, there have been some places where we've changed the runtime to handle translated code slightly differently, meaning that the translated C code might call a similar, but different function than interpreted code does.
Without knowing anything about the programs you're talking about, it's hard to say why they load slower. We've had a couple of releases with memory leaks (there aren't any known memory leaks right now) that slowed things down quite a bit.
Another gotcha has to do with global symbols in files that aren't directly included. This relates to the new feature that allows forward referencing, and also the improved method for resolving symbols and avoiding namespace conflicts. It actually simplifies coding, so that two files can rely on each other, and are able to use the full power of each without having to manage carefully which is included first, or moving the location of the include statements in each file, or requiring routine id and call_proc/func.
The problem arises when you use a library that declares lots of global symbols. Of course, this was the only way to write a library pre-4.0. But the real issues occur when you have source files that don't include the library. This is very common in win32lib apps (and, in fact, the IDE is probably one of the worst). Basically, the main file includes win32lib, and then includes lots of other application modules. Each of those modules uses stuff in win32lib, but never includes win32lib.
By simply adding "include win32lib.ew" to the top of each file, the parsing will speed up a lot. Plus, less memory will be used during parsing. I could talk a lot more about this, but that's enough for one post.
Matt
tl;dr The front end is a special case of translated code, and in 4.0, always include libraries that you use in each file.
4. Re: Euphoria dlls with multitasking
- Posted by useless Oct 03, 2009
- 1611 views
- Last edited Oct 04, 2009
Thanks for the explanation, Matt. I'd seen a bit of the discussion on tasking itself here on the forum but I was looking more for the source of the conflict between Kat and others and not finding it.
I posted the relavant emails in the other thread.
Basically, Jeremy said something about multitasking in dlls, because when i added a task_yield call to http.e, and was adding task_msg'ing too, it broke compiling http.e to a dll. And whatever he meant in that proposal, i understood him to mean something else, and i disagreed with the proposal i understood him to be making. Matt then went all Elizabot on me, and somehow thought i believed the proposal for future functionality was already present, and i could not and cannot find the words to disuade him from his belief.
In the meantime, no one understood how i understood Jeremy's original statement, and even after he clarified it some days later, and i thought that subject was dropped, it went on and on and on because i was still accused of believeing the proposal was actually how it already was, and as that made no sense to me, i didn't and don't understand how this was a basis for attacking me, and eventually i blew up.
As a different but slightly related side issue, i sped up news.ex some 10x faster than it was, and showed a progression of newsX.ex on how to take a slow app that almost wasn't really tasking at all, and make it really use the task system. In this progression of code, i left original code in news.ex as i was working on http.e and task.e at the same time, finding a code change to improve one often meant code needed changing to improve the other to keep pace. Matt had extreme issues, again believeing since i left in useless code i did not write, that i believed it was necessarily magical and required, and for the longest time he didn't even look at the 3 code bases i was working on, yet critiqued ME as well as the code.
This is why i am so peeved at the dev community in general, the assumptions forced on me were so illogical, yet persistantly pushed, there was no defending myself, because my words never got thru. (Except that Jeremy did have a clue what i meant was not what he meant and did clarify the proposal.) Even having two topics going was confusing to them: news3.ex and the misunderstood proposal. All i know is i was the butt of the attack, much like when i did post task msg'ing code, and Matt went after me, even admitting later he never took a look at the code, he coincidently left me alone after i took it down.
So i took down the task messaging handler code, and won't release news4, news5, the code to task_msg across a lan, or the fix to certain files nuking http.e, or further speedups to it. I don't need the harrassment, and i know any words i use to answer questions about it will not be heard or recieved in a fair light.
This is.... the 6th time (?) this has been gone over?
useless
5. Re: Euphoria dlls with multitasking
- Posted by jaygade Oct 03, 2009
- 1621 views
- Last edited Oct 04, 2009
Right. I saw some stuff here about the misunderstanding of how stuff worked, I just didn't see anything that I would constitute an attack or harassment.
I've just been wondering because it's a pretty serious accusation, IMO.
I remember Jeremy mentioning that you should add your improvements to news.ex and http.e to SVN repository.
Edit: Actually, until you had posted a quote from Jeremy, your posts about what "to" means were pretty meaningless to me. It helped clue me in a bit.
6. Re: Euphoria dlls with multitasking
- Posted by useless Oct 04, 2009
- 1621 views
Right. I saw some stuff here about the misunderstanding of how stuff worked, I just didn't see anything that I would constitute an attack or harassment.
I've just been wondering because it's a pretty serious accusation, IMO.
I remember Jeremy mentioning that you should add your improvements to news.ex and http.e to SVN repository.
Edit: Actually, until you had posted a quote from Jeremy, your posts about what "to" means were pretty meaningless to me. It helped clue me in a bit.
My "rights" to svn up contributions was withdrawn. Even tho Derek said in the other topic that i was still welcome, i am also not on the dev list.
useless
7. Re: Euphoria dlls with multitasking
- Posted by jeremy (admin) Oct 04, 2009
- 1586 views
My "rights" to svn up contributions was withdrawn. Even tho Derek said in the other topic that i was still welcome, i am also not on the dev list.
This was because you said you quit and you withdrew yourself from the dev list swearing off any future development ever. You asked/demanded that you not be a developer any longer, thus, you were removed from the SF.net project as well... all of that was in response to you leaving the Euphoria team as a developer, which you choose to do. Matt, Derek and I were sorry to see you leave, we did not ask you to leave or even come close to implying that you should. You did so on your own. Don't blame the dev team for kicking you off the SVN server.
Jeremy
8. Re: Euphoria dlls with multitasking
- Posted by useless Oct 04, 2009
- 1545 views
My "rights" to svn up contributions was withdrawn. Even tho Derek said in the other topic that i was still welcome, i am also not on the dev list.
This was because you said you quit and you withdrew yourself from the dev list swearing off any future development ever. You asked/demanded that you not be a developer any longer, thus, you were removed from the SF.net project as well... all of that was in response to you leaving the Euphoria team as a developer, which you choose to do. Matt, Derek and I were sorry to see you leave, we did not ask you to leave or even come close to implying that you should. You did so on your own. Don't blame the dev team for kicking you off the SVN server.
Jeremy
I am not contesting anything you said there, Jeremy, except the last line. The quote of me you gave didn't assign blame, and wasn't intended to. I put "rights" in quotes because it's not actual rights, it's a perk given to those in favor, like a popularity contest.
Allowing or creating a hostile environment that's enough to run off someone who wasn't there for the fighting aspects of contributing, is enough to assign blame. I signed up for contributing with no fighting. But instead i got fighting as the primary, over the contributing. No thanks.
It's as if you want to get rid of a pacifist, you put up pro-war posters and gun racks all over the office and hold the staff meetings at boxing matches. It's not your fault they left.
useless
9. Re: Euphoria dlls with multitasking
- Posted by Tabishi Oct 04, 2009
- 1563 views
My "rights" to svn up contributions was withdrawn. Even tho Derek said in the other topic that i was still welcome, i am also not on the dev list.
This was because you said you quit and you withdrew yourself from the dev list swearing off any future development ever. You asked/demanded that you not be a developer any longer, thus, you were removed from the SF.net project as well... all of that was in response to you leaving the Euphoria team as a developer, which you choose to do. Matt, Derek and I were sorry to see you leave, we did not ask you to leave or even come close to implying that you should. You did so on your own. Don't blame the dev team for kicking you off the SVN server.
Jeremy
I am not contesting anything you said there, Jeremy, except the last line. The quote of me you gave didn't assign blame, and wasn't intended to. I put "rights" in quotes because it's not actual rights, it's a perk given to those in favor, like a popularity contest.
Allowing or creating a hostile environment that's enough to run off someone who wasn't there for the fighting aspects of contributing, is enough to assign blame. I signed up for contributing with no fighting. But instead i got fighting as the primary, over the contributing. No thanks.
It's as if you want to get rid of a pacifist, you put up pro-war posters and gun racks all over the office and hold the staff meetings at boxing matches. It's not your fault they left.
useless
I don't think you understood what he said
This was because you said you quit and you withdrew yourself from the dev list swearing off any future development ever. You asked/demanded that you not be a developer any longer, thus, you were removed from the SF.net project as well... all of that was in response to you leaving the Euphoria team as a developer, which you choose to do. Matt, Derek and I were sorry to see you leave, we did not ask you to leave or even come close to implying that you should. You did so on your own. Don't blame the dev team for kicking you off the SVN server.
Jeremy
It was YOU who quit. Nobody did anything to you.
10. Re: Euphoria dlls with multitasking
- Posted by useless Oct 04, 2009
- 1535 views
My "rights" to svn up contributions was withdrawn. Even tho Derek said in the other topic that i was still welcome, i am also not on the dev list.
This was because you said you quit and you withdrew yourself from the dev list swearing off any future development ever. You asked/demanded that you not be a developer any longer, thus, you were removed from the SF.net project as well... all of that was in response to you leaving the Euphoria team as a developer, which you choose to do. Matt, Derek and I were sorry to see you leave, we did not ask you to leave or even come close to implying that you should. You did so on your own. Don't blame the dev team for kicking you off the SVN server.
Jeremy
I am not contesting anything you said there, Jeremy, except the last line. The quote of me you gave didn't assign blame, and wasn't intended to. I put "rights" in quotes because it's not actual rights, it's a perk given to those in favor, like a popularity contest.
Allowing or creating a hostile environment that's enough to run off someone who wasn't there for the fighting aspects of contributing, is enough to assign blame. I signed up for contributing with no fighting. But instead i got fighting as the primary, over the contributing. No thanks.
It's as if you want to get rid of a pacifist, you put up pro-war posters and gun racks all over the office and hold the staff meetings at boxing matches. It's not your fault they left.
useless
I don't think you understood what he said
This was because you said you quit and you withdrew yourself from the dev list swearing off any future development ever. You asked/demanded that you not be a developer any longer, thus, you were removed from the SF.net project as well... all of that was in response to you leaving the Euphoria team as a developer, which you choose to do. Matt, Derek and I were sorry to see you leave, we did not ask you to leave or even come close to implying that you should. You did so on your own. Don't blame the dev team for kicking you off the SVN server.
Jeremy
I don't think you understood what i said:
I am not contesting anything you said there, Jeremy, except the last line.
Are you starting a fight over that now? I didn't contest that, it's true, i wasn't arguing over that, it's not the topic of my post, why did you bring that up?
useless
11. Re: Euphoria dlls with multitasking
- Posted by Tabishi Oct 04, 2009
- 1557 views
because you keep accusing the hard working dedicated developers of persecuting you.
12. Re: Euphoria dlls with multitasking
- Posted by useless Oct 04, 2009
- 1603 views
- Last edited Oct 06, 2009
because you keep accusing the hard working dedicated developers of persecuting you.
Which is what happened. They can be the hardest working people on the entire planet, and make the best language on the planet, and still prefer to not have me there. That you merged the emotionally positive "hard working dedicated" to contrast the emotionally negative "persecuting" shows you cannot keep the two topics separate. I can't say they aren't hard working, because they are hardworking. No doubt about it.
Arguing over the smallest stuff, and unrelated to programming, should tell you there's nothing wrong with the code i submitted. But some just had to fight over something. Like you are doing now.
And AGAIN i think Eu is the best language for general use, but that doesn't mean i get along with the dev crew, or they with me. Until arguements are directed against the code, and not against me, i won't submit code. I suspect (and hope) eventually they'll find those types of files that crash http.e, and fix the problems in Eu that cause the crash. They are fixed in my code, and need to be fixed in the official release.
useless
