1. IL Engineering Library
- Posted by jimcbrown (admin) Mar 05, 2015
- 4147 views
Forked from Re: string_exec()
Which is why i suggested it be converted to IL code or compiled or whatever the wizardry du jour is, and the result inlined (or gosub'd) where it was called. That automatically restricts it to the same scope as the include file and the procedure containing it.
It really sounds like you do not want an eval() function at all. Instead, you want something like http://commons.apache.org/bcel/
Adding functionality to allow Euphoria IL to be directly modified (read&write) live, at run time, is simple enough. Dealing with raw IL is somewhat like dealing with raw Java bytecode (all numbers and no human readable words) - but we already have some tools like the shrouder and eudis which converts between the two. So with some simple changes, you could call the shrouder to compile your Euphoria code into IL, then inject that IL into your running program. This sounds like it'd do exactly what you are asking for, IIUC.
It'd go something like this:
include euphoria/shrouder.e -- new include based on bind.ex ? 0 sequence newIL = shrouder:shroud("atom a a = 1 ? a") integer pos = get_il_current_pos() -- new builtin inject_il_at(pos, newIL) -- new builtin ? 2
Output:
0 1 2
2. Re: IL Engineering Library
- Posted by katsmeow Mar 05, 2015
- 3922 views
Oooooooooooo , self modifying code! I've done that too, actually, but in machine code, many years ago. It saved me some 100 bytes of code, it was simply "this is the first time here", and "now we are exiting". It wasn't thread safe, but i very seldom ran multiple threads on the C64.
One small item about that, if it is permanently injected into the existing IL code that's running, the next string, or file, that needs evaluating/executing, likely will not be the same size, so perfectly overwriting it is unlikely. Can the injection operation accomodate different string sizes with each pass thru the code?
Kat
3. Re: IL Engineering Library
- Posted by jimcbrown (admin) Mar 05, 2015
- 3906 views
One small item about that, if it is permanently injected into the existing IL code that's running, the next string, or file, that needs evaluating/executing, likely will not be the same size, so perfectly overwriting it is unlikely. Can the injection operation accomodate different string sizes with each pass thru the code?
All Euphoria code is turned into IL before any runtime execution of code happens. So by the time the IL operation occurs, there are no more files to be included, and no more passes of code by the byte-code compiler.
Since there are no strings, this concern is not applicable.
4. Re: IL Engineering Library
- Posted by katsmeow Mar 05, 2015
- 3959 views
One small item about that, if it is permanently injected into the existing IL code that's running, the next string, or file, that needs evaluating/executing, likely will not be the same size, so perfectly overwriting it is unlikely. Can the injection operation accomodate different string sizes with each pass thru the code?
All Euphoria code is turned into IL before any runtime execution of code happens. So by the time the IL operation occurs, there are no more files to be included, and no more passes of code by the byte-code compiler.
Since there are no strings, this concern is not applicable.
Given:
string = "atom a\na+=1\n? a" -- 15 bytes existing code1 convert_and_inject(string) existing code2and then i do
string = getf("c:\\euphoria\\200bytes_of_code.eux") -- 200 bytes
existing code1
convert_and_inject(string)
existing code2
and then i do do it again
string = "atom a\na+=1\n? a" -- 15 bytes existing code1 convert_and_inject(string) existing code2and you are saying the string lengths do not matter and files don't exist? What?
The first run thru the code, the conversion and injection of 15 bytes of source is straight forward. But the 2nd time, the previous injected IL is wrong, string is now whatever was in that file, so the effects of the first injection need undoing, or overwriting. In the 3rd case, the 15 bytes won't overwrite the 200 bytes, but that 200 bytes is old and now wrong, and shouldn't be left in place. Yes, i know that the IL code isn't the same size as the source code, but i needed some numbers to toss around. Either a large block, with potentially a lot of NOPs must be allocated between existing code1 and existing code2, or it must be dynamically relocated each time string is IL'd and injected.
I hope i said it correctly this time.
Kat
5. Re: IL Engineering Library
- Posted by jimcbrown (admin) Mar 05, 2015
- 3966 views
Given:
string = "atom a\na+=1\n? a" -- 15 bytes existing code1 convert_and_inject(string) existing code2and then i do
string = getf("c:\\euphoria\\200bytes_of_code.eux") -- 200 bytes
existing code1
convert_and_inject(string)
existing code2
and then i do do it again
string = "atom a\na+=1\n? a" -- 15 bytes existing code1 convert_and_inject(string) existing code2and you are saying the string lengths do not matter
Yes, i know that the IL code isn't the same size as the source code, but i needed some numbers to toss around.
That's correct. String lengths do not matter, IL code lengths do. Lets assume then that the IL bytecode length is the same as the source code lengths. (This is obviously not true, the IL will be a lot shorter, but calculating the true lengths without a bytecode compiler would be a headache - at least for me.)
The first run thru the code, the conversion and injection of 15 bytes of source is straight forward. But the 2nd time, the previous injected IL is wrong, string is now whatever was in that file, so the effects of the first injection need undoing, or overwriting.
Naturally. The programmer has control here and has the ability to choose to overwrite, instead of inserting, IL code. The programmer will also have the ability to delete segments of IL to undo an injection operation.
Either a large block, with potentially a lot of NOPs must be allocated between existing code1 and existing code2, or it must be dynamically relocated each time string is IL'd and injected.
Currently, the former is true for the C-based bytecode interpreter. The Eu-based bytecode interpreter (eu.ex) stores the IL in a sequence.
6. Re: string_exec()
- Posted by jimcbrown (admin) Mar 06, 2015
- 3861 views
What I had envisioned includes the ability to modify true top-level code and then re-execute it.
My concern is that you'll effectively be in the middle of the top-level code:
-- this string_exec(...) -- that
If I understand your scheme, it would effectively become:
-- this string_exec(...) -- code generated by string_exec() <-- code begins re-executing here -- that
Is that correct?
That's effectively what happens in the example I posted, though I don't call it string_exec() anymore. However, I envisioned a greater level of control over this however, in that you could control exactly where the code generated is inserted, and also you could separately control where re-execution begins. They don't have to be the current position or even the same place at all.
If the request is allowing self-modifying code, what exactly does that mean? Can a routine be redefined?
Yes.
It seems you can get the same result via routine_id, at a much cheaper cost.
- David
How does one redefine a routine with routine_id() ?
I had envisioned the ability to rewrite the code of a pre-existing routine on the fly, to make it behave completely different.
7. Re: string_exec()
- Posted by jmduro Mar 12, 2015
- 3772 views
This makes me think of CA-Realizer, a Basic language of the 90's, which was able to build and run source code dynamically. It was a Windows-based interpreter using IL code as far as I know (I might be wrong).
As a result someone wrote a Realizer interpreter written in CA-Realizer and released it to public domain which hurted the license of this commercial software. It was known as PC-CA2, based on CA-Realizer 2. I built the same self-interpreter for my own usage as a regular licensee. As far as I remember it took less then 100 lines of code.
I still own CA-Realizer 2 which runs even on Seven 64 bits but builds applications with a Windows 95 look. Not very nice today, but it works! It had some kind of Win32Lib included.
Reagrds
Jean-Marc
8. Re: string_exec()
- Posted by katsmeow Mar 12, 2015
- 3754 views
This makes me think of CA-Realizer, a Basic language of the 90's, which was able to build and run source code dynamically. It was a Windows-based interpreter using IL code as far as I know (I might be wrong).
As a result someone wrote a Realizer interpreter written in CA-Realizer and released it to public domain which hurted the license of this commercial software. It was known as PC-CA2, based on CA-Realizer 2. I built the same self-interpreter for my own usage as a regular licensee. As far as I remember it took less then 100 lines of code.
I still own CA-Realizer 2 which runs even on Seven 64 bits but builds applications with a Windows 95 look. Not very nice today, but it works! It had some kind of Win32Lib included.
Reagrds
Jean-Marc
I cannot find it for sale or download online. It looks like orphanware circa 1996. Any idea how i can get a copy of it to try out?
Kat
9. Re: IL Engineering Library
- Posted by jimcbrown (admin) Apr 18, 2015
- 3581 views
I've implemented the low-level part of this. See http://scm.openeuphoria.org/hg/euphoria/rev/e5aa21896e3f
10. Re: IL Engineering Library
- Posted by jimcbrown (admin) Apr 18, 2015
- 3513 views
See also http://scm.openeuphoria.org/hg/euphoria/rev/b9cba54d622a for a very primitive demo.
11. Re: IL Engineering Library
- Posted by Rasti Jul 09, 2015
- 3305 views
I'm interested in Euphoria's self-modifying code capabilities.
Jim, could you please elaborate a bit further on Euphoria's potential in this area? For instance, in addition to compiling code at runtime and injecting it into the running program, on the fly, could Euphoria also eliminate from memory some parts of bytecode that are no longer needed? Can the routine_id indentification be used for this purpose, to reference the chunks of bytecode to be injected and those to be eliminated?
12. Re: IL Engineering Library
- Posted by jimcbrown (admin) Jul 09, 2015
- 3311 views
I'm interested in Euphoria's self-modifying code capabilities.
Right now, it doesn't really have any... except in the highly experimental branch.
Jim, could you please elaborate a bit further on Euphoria's potential in this area?
Sure thing, but remember that what's implemented so far is still in the highly experimental phase, and only implemented for the euphoria backend.
For instance, in addition to compiling code at runtime and injecting it into the running program, on the fly, could Euphoria also eliminate from memory some parts of bytecode that are no longer needed?
This is already implemented in the equals branch as M_DELETE_IL_AT. See http://scm.openeuphoria.org/hg/euphoria/rev/e5aa21896e3f
Can the routine_id indentification be used for this purpose, so that chunks of bytecode are injected while others are eliminated?
Yes, it could. This is not implemented yet, but in the euphoria backend it's as easy as returning something like SymTab[e_routine[val[a]+1]][S_CODE].
13. Re: IL Engineering Library
- Posted by Rasti Jul 09, 2015
- 3281 views
This is impressive. Is that code in the experimental branch already usable? And when are these features likely to be implemented in the main branch? Six months/one year?
14. Re: IL Engineering Library
- Posted by jimcbrown (admin) Jul 09, 2015
- 3270 views
This is impressive. Is that code in the experimental branch already usable?
Depends on what you mean by "usable". 
There is a single, working demo.
And when are these features likely to be implemented in the main branch? Six months/one year?
Don't know. It's possible that this feature will never get merged in - the purpose of the branch was to provide a playground to try out new ideas and encourage experimentation. It's there to enhance the discussion of what new features should be added, not as a staging ground to handle merging.
There hasn't really been a whole lot of interest from the community in having an IL Engineering Library for Euphoria. No one has even proposed holding a vote on it.
15. Re: string_exec()
- Posted by jmduro Jul 10, 2015
- 3300 views
I cannot find it for sale or download online. It looks like orphanware circa 1996. Any idea how i can get a copy of it to try out?
Kat
I beg your pardon, Kat. I missed your post. CAR 3.0 is available on Vetusware: http://www.vetusware.com/download/CA%20Realizer%203.0%203.0/?id=6888
Regards
Jean-Marc
16. Re: string_exec()
- Posted by jmduro Jul 11, 2015
- 3220 views
To build the free interpreter fom the commercial one, it was as simple as this with CA-Realizer 2(first line untested)
SourceFile = QSys(_CmdLine) FileOpen(1, SourceFile) LOOP IF NOT FileEOF(1) THEN FileRead(1, s) ELSE EXIT LOOP END IF IF Len(s) > 0 THEN EXECUTE s END IF END LOOP FileClose(1)
People had just to compile this as an executable to build a new interpreter.
17. Re: IL Engineering Library
- Posted by Rasti Jul 16, 2015
- 3216 views
- Last edited Jul 17, 2015
Sorry to quote from a very old (2004) post [1].
You still won't be able to create new routines at runtime and have them execute in the same address space as the application (eg. have them share variables). With using routine-id, you can only deal with routines that already exist at runtime. There is a convoluted method of doing this in WIndows/Linux environments, but it involves compiling a new routine as a shared library (eg. a Windows DLL) at runtime then loading that library, then running the routine - not a pretty method at all.
@Jim: I was wondering, how was your IL Engineering Library able to overcome the limitations mentioned by Derek? And are the new routines created at runtime actually sharing the same address space as the rest of the program?
18. Re: IL Engineering Library
- Posted by jimcbrown (admin) Jul 20, 2015
- 3122 views
Sorry to quote from a very old (2004) post [1].
You still won't be able to create new routines at runtime and have them execute in the same address space as the application (eg. have them share variables). With using routine-id, you can only deal with routines that already exist at runtime. There is a convoluted method of doing this in WIndows/Linux environments, but it involves compiling a new routine as a shared library (eg. a Windows DLL) at runtime then loading that library, then running the routine - not a pretty method at all.
@Jim: I was wondering, how was your IL Engineering Library able to overcome the limitations mentioned by Derek?
Actually, your quote shows that Derek already knew of a way of overcoming those limitations.
Meanwhile, my example implementation does not support creating new routines at runtime yet. This would require hooking into the routine_id table and stuff, which can be done, but simply has not been implemented yet.
And are the new routines created at runtime actually sharing the same address space as the rest of the program?
Yes.
