1. string_exec()
- Posted by petelomax Feb 28, 2015
- 2542 views
- Last edited Mar 01, 2015
An old idea, but I thought it is about time I at least got a start on some design.
Implementation may be some time off, this is all just some preliminary musings, and not particularly well thought through at that, but as I'm currently doing some design work on the optable (not that you need to know or care what that is), now seems as good a time as any to consider this. I haven't seen any examples that make me think "you know what, that's a really good idea", so now is your chance.
Obviously this has something of a Phix bias that may need ironing out. It must (somehow) include a complete copy of the compiler/interpreter, or re-use the one already running. I imagine OpenEuphoria might use eu.ex, the hll interpreter.
Firstly, there is no security model whatsoever. If you put this on a web server and let visitors type in and execute code, you may as well just give them the root password. However, if this is an app they have downloaded and installed locally, it is, literally, the same deal as installing a compiler. All I am really saying is that I have no plans to go down the Java route of 12 security patches a day from now until the end of time; if someone else wants that then they can jolly well do it themselves! That said, I would support an optional "draconian safe mode" whereby such things as file i/o, system_exec, inline assembly, etc are all completely prohibited inside string_exec() - if someone else takes the time to carefully sift through the manual and compose a comprehensive list of all potential dangers.
Let's begin simple:
integer i = 5 ctrl = string_exec(ctrl,"i+=1")
While (on Phix) when interpreted there might be a few salvage-able bits of lookup and the possibility of extending the symbol table, when compiled we have:
- original symbol table (damaged, fixed size, non-extendable, no lookup)
- static data block (fixed size, non-extendable)
- executable code (fixed size, non-extendable)
So during string_exec(), we will need:
- a completely rebuilt symbol table, extendable, with rebuilt lookup
- original and new static data blocks, with index ranges (lists of)
- original and new executable code blocks (lists of)
Such things could be preserved in ctrl1 so that subsequent invocations are not quite so expensive, even if that might only be half a second per call to string_exec, and for reasons that should become clear soon.
Question: Are we hoping for something like this?
integer i = 5 ctrl = string_exec(ctrl,"integer j\n"& "function add1(integer i)"& " return i+1"& "end function") ctrl = string_exec(ctrl,"j = add1(i)")
In other words, are the rebuilt and extended symbol table, new data, and new code all available for use in subsequent invocations of string_exec()? Or should the symbol table be reset for each invocation and any temporary code/data discarded? Or do we need to cater for both? [Examples to backup any claims, please.]
Next question: What sort of context are we looking for?
procedure p() integer i=5 ctrl = string_exec(ctrl,"i+=1") end procedure
Should scope be as per EOF of main file or at the point of invoking string_exec? The latter is much harder - and if there are multiple string_exec calls, we might need different scopes for each. I would need some blinding examples to convince me that is worthwhile.
What error messages (compile-time and run-time) if any do we really need?
What sort of clean-up (deallocating temps, freeing allocated memory, closing files, etc) do we really need? As per the add1 example above, we could go too far and trash something we might want later.
Is there any need for expression results?
integer i = 5, j j = string_expr(ctrl,"i") -- (or maybe -- ctrl = string_expr(ctrl,"i") -- j = ctrl[RESULT])
Note that you couldn't define any vars/funcs, unless you made "return <expr>" at top level legal in string expressions. Again, I'd need convincing examples.
We probably ought to be able to "nest" string evaluation, though I hope someone can provide a much better example than this:
j = string_expr(ctrl,"string_expr(ctrl,\"i\")")
I can see a use for "standalone" string evaluation, that starts with the basic "empty" symbol table2, such as:
a = string_expr(NULL,"5+4/3")
which might be quite useful in say a calculator, anything else? (Obviously that assumes support for expression results, but we could achieve equivalent functionality via something like "res=5+4/3")
Also, it is not particularly clear what command_line() should return when it is invoked from inside a string_exec().
This is far from over, but I think that's probably enough for now, plus I just spotted a major flaw in nested opInterp/thread safety that I must go and fix asap.
Please try to provide examples that I have a vague chance of understanding
Pete
1 the content and structure of ctrl is deliberately left vague and undefined.
2 maybe: string_expr(NULL,...)=>empty; string_expr({},...)=>rebuild, or vice versa.
2. Re: string_exec()
- Posted by dcuny Feb 28, 2015
- 2524 views
I'd assume that there string code would be running in it's own context.
It would be nice if there were a way to pass values to that context, such as:
-- compile code into a context {aContext, resultCode} = compile_code("myfile.ex") -- set top-level value for variable "some_string" to "a_value" {aContext, resultCode} = set_value(aContext, "some_string", "a value") -- execute procedure in context {aContext, resultCode} = call_routine(aContext, "some_procedure", {1,2,3}) -- execute function in context {aContext, returnValue, resultCode} = call_routine(aContext, "some_function", {1,2,3}) -- create an empty context that code can be compiled to sequence newContext = create_context() -- compile code into the context {newContext, resultCode} = compile(newContext, "integer a\na=22") -- retrieve top-level "a" value from the context {result, resultCode} = get_value(newContext, "a")
I'd also assume that if there was an error while executing something in the context, there would be routines for retrieving that information.
- David
3. Re: string_exec()
- Posted by Ekhnat0n Feb 28, 2015
- 2521 views
Great idea, but dangerous if not really well safe-guarded against any hacking possibilities.
Would be quite hard to write that all in assembler though
as you know I am currently working on a basic set
of commands to do general things like
Do......while
Loop....Until
If....Then===== Elsif======(etc) Endif
Case x .... y..... etc Endcase
Put {screen/file/mem} & Get {ditto}
all in native ASM for 68K, 8086,8086-64 and ARM
4. Re: string_exec()
- Posted by katsmeow Feb 28, 2015
- 2526 views
Please try to provide specific examples to back up any comments or claims.
I have no clue to the structure of compiled OE, but what if the line containing the string_eval() caused a Non-Fatal interrupt, the line was evaluated to be a valid string_eval(), it (or the file it mentions) is compiled and then inlined (rather more of a local gosub in machine code) in place of the line which caused the interrupt, and then program flow resumes normally from the point of interrupt?
Again, from a clueless perspective, this smells like it will cause the least headaches in var/procedure scopes, as well as goto and name scopes (include balh.e as balh). No new environments would need to be built, no massive copying of var tables, permission slips, colors, or battery voltages. If the string crashes because it assumes something blah blah blah, then it's a normal blah blah blah program crash.
A working code example:
.timer 1 %defflooddelay %temptext
Kat
5. Re: string_exec()
- Posted by jimcbrown (admin) Feb 28, 2015
- 2516 views
Please try to provide specific examples to back up any comments or claims.
I have no clue to the structure of compiled OE,
No problem. It should suffice to talk about the structure of eu.ex and execute.e
but what if the line containing the string_eval() caused a Non-Fatal interrupt, the line was evaluated to be a valid string_eval(), it (or the file it mentions) is compiled and then inlined (rather more of a local gosub in machine code) in place of the line which caused the interrupt, and then program flow resumes normally from the point of interrupt?
So basically, if a NFI occurs, but it happens to a statement that's usable inside a string_eval() call, then replace the line with a string_eval() call to the original statement.
So, assuming that division by zero is an NFI, when we execute
? 1/0
and hit the NFI, we rollback and replace it with
string_eval("? 1/0")
Presumably, calls to string_eval() that throw an error will not crash the main program, but only stop execution of the eval'd code and set some kind of error flag that can be checked later.
In short, you are suggesting that we use string_eval() to implement "ON ERROR GOTO NEXT" error handling behavior, with perhaps some minor differences.
The basic concept is sound.
Again, from a clueless perspective, this smells like it will cause the least headaches in var/procedure scopes, as well as goto and name scopes (include balh.e as balh). No new environments would need to be built, no massive copying of var tables, permission slips, colors, or battery voltages. If the string crashes because it assumes something blah blah blah, then it's a normal blah blah blah program crash.
A working code example:
.timer 1 %defflooddelay %temptext
Kat
Ok, so if code inside of a string_eval() string raises an error, we treat it exactly the same as if it had happened in the top-level scope. E.g. If we have try-catches, then we'd throw the exception from string_eval() up to the caller, and so on.
If we have try-catches or some other sane error handling method, this works. Right now, due to the issues dcuny raised previously in his thread about try/catch, I believe that string_eval() should not do this. I do not think that an error in string_eval() should shut down the entire program and generate an ex.err file. But once we implement better error handling, then this can be changed too.
6. Re: string_exec()
- Posted by petelomax Mar 01, 2015
- 2465 views
I'd assume that there string code would be running in it's own context.
That certainly makes life easier, but I suspect the requirement to obtain/set stuff in the host will turn up soon enough.
It would be nice if there were a way to pass values to that context, such as:
-- compile code into a context {aContext, resultCode} = compile_code("myfile.ex") -- set top-level value for variable "some_string" to "a_value" {aContext, resultCode} = set_value(aContext, "some_string", "a value") -- execute procedure in context {aContext, resultCode} = call_routine(aContext, "some_procedure", {1,2,3}) -- execute function in context {aContext, returnValue, resultCode} = call_routine(aContext, "some_function", {1,2,3}) -- create an empty context that code can be compiled to sequence newContext = create_context() -- compile code into the context {newContext, resultCode} = compile(newContext, "integer a\na=22") -- retrieve top-level "a" value from the context {result, resultCode} = get_value(newContext, "a")
I was thinking that set_value and get_value might not be as powerful and flexible as simple assignments/return expressions. How about this, to achieve the same result:
ctrl = string_exec(NULL,"include myfile.ex\n"& "some_string = \"a_value\"\n"& "some_procedure(1,2,3)\n"& "return some_function(1,2,3)") returnValue = ctrl[RESULT] newContext = string_exec(NULL,"integer a\na=22\nreturn a") result = newContext[RESULT]
or, also equally:
ctrl = string_exec(NULL,"include myfile.ex") ctrl = string_exec(ctrl,"some_string = \"a_value\"") ctrl = string_exec(ctrl,"some_procedure(1,2,3)") {context,returnValue,errors} = string_exec(ctrl,"return some_function(1,2,3)") newContext = string_exec(NULL,"integer a") newContext = string_exec(newContext,"a=22") {newContext,result,errors} = string_exec(newContext,"return a")
I'd also assume that if there was an error while executing something in the context, there would be routines for retrieving that information.
Sure. I've no problem with or real preference for either ctrl = string_exec() and retrieving by index, or {context,value,error} = string_exec() at this stage1.
It also strikes me that a simple (integer) error is unlikely to be sufficient, instead {code snippet,line/col,human-readable message,...}, not unlike what we get in an ex.err.
Oh, yes, I see what you were asking now, the context/ctrl is it - everything you could ever possibly need must be available somewhere within it. Details tba.
Pete
1 Random petty detail: by defining context[1] to be a specific integer, say #0C057347, string_exec() could easily cope with getting context or {context,value,error} passed back to it. In other words, in a loop, both ctrl=string_exec(ctrl,s) and {context,value,error}=string_exec(context,s) would be fine. Right now the latter looks better, but needing to throw these things about might tip things in favour of the former.
7. Re: string_exec()
- Posted by petelomax Mar 01, 2015
- 2452 views
Please try to provide examples that I have a vague chance of understanding
8. Re: string_exec()
- Posted by jimcbrown (admin) Mar 01, 2015
- 2436 views
Please try to provide examples that I have a vague chance of understanding
Can you explain what's so hard to understand about this?
So, assuming that division by zero is an NFI, when we execute ? 1/0 and hit the NFI, we rollback and replace it with string_eval("? 1/0")
9. Re: string_exec()
- Posted by petelomax Mar 01, 2015
- 2490 views
Please try to provide examples that I have a vague chance of understanding
Can you explain what's so hard to understand about this?
So, assuming that division by zero is an NFI, when we execute ? 1/0 and hit the NFI, we rollback and replace it with string_eval("? 1/0")
Well, presumably it is an NFI because it is inside a string_eval, so if we rollback (whatever that is) and replace it with a string_eval, it will cause an NFI which presumably we will rollback (whatever that is) and replace with a string_eval, which will cause an NFI which we can replace with a string_eval that causes an NFI that we replace with a string evalthatcausesanNFIthatwereplacewithastringevalthat...
What is the point?
Pete
10. Re: string_exec()
- Posted by jimcbrown (admin) Mar 01, 2015
- 2466 views
Well, presumably it is an NFI because it is inside a string_eval,
Ah, I was under the impression that this was a statement not originally inside a string_eval. Perhaps I misunderstood something.
Right now we have a lot of errors in OE that are fatal that don't have to be. Some things, like an OOM perhaps, would probably stay fatal no matter how much error handling changes, but others, like floating point exceptions, could be changed to become NFIs.
so if we rollback (whatever that is) and replace it with a string_eval, it will cause an NFI which presumably we will rollback (whatever that is) and replace with a string_eval, which will cause an NFI which we can replace with a string_eval that causes an NFI that we replace with a string evalthatcausesanNFIthatwereplacewithastringevalthat...
What is the point?
There wouldn't be much point in that.
11. Re: string_exec()
- Posted by dcuny Mar 01, 2015
- 2429 views
My assumption was that the string execution would be implemented via the current Euphoria interpreter. The context would be all the information the Euphoria-to-opcode program generates.
If the context is re-entrant and the data structure can be appended to, that would be awesome.
But I really don't know enough about the compiler to offer any suggestions here.
I've been kicking around the idea of eventually writing my own "assembler" for Euphoria, which would look like:
Euphoria source -> assembly -> context (opcodes + dictionary)
... but like everyone else, I haven't gotten around to it yet.
- David
12. Re: string_exec()
- Posted by katsmeow Mar 01, 2015
- 2408 views
Please try to provide examples that I have a vague chance of understanding
Can you explain what's so hard to understand about this?
So, assuming that division by zero is an NFI, when we execute ? 1/0 and hit the NFI, we rollback and replace it with string_eval("? 1/0")
Well, presumably it is an NFI because it is inside a string_eval, so if we rollback (whatever that is) and replace it with a string_eval, it will cause an NFI which presumably we will rollback (whatever that is) and replace with a string_eval, which will cause an NFI which we can replace with a string_eval that causes an NFI that we replace with a string evalthatcausesanNFIthatwereplacewithastringevalthat...
What is the point?
Pete
No. What i was saying is the "string_eval(", is now a fatal error, you make it a non-fatal error. It is just one way to hook into runtime smarts like an interpreter should have. If it isn't a valid syntax, you could always progress to being fatal. Not that any error in a string should automatically be fatal.
If you have no other way of adding "string_eval()" to the keyword list, or other way of using runtime interpreter smarts to process "string_eval(", then maybe it's best to trip the interpreter via an NFI and to execute runtime code to simply strictly dynamically generate and replace the portion of the line containing "string_eval(blargboroo)" with whatever the interpreter would prefer to execute. Because really, when the code begins running, you have no clue what is in the string (or file, or console input, or table cell), so you cannot check anything in the string, which may indeed be null at the time.
Kat
13. Re: string_exec()
- Posted by jimcbrown (admin) Mar 01, 2015
- 2410 views
What i was saying is the "string_eval(", is now a fatal error,
What i was saying is the "string_eval(", ... you make it a non-fatal error.
Huh? Which one should it be?
14. Re: string_exec()
- Posted by katsmeow Mar 01, 2015
- 2398 views
What i was saying is the "string_eval(", is now a fatal error,
What i was saying is the "string_eval(", ... you make it a non-fatal error.
Huh? Which one should it be?
This is where i bang my head against a wall, because it's as if i am speaking a non-earthling language to you. Let me ask a couple of questions in an effort to discern where communications have broken down.
Is it true that right now any attempt to use the words "string_eval" as a built-in OE function or procedure is a fatal error?
Is it true that regardless of the current situation regarding OE and the words "string_eval", it is possible to flag that phrase as non-fatal?
Is it possible that a directive to do something, using the explicit phrase "you make it a non-fatal error" is understood to be a statement that it's best to not do the directive?
Kat
15. Re: string_exec()
- Posted by jimcbrown (admin) Mar 01, 2015
- 2397 views
Is it true that right now any attempt to use the words "string_eval" as a built-in OE function or procedure is a fatal error?
Yes.
Is it true that regardless of the current situation regarding OE and the words "string_eval", it is possible to flag that phrase as non-fatal?
No.
Is it possible that a directive to do something, using the explicit phrase "you make it a non-fatal error" is understood to be a statement that it's best to not do the directive?
Only if it is misread/misheard. But unfortunately that type of situation is quite common.
16. Re: string_exec()
- Posted by katsmeow Mar 01, 2015
- 2402 views
Is it true that regardless of the current situation regarding OE and the words "string_eval", it is possible to flag that phrase as non-fatal?
No.
Regardless of the current situation regarding OE and the words "string_eval", is it possible to flag a phrase as non-fatal?
Kat
17. Re: string_exec()
- Posted by jimcbrown (admin) Mar 01, 2015
- 2389 views
Is it true that regardless of the current situation regarding OE and the words "string_eval", it is possible to flag that phrase as non-fatal?
No.
Regardless of the current situation regarding OE and the words "string_eval", is it possible to flag a phrase as non-fatal?
Kat
No.
18. Re: string_exec()
- Posted by katsmeow Mar 01, 2015
- 2393 views
Is it true that regardless of the current situation regarding OE and the words "string_eval", it is possible to flag that phrase as non-fatal?
No.
Regardless of the current situation regarding OE and the words "string_eval", is it possible to flag a phrase as non-fatal?
Kat
No.
How is it that the phrase "puts" is non-fatal?
Kat
19. Re: string_exec()
- Posted by jimcbrown (admin) Mar 01, 2015
- 2378 views
Is it true that regardless of the current situation regarding OE and the words "string_eval", it is possible to flag that phrase as non-fatal?
No.
Regardless of the current situation regarding OE and the words "string_eval", is it possible to flag a phrase as non-fatal?
Kat
No.
How is it that the phrase "puts" is non-fatal?
Kat
False premise. By itself, that phrase causes a fatal error.
<0132>:: Syntax error - expected to see possibly '(', not the end of file <end-of-file> ^
Anyways, statements are either fatal or not. But there's no way to flag one kind into the other.
20. Re: string_exec()
- Posted by katsmeow Mar 01, 2015
- 2392 views
Ok, i give up, completely. Have it your way, as usual. String execution isn't possible.
Kat
21. Re: string_exec()
- Posted by jimcbrown (admin) Mar 01, 2015
- 2385 views
Ok, i give up, completely.
I respect your right and choice to withdraw from this conversation. In general though, I suggest that - unless someone is requesting that one posts a response - the best way to withdraw is simply not to post a reply at all.
Have it your way, as usual. String execution isn't possible.
A friendly reminder: Putting words in my mouth is a CodeOfConduct violation...
String execution isn't possible.
It's definitely possible, since other languages exist that do it.
22. Re: string_exec()
- Posted by katsmeow Mar 01, 2015
- 2392 views
It's definitely possible, since other languages exist that do it.
True, but i have been asking for the feature in Euphoria for over a decade.
useless
23. Re: string_exec()
- Posted by dcuny Mar 01, 2015
- 2424 views
Regardless of the current situation regarding OE and the words "string_eval", is it possible to flag a phrase as non-fatal?
I don't see why you'd want string_eval to throw an exception unless there was a mechanism to catch it with.
Even then, the exception would have happened within the context, not the caller.
The point of returning a result code is to interrogate the result of string_eval.
Why would you want string_eval throw a fatal exception in the calling routine? What's the benefit?
- David
24. Re: string_exec()
- Posted by mattlewis (admin) Mar 01, 2015
- 2409 views
No. What i was saying is the "string_eval(", is now a fatal error, you make it a non-fatal error. It is just one way to hook into runtime smarts like an interpreter should have. If it isn't a valid syntax, you could always progress to being fatal. Not that any error in a string should automatically be fatal.
I'm not quite clear on what you're trying to say here. Do you mean that errors in dynamically generated code should result in an error code returned (or, if we have structured exception handling, one of those)?
If you have no other way of adding "string_eval()" to the keyword list, or other way of using runtime interpreter smarts to process "string_eval(",
I'm completely lost at this point.
Matt
25. Re: string_exec()
- Posted by mattlewis (admin) Mar 01, 2015
- 2381 views
I don't see why you'd want string_eval to throw an exception unless there was a mechanism to catch it with.
Even then, the exception would have happened within the context, not the caller.
The point of returning a result code is to interrogate the result of string_eval.
Why would you want string_eval throw a fatal exception in the calling routine? What's the benefit?
As I look at it, we're talking about calling some code. The only difference here is that the code wasn't parsed, etc, at the beginning. So the string_eval() call is just another part of the call stack, so (assuming exceptions have been implemented) an exception should bubble up to the caller like any other function call.
Unless we're talking about this as something sandboxed away from the current program. I don't think that's what Kat wants, since I believe she wants to interact with the data and code of the program that's executing the dynamic code.
Matt
26. Re: string_exec()
- Posted by jimcbrown (admin) Mar 01, 2015
- 2384 views
Unless we're talking about this as something sandboxed away from the current program.
That's the easiest version to implement though. Just do it eueval-style (aka turning eu.ex into a library). (And even here, eueval5.0 could easily be modified to recreate and throw actual exceptions to the caller, if a fatal or uncaught exception was detected in the simulated code.)
27. Re: string_exec()
- Posted by andi49 Mar 01, 2015
- 2399 views
Unless we're talking about this as something sandboxed away from the current program.
That's the easiest version to implement though. Just do it eueval-style (aka turning eu.ex into a library). (And even here, eueval5.0 could easily be modified to recreate and throw actual exceptions to the caller, if a fatal or uncaught exception was detected in the simulated code.)
From my point of view this is the only way to implement!
Eval in the current program is something i do not want to even think about.
(eval needs to live in a Sandbox and nowhere else!)
Thank you
Andreas
Forked into: Multiple interpreter instances
28. Re: string_exec()
- Posted by dcuny Mar 01, 2015
- 2377 views
Unless we're talking about this as something sandboxed away from the current program. I don't think that's what Kat wants, since I believe she wants to interact with the data and code of the program that's executing the dynamic code.
Yes, I'm suggesting that we sandbox the code. That's what the context is.
If Kat is looking for a use case where she can call code dynamically, there are other ways this could approached. But this wouldn't work well for that.
- David
29. Re: string_exec()
- Posted by mattlewis (admin) Mar 01, 2015
- 2416 views
Unless we're talking about this as something sandboxed away from the current program.
That's the easiest version to implement though. Just do it eueval-style (aka turning eu.ex into a library). (And even here, eueval5.0 could easily be modified to recreate and throw actual exceptions to the caller, if a fatal or uncaught exception was detected in the simulated code.)
Absolutely. You can use ooeu that way. I used that for wxedb. But that's not what Kat's talking about when she asks for an eval function.
Matt
30. Re: string_exec()
- Posted by katsmeow Mar 01, 2015
- 2392 views
Unless we're talking about this as something sandboxed away from the current program. I don't think that's what Kat wants, since I believe she wants to interact with the data and code of the program that's executing the dynamic code.
Yes, I'm suggesting that we sandbox the code. That's what the context is.
If Kat is looking for a use case where she can call code dynamically, there are other ways this could approached. But this wouldn't work well for that.
- David
There is no point in executing a string outside the context of exactly where it was called in the code. To place the string outside the context of where the eval() was called, is to simply run another program. We can do that now. So sandboxing eval() does me no good. I can run other OE programs already.
Security is not an issue, i can run the application with an airgap between the computers and the internet. I can filter what goes into the string. I can filter what is in the string before it is eval()'d.
Some of you are standing against what i ask for because it is not good for you. Then don't use it, just like you don't use more than 50% of what is already in OE. You can always disable it, compile without it, or override it.
Here's a thought: i wrote Tiggr many years ago, the syntax parser was working ~1993. Tiggr uses string execution. One week, maybe 10 years ago, i downloaded the entire RDS archive, unzipped every file, multiply indexed every file, munged every index, plugged the results into Tiggr for intelligent searching. Guess why that feature is not on the OE website? Because Tiggr uses string execution, and access to the var table, which means she isn't written in OE, and that is unacceptable.
useless
31. Re: string_exec()
- Posted by mattlewis (admin) Mar 01, 2015
- 2399 views
There is no point in executing a string outside the context of exactly where it was called in the code. To place the string outside the context of where the eval() was called, is to simply run another program. We can do that now. So sandboxing eval() does me no good. I can run other OE programs already.
There absolutely is a point. You can provide integration points where the internal code can interface with the main program. Think of a plugin, or user scripts in a game. There are many reasons why security could still be an issue there. You might also want to have multiple contexts that do not interfere with each other.
Here are the oeu docs on embedding an interpreter inside your program:
http://ooeu.sourceforge.net/docs/EMBEDDING.htm
Matt
32. Re: string_exec()
- Posted by katsmeow Mar 01, 2015
- 2427 views
There is no point in executing a string outside the context of exactly where it was called in the code. To place the string outside the context of where the eval() was called, is to simply run another program. We can do that now. So sandboxing eval() does me no good. I can run other OE programs already.
There absolutely is a point. You can provide integration points where the internal code can interface with the main program. Think of a plugin, or user scripts in a game. There are many reasons why security could still be an issue there. You might also want to have multiple contexts that do not interfere with each other.
Here are the oeu docs on embedding an interpreter inside your program:
http://ooeu.sourceforge.net/docs/EMBEDDING.htm
Matt
I thought there were runtime problems in ooeu, that built up over time, like memory leaks? And errors related to more than one operation per line? Plus the small detail that OOEU is RDS v2.5?
useless
33. Re: string_exec()
- Posted by jimcbrown (admin) Mar 02, 2015
- 2433 views
Absolutely. You can use ooeu that way. I used that for wxedb. But that's not what Kat's talking about when she asks for an eval function.
Understood, but I agree with dcuny here:
If Kat is looking for a use case where she can call code dynamically, there are other ways this could approached. But this wouldn't work well for that.
34. Re: string_exec()
- Posted by jimcbrown (admin) Mar 02, 2015
- 2372 views
Some of you are standing against what i ask for because it is not good for you.
I haven't seen anyone speak against eval() or string execution per se in this thread. I haven't seen anyone speak against calling code dynamically per se either. It was merely pointed out that these are two separate things, and linking them together in the implementation isn't necessarily the best way do implement them.
But, the best way to prove those naysayers (such as myself) wrong is to prove it with a working implementation!
Then don't use it, just like you don't use more than 50% of what is already in OE. You can always disable it, compile without it, or override it.
Agreed in full.
Here's a thought: i wrote Tiggr many years ago, the syntax parser was working ~1993. Tiggr uses string execution. One week, maybe 10 years ago, i downloaded the entire RDS archive, unzipped every file, multiply indexed every file, munged every index, plugged the results into Tiggr for intelligent searching. Guess why that feature is not on the OE website?
Because RDS kept control over the archives. So even copy-cats can't get this hosted here. I have high hopes that ryanj's proposal at http://openeuphoria.org/forum/m/126584.wc will be able to support something akin to this though.
Because Tiggr uses string execution, and access to the var table, which means she isn't written in OE, and that is unacceptable.
Actually, Tiggr was requested to become part of the website: http://openeuphoria.org/forum/m/108309.wc
Edit: It is true that we generally prefer the website to use OE whenever possible. However, due to its historical ties to the Euphoria community, an exception would be made for a 100% mIRC Tiggr.
35. Re: string_exec()
- Posted by jimcbrown (admin) Mar 02, 2015
- 2408 views
I thought there were runtime problems in ooeu, that built up over time, like memory leaks? And errors related to more than one operation per line? Plus the small detail that OOEU is RDS v2.5?
Which is why I haven't advocated OOEU as the official new eueval library.
No matter which way you look at it, a new implementation is going to be necessary.
36. Re: string_exec()
- Posted by mattlewis (admin) Mar 02, 2015
- 2360 views
I thought there were runtime problems in ooeu, that built up over time, like memory leaks? And errors related to more than one operation per line? Plus the small detail that OOEU is RDS v2.5?
I don't have any specific memories of memory leaks, but it wouldn't totally surprise me. It started as 2.5, and I think that some things got upgraded, but it's definitely an older dialect.
Still, it works now for what it is, and I was mostly bringing it up to show how stuff worked there: a completely sand boxed implementation with the ability to specify integration points.
Matt
37. Re: string_exec()
- Posted by petelomax Mar 04, 2015
- 2261 views
Not that any of this is likely to happen any time soon, but what I've gathered so far:
The notion of extendable, re-usable "contexts" is getting more and more appealing to me.
Fatal errors should(must) terminate (in) the supplied context rather than the host context.
A context contains enough (error) information to determine precisely what just happened, eg:
code = "a = 1\n"& "b = 2/0\n"& "c = 3" context = string_exec(context,code)
Somewhere in context (I am implying the above is not the first time it is used), there should now be one of:
- line 1, variable a has not been defined
- line 2, variable b has not been defined
- line 3, variable c has not been defined
- line 2, attempt to divide by zero
Obviously, if code contains no '\n' then everything gets reported on line 1, and it is left as a programming challenge to map whatever lines numbers you get back to whatever source code you threw at string_exec, but there will be reasonable clues, normally including a <=80-character code snippet.
Under no circumstances will c have been set to 3, and in that sense it is still a fatal error, but there is a separate outer context still running that can trap it.
Perhaps string_exec(or_all({NO_MSG,NO_ERRFILE}),code) or WITH_MSG,WITH_ERRFILE or even WITH_FATAL or something of that ilk could also be allowed, as an alternative to the initial string_exec(NULL,code). Obviously any such flags would be preserved within the returned context so they would not need to (and in fact could not) be re-specified on every call, and we can decide on the default settings so that in most cases no flags are required, later.
I am also thinking that should you pass a context with any non-zero error information back to string_exec then it should cause a fatal (host) error, because you clearly just completely ignored a problem. Or maybe once you get a fatal error, that context is forever non-reusable.
Migration
=======
Instead of (my initial ideas of) rebuilding a context from some previous (static) compilation, dynamic programming should be achieved by migrating more and more of the host app into the naturally dynamic playground of string_exec().
Instead of string_exec(contextX,...) being able to "magically" refer to data and routines declared in the hosting application, anything and everything needed should have previously been migrated into "contextX".
Scope issues vanish because what you are effectively doing is interperting some source, and later optionally extending that source code with some more code.
Multiple contexts can be declared as completely isolated from one another, or if you prefer you can run everything in a single context. You can save a "snapshot" or continuously extend a context as you please, eg:
context1 = string_exec(NULL,"integer a") context2 = string_exec(context1,"integer b") trash = string_exec(context1,"b = 3") -- error, b has not been defined trash = string_exec(context2,"integer b") -- error, b has already been defined trash = string_exec(context2,"b = 4") -- fine trash = string_exec(context1,"a = 5") -- fine trash = string_exec(context2,"b = a") -- sets b to 5
(I've used "trash" for clarity only, in general that sort of thing would be a terrible idea.)
The last line works because a context does not contain data but pointers to data blocks. Or we might want a WITH_CLONE flag to ensure that is not the case. It would make sense to automatically give contexts some (private) delete-routine-type and extra-reference-count-style handling. There would also no doubt be issues with shared or overlapping contexts being used by multiple threads, that are probably not very productive to get into just yet, given that we ain't got threads yet.
I am undecided whether lines 3 and 4 should scupper any attempts to carry on. At some level you must have a "clean compile", and it is not yet clear whether line 3 should scupper lines 4/5, because context2 is a derivative/extension of the now-invalidated context1 (and we did not specify WITH_CLONE anywhere).
Going somewhat beyond a 5-line example, a natural division might be to write the (largely static) gui logic as a host and the "business logic" in an entirely separate and dynamic string_exec context, obviously with an appropriate mechanism to retrieve items for display.
The one big obvious and compelling example that I somehow completely forgot about is a REPL.
Pete
38. Re: string_exec()
- Posted by katsmeow Mar 04, 2015
- 2250 views
Not that any of this is likely to happen any time soon, but what I've gathered so far:
The notion of extendable, re-usable "contexts" is getting more and more appealing to me.
Fatal errors should(must) terminate (in) the supplied context rather than the host context.
A context contains enough (error) information to determine precisely what just happened, eg:
code = "a = 1\n"& "b = 2/0\n"& "c = 3" context = string_exec(context,code)
Somewhere in context (I am implying the above is not the first time it is used), there should now be one of:
- line 1, variable a has not been defined
What if variable a was declared in the program already? And what if
atom b = 9 b += 1would b now be accessable to the program that called the string_exec()?
Kat
39. Re: string_exec()
- Posted by jimcbrown (admin) Mar 04, 2015
- 2246 views
A context contains enough (error) information to determine precisely what just happened, eg:
code = "a = 1\n"& "b = 2/0\n"& "c = 3" context = string_exec(context,code)
Somewhere in context (I am implying the above is not the first time it is used), there should now be one of:
- line 1, variable a has not been defined
What if variable a was declared in the program already?
In the caller of string_exec() you mean? In that case, IIUC petelomax correctly, it wouldn't make a difference in his implementation, since the context is still different - so you'd still get that error since variable a was declared in the wrong context.
And what if
atom b = 9 b += 1would b now be accessable to the program that called the string_exec()?
No, not in petelomax's implementation, because the context is different.
Edit: well, not directly of course. You probably would be able to do something like "set_string_exec_variable(context, "b", 10)" or "get_string_exec_variable(context, "b")" though.
40. Re: string_exec()
- Posted by petelomax Mar 04, 2015
- 2217 views
after
context = string_exec(context,"a = 1")
...error should be one of:
- line 1, variable a has not been defined
What if variable a was declared in the program already?
If you mean outside a string_exec, it would make no difference, whereas a previous
context = string_exec(context,"atom a")
would. If you think carefully about it, a scheme whereby string_exec()'d code has full access to the host environment is a support and security nightmare, and it makes a huge amount of sense to properly separate, safe-guard, and ring-fence things. Plus there are reasonable workarounds, see below. As I tried to explain under Migration the idea is to replace say
atom a context = string_exec(context,"a = 1")
with
context = string_exec(context,"atom a\n"& "a = 1")
Of course it may not just be a single variable, but a huge chunk of program logic that goes with. It may involve significant refactoring (possible workaround below) but at least there is nothing to stop you using include statements in the string_exec.
And what if
atom b = 9 b += 1would b now be accessable to the program that called the string_exec()?
Yes. After
atom c context = string_exec(context,"atom b = 9\n"& "b += 1") c = get_value(context,"b")
then c would be set to 10. Personally, I'm not convinced that the corresponding set_value has much merit, since string_exec(context,"a = value") would have the same effect, but I'm not staunchly opposed to it or anything. I suppose you could perhaps code something like:
atom a = 1 sequence context = string_exec(NULL,"atom a") set_value(context,"a",a) context = string_exec(context,"a += 1") a = get_value(context,"a")
which, horrible confusion aside, should leave a 2 in both the "inner" and "outer" a.
Pete
41. Re: string_exec()
- Posted by ryanj Mar 04, 2015
- 2211 views
I still do not understand why string_exec() would be useful. Would someone please show a code example that demonstrates what string_exec() can do that can't currently be done in Euphoria, or that would make something easier to do?
42. Re: string_exec()
- Posted by katsmeow Mar 04, 2015
- 2227 views
I still do not understand why string_exec() would be useful. Would someone please show a code example that demonstrates what string_exec() can do that can't currently be done in Euphoria, or that would make something easier to do?
I cannot show you that using a sandboxed string_exec().
However, using a eval() with access to the program environment,
Tiggr, what is 5 + 4? -- Tiggr munges to the question to discover the unknown you ask about -- Tiggr builds a variable sentence in various ways, -- including your nick (or a variation), -- various pleasantries (or not), various text colors, etcs, -- and appends the unknown, in this case : $calc(4+5) -- and then simply putting the variable name on a line in the code will execute itYou may think that's trivial to do, all you need do it write a case stack (or maybe a map) using all the numbers that will ever be asked, and all the variables that will ever be asked, and all the math/boolean operations on those numbers and vars that will ever be asked. And make sure those vars exist before you see if they have a value, or your app will crash.
You'd use this in your forthcoming spreadsheet, where a user entered a previously unknown formula into a cell, using the results of other formulas entered into other cells. Hopefully, you will lack the constraints of a sandbox on the eval(), in which case you'll filter the formulas against a list of illegal commands. You could also place all your eval calls in one include file, so the eval could be restricted to the scope of that include.
There's various procedures using string eval that are useful in some forms of Ai.
Kat
43. Re: string_exec()
- Posted by dcuny Mar 05, 2015
- 2177 views
Tiggr, what is 5 + 4?
It's pretty easy to write a string_exec() routine that does this:
string_exec("5+4")
Since Euphoria can see it's own variables and values (trace does this), it should be straight forward to do this as well:
string_exec("5+4-myVariable")
And since Euphoria can see user defined functions, it's not hard to extend string_exec to be able to this as well:
string_exec("5+4-myRoutine(myVariable)")
Is this what you're asking for? If so, it's a lot simpler than the solutions that have been proposed.
- David
44. Re: string_exec()
- Posted by DerekParnell (admin) Mar 05, 2015
- 2194 views
An old idea, but I thought it is about time I at least got a start on some design. ...
I've not had much need for this facility (that I can recall), so what would be its purpose? Can anyone give me some useful, concrete examples?
So far it seems like a method for an ad hoc user of "my app" to do some programming. I think I can see some merit in give such unknown code values from variables that "my app" controls, but I'm pretty sure I'd want some control over how such unknown code can update "my app"'s environment.
The closest I've gotten is that I've written some macro's that run in an app's space (eg. Excel, Word, Access, etc...) but they have pretty limited access to the underlying application's data space. Oh, and I wrote a game engine that allowed game designers to write "code" to control a game, but still they couldn't get access to the engine's data space.
I think the safest route to go would be to have string_exec() take two parameters - the code text, and a work-space environment that would contain values supplied by the calling app. string_exec() would return two things: A success/failure code and an updated work-space. The calling app could then decide what to do with the updated values.
45. Re: string_exec()
- Posted by katsmeow Mar 05, 2015
- 2224 views
Tiggr, what is 5 + 4?
It's pretty easy to write a string_exec() routine that does this:
string_exec("5+4")
Since Euphoria can see it's own variables and values (trace does this), it should be straight forward to do this as well:
string_exec("5+4-myVariable")
Well, except the programmer cannot see the var table. If you ask for the value of myVariable, it's more likely to crash because the var doesn't exist.
And since Euphoria can see user defined functions, it's not hard to extend string_exec to be able to this as well:
string_exec("5+4-myRoutine(myVariable)")
You can of course put anything in the string, that doesn't mean it gets converted to IL code and executes in the scope of where it was called.
Is this what you're asking for? If so, it's a lot simpler than the solutions that have been proposed.
- David
Show us. I'll pause taking Tiggr apart for a lil while, to see what you come up with.
Kat
46. Re: string_exec()
- Posted by katsmeow Mar 05, 2015
- 2207 views
An old idea, but I thought it is about time I at least got a start on some design. ...
I've not had much need for this facility (that I can recall), so what would be its purpose? Can anyone give me some useful, concrete examples?
I have used this in Tiggr for over 10 years, and if Ryan builds his spreadsheet app, he'll likely need it as well. I seem to recall a number of people were using includes late in an app's program, so that the app could write the include, and when RDS made that impossible, there was a fuss, so someones were doing a faux string_exec.
<snip>
I think the safest route to go would be to have string_exec() take two parameters - the code text, and a work-space environment that would contain values supplied by the calling app. string_exec() would return two things: A success/failure code and an updated work-space. The calling app could then decide what to do with the updated values.
Your suggestion that the programmer specifically state what vars be in the string_exec context means the programmer must know what vars the string will need, which is a difficult task if you don't know who will enter what string when. Same with stating a new set of includes for the new environment, how will you know what to include?
Which is why i suggested it be converted to IL code or compiled or whatever the wizardry du jour is, and the result inlined (or gosub'd) where it was called. That automatically restricts it to the same scope as the include file and the procedure containing it.
You'd want to filter out dangerous words and phrases like "system_exec(". Or maybe not, it depends on what you want to allow, what your app is to do.
Kat
Forked into: IL Engineering Library
47. Re: string_exec()
- Posted by petelomax Mar 05, 2015
- 2179 views
I still do not understand why string_exec() would be useful. Would someone please show a code example that demonstrates what string_exec() can do that can't currently be done in Euphoria, or that would make something easier to do?
An old idea, but I thought it is about time I at least got a start on some design. ...
I've not had much need for this facility (that I can recall), so what would be its purpose? Can anyone give me some useful, concrete examples?
The only half-decent concrete example I've got is a REPL.
To be honest, I was expecting more interest, and examples...
I'm pretty sure I'd want some control over how such unknown code can update "my app"'s environment.
My current thinking is that sting_exec can only see the things it has been explicitly told about. The reference-counted nature of the language should make it easy/fast to shunt even big data into and out of workspaces as needed.
One specific case I mused on the other day was that if it automatically granted full access to the underlying data, then string_exec("symtab=0") would guarantee instant death with no error, no warning, and no diagnostics. Since the presence of string_exec means a complete embedded copy of the compiler/interpreter, literally hundreds of other identifiers that could cause strange and highly intermittent bugs might also be exposed. While I probably/hopefully could avoid such issues, I took that level of potential devastation as fair warning.
The closest I've gotten is that I've written some macro's that run in an app's space (eg. Excel, Word, Access, etc...) but they have pretty limited access to the underlying application's data space.
Indeed, from a programming language perspective such things have zero access to the application's data space, except for some user-defined bits in a handful of specific tables, that it doesn't even know the names of.
I think the safest route to go would be to have string_exec() take two parameters - the code text, and a work-space environment that would contain values supplied by the calling app. string_exec() would return two things: A success/failure code and an updated work-space. The calling app could then decide what to do with the updated values.
I've gone with "context" as in context = string_exec(context,code). The (as yet loosely defined) context would contain extend-able symbol and lookup tables, code and data block(s), result codes, and anything else needed.
One other thing I'd like to mention is Matt's embed_routine idea. I've no real problem with that method of granting selected access to the underlying application's data, except for potential confusion if it plays with a different "mytable" to the one in the string_exec workspace. Hence I'm somewhat hesitant on that.
Pete
48. Re: string_exec()
- Posted by dcuny Mar 05, 2015
- 2146 views
Well, except the programmer cannot see the var table. If you ask for the value of myVariable, it's more likely to crash because the var doesn't exist.
In trace, you can ask for the value of a variable and it'll be displayed. If you ask for a variable that doesn't exist in the current scope, it doesn't crash.
I seem to recall someone asking for this mechanism recently, and reading that Euphoria provided it. So I'll have to look to see how it's implemented.
You can of course put anything in the string, that doesn't mean it gets converted to IL code and executes in the scope of where it was called.
I'm not talking about executing as IL. You can write a simple expression parser. If an identifier is followed by an open parenthesis (, the parser knows it's a function, and uses routine_id to find the routine name. If not, it's a variable and it looks in the variable table, using a mechanism similar to trace.
Show us. I'll pause taking Tiggr apart for a lil while, to see what you come up with.
Is an expression evaluator what you're after? Note that you can inquire about variables, and call functions, but you can't actually extend the current application by adding new variables and functions.
This solution would work if you're only interested in executing the right hand side of an expression.
- David
49. Re: string_exec()
- Posted by dcuny Mar 05, 2015
- 2180 views
As a follow up, I'm not sure I understand why you need to see Euphoria's variables. For example:
a = string_exec("a+b")
The code that builds the string "a+b" obviously thinks there are variables called a and b, because it built a string expression containing those variables. But how does the code know about them?
Put a different way, if string_exec() were able to used named variables, would that be enough? For example:
sequence context = {{"a", 100}, {"b", 200}} string_exec(context, "myRoutine(a+b)+sin(a)")
Would something like that satisfy your needs? That would be even simpler to write, and it could be written in pure Euphoria. I could cannibalize the parser I've written for Py to do this.
The question then becomes what's the best way to pass values of variables to string_exec. For something like a spreadsheet, the context would be pretty clunky. Passing the routine_id of a user-defined function that would resolve variables would be a more flexible approach, along the lines of:
sequence context = {{"a", 100}, {"b", 200}} -- returns {value, success_flag} function lookup_variable(sequence varName) for i = 1 to length(context) if equal(context[i][1], varName) then -- return value and success flag return {context[i][2], 1} end if end for -- return bogus value and failure flag return {0, 0} end function string_exec(routine_id("lookup_variable"), "myRoutine(a+b)+sin(a)")
Or the function lookup_variable() could be supplied, and you'd just recode it to your needs.
Would that work?
- David
50. Re: string_exec()
- Posted by mattlewis (admin) Mar 05, 2015
- 2164 views
Since Euphoria can see it's own variables and values (trace does this), it should be straight forward to do this as well:
string_exec("5+4-myVariable")
That's true, but trace has pointers to the internal structure to work with. It's not that the symbol table isn't available, but scopes make this somewhat complicated.
Matt
51. Re: string_exec()
- Posted by dcuny Mar 05, 2015
- 2115 views
That's true, but trace has pointers to the internal structure to work with. It's not that the symbol table isn't available, but scopes make this somewhat complicated.
I suspect that for Kat's use case, it's probably not necessary anyway (see prior post).
- David
52. Re: string_exec()
- Posted by katsmeow Mar 05, 2015
- 2095 views
The only half-decent concrete example I've got is a REPL. To be honest, I was expecting more interest, and examples...
I expect those who were doing dynamic includes as a means to string execution in RDS v2.5 have long gone elsewhere. There's a few examples in the archives.
<warnings of doom>
<warnings of doom and gloom>
And the two of you see no way to sanitize bad words or procedures?
Kat
53. Re: string_exec()
- Posted by katsmeow Mar 05, 2015
- 2115 views
As a follow up, I'm not sure I understand why you need to see Euphoria's variables. For example:
a = string_exec("a+b")
The code that builds the string "a+b" obviously thinks there are variables called a and b, because it built a string expression containing those variables. But how does the code know about them?
It might not know about them. Do you have the variable TheSecondCowThatJoeBought?
Is an expression evaluator what you're after? Note that you can inquire about variables, and call functions, but you can't actually extend the current application by adding new variables and functions.
What if you were in the 3rd grade, and i offered to teach you a new trick: how to calculate the average age of Joe's cow herd, but you are unable to learn new procedures or functions or variable names?
Kat
54. Re: string_exec()
- Posted by katsmeow Mar 05, 2015
- 2147 views
Since Euphoria can see it's own variables and values (trace does this), it should be straight forward to do this as well:
string_exec("5+4-myVariable")
That's true, but trace has pointers to the internal structure to work with. It's not that the symbol table isn't available, but scopes make this somewhat complicated.
Matt
Unless you drop the new IL into a situation where the scope is already defined and limited, see http://openeuphoria.org/forum/m/127270.wc
Kat
55. Re: string_exec()
- Posted by mattlewis (admin) Mar 05, 2015
- 2111 views
Since Euphoria can see it's own variables and values (trace does this), it should be straight forward to do this as well:
string_exec("5+4-myVariable")
That's true, but trace has pointers to the internal structure to work with. It's not that the symbol table isn't available, but scopes make this somewhat complicated.
Unless you drop the new IL into a situation where the scope is already defined and limited, see http://openeuphoria.org/forum/m/127270.wc
Right, but that's using all new stuff, not trying to look stuff up. As I said, my preference is to keep that stuff sandboxed but allow integration points to be defined.
Matt
56. Re: string_exec()
- Posted by katsmeow Mar 05, 2015
- 2118 views
Since Euphoria can see it's own variables and values (trace does this), it should be straight forward to do this as well:
string_exec("5+4-myVariable")
That's true, but trace has pointers to the internal structure to work with. It's not that the symbol table isn't available, but scopes make this somewhat complicated.
Unless you drop the new IL into a situation where the scope is already defined and limited, see http://openeuphoria.org/forum/m/127270.wc
Right, but that's using all new stuff, not trying to look stuff up. As I said, my preference is to keep that stuff sandboxed but allow integration points to be defined.
Matt
Would it be easy to call the existing routines, slightly modified, as jimcbrown mentions, and use them in an include with the new top-level option of "include nothing" or "inherit nothing" or something more (mild and controllable), to make the desired clean sterile environment?
Kat
57. Re: string_exec()
- Posted by mattlewis (admin) Mar 05, 2015
- 2101 views
Would it be easy to call the existing routines, slightly modified, as jimcbrown mentions, and use them in an include with the new top-level option of "include nothing" or "inherit nothing" or something more (mild and controllable), to make the desired clean sterile environment?
I guess it depends on what you're doing. I assume, however, that the point of running some dynamic code is to have it affect your program. If it's completely isolated from your program it doesn't seem very useful.
Matt
58. Re: string_exec()
- Posted by katsmeow Mar 05, 2015
- 2110 views
Would it be easy to call the existing routines, slightly modified, as jimcbrown mentions, and use them in an include with the new top-level option of "include nothing" or "inherit nothing" or something more (mild and controllable), to make the desired clean sterile environment?
I guess it depends on what you're doing. I assume, however, that the point of running some dynamic code is to have it affect your program. If it's completely isolated from your program it doesn't seem very useful.
Matt
Is there no grey area? I was trying to find a way to allow your sandbox, and allow me to include the whole world, which is why i said "or something more (mild and controllable), to make the desired clean sterile environment". Your desired environment is "blistering white", mine is "don't reformat the harddrives". You would not allow anything useful to happen, i would just trounce the system calls.
Kat
59. Re: string_exec()
- Posted by jimcbrown (admin) Mar 05, 2015
- 2116 views
Is there no grey area? I was trying to find a way to allow your sandbox, and allow me to include the whole world
I don't think so. Doing what you want via string_exec() would almost seem to require that string_exec() perform dynamic runtime self-modifying bytecode manipulation anyways. Which certainly isn't out of the question - but if you're going to be manipulating bytecode anyways, why not just go straight to the source instead of dealing with an extra layer?
Another point is that implementing a sandbox'd string_exec() or eueval4.1 is a lot easier than implementing an unsandbox'd string execution mechanism via self-modifying bytecode.
Edit: Although there may be no grey area, I think that there is room for both concepts to co-exist side-by-side (at least IMVHO).
60. Re: string_exec()
- Posted by dcuny Mar 05, 2015
- 2078 views
What if you were in the 3rd grade, and i offered to teach you a new trick: how to calculate the average age of Joe's cow herd, but you are unable to learn new procedures or functions or variable names?
I'm trying to what sort of solution would fit your needs. There are many possible solutions that could be proposed, but if they don't fit your needs, what good does that do?
You gave two examples, neither of which needed to create functions on the fly.
So I'm trying to clarify what you're doing, to ensure I understand the problem we're trying to solve.
At this point, I don't think I've got a good understanding of your needs.
For example, I don't understand why a sandbox won't work. Why does your solution need to know about it's internal variables?
- David
61. Re: string_exec()
- Posted by katsmeow Mar 05, 2015
- 2073 views
What if you were in the 3rd grade, and i offered to teach you a new trick: how to calculate the average age of Joe's cow herd, but you are unable to learn new procedures or functions or variable names?
I'm trying to what sort of solution would fit your needs. There are many possible solutions that could be proposed, but if they don't fit your needs, what good does that do?
You gave two examples, neither of which needed to create functions on the fly.
I said "string execution". On an OE forum, i expected everyone would understand the string was of OE code. Any OE code. I intentially did not restrict the code to certain subsets of OE.
So I'm trying to clarify what you're doing, to ensure I understand the problem we're trying to solve.
At this point, I don't think I've got a good understanding of your needs.
For example, I don't understand why a sandbox won't work. Why does your solution need to know about it's internal variables?
My solution doesn't need to know about it's variables any more than the encompassing program does (which is a different thread, and i think it does). The way Matt, Derek, petelomax, etc see sandboxed string execution, one must know a var exists and be able to copy it into the sandbox, and then know it exists inside the sandbox and have code already written to copy it out of the sandbox.
Way back when, the solution was dynamic includes, but of course they couldn't be un-included, and there were other issues (like eventually you simply ran out of the encompassing program). Dynamic includes were normal includes, but sprinkled thruout the program, possibly written by the program itself before they were included. I don't recall anyone suggesting those be sandboxed.
Pleading for string execution is like beating a dead horse, and i am weary and feel hopeless. It's not going to happen. I made a mistake sticking with wintel, trying nix, and waiting on Euphoria to grow up. It's that simple.
Kat
62. Re: string_exec()
- Posted by mattlewis (admin) Mar 05, 2015
- 2080 views
Is there no grey area? I was trying to find a way to allow your sandbox, and allow me to include the whole world, which is why i said "or something more (mild and controllable), to make the desired clean sterile environment". Your desired environment is "blistering white", mine is "don't reformat the harddrives". You would not allow anything useful to happen, i would just trounce the system calls.
I would have described my idea as the middle way here. You could control how much the dynamic code could do. In your case, if you're really controlling it all, you might very well go whole hog. If you're running user input or something, you'd probably be more locked down.
I don't see how you get "blistering white" from that.
Matt
63. Re: string_exec()
- Posted by mattlewis (admin) Mar 05, 2015
- 2074 views
Way back when, the solution was dynamic includes, but of course they couldn't be un-included, and there were other issues (like eventually you simply ran out of the encompassing program). Dynamic includes were normal includes, but sprinkled thruout the program, possibly written by the program itself before they were included. I don't recall anyone suggesting those be sandboxed.
Pleading for string execution is like beating a dead horse, and i am weary and feel hopeless. It's not going to happen. I made a mistake sticking with wintel, trying nix, and waiting on Euphoria to grow up. It's that simple.
You talk like this is an easy thing to make happen and we're not doing it to spite you. There are legitimate concerns that a lot of people have in addition to the implementation challenges, which are significant. I'm not your slave and trying to guilt trip me into something like this isn't going to work, either.
Matt
64. Re: string_exec()
- Posted by katsmeow Mar 05, 2015
- 2078 views
Way back when, the solution was dynamic includes, but of course they couldn't be un-included, and there were other issues (like eventually you simply ran out of the encompassing program). Dynamic includes were normal includes, but sprinkled thruout the program, possibly written by the program itself before they were included. I don't recall anyone suggesting those be sandboxed.
Pleading for string execution is like beating a dead horse, and i am weary and feel hopeless. It's not going to happen. I made a mistake sticking with wintel, trying nix, and waiting on Euphoria to grow up. It's that simple.
You talk like this is an easy thing to make happen and we're not doing it to spite you. There are legitimate concerns that a lot of people have in addition to the implementation challenges, which are significant. I'm not your slave and trying to guilt trip me into something like this isn't going to work, either.
Matt
I didn't invalidate other's concerns, except they don't allow what i wish to do, they don't apply in my case. I understand it's significant work, altho i don't understand why, because i did it 30 years ago, and i have asked for it in Euphoria for over a decade. And there's not any guilt trip, you all decided not to do string execution back when i was still using win95, and i hung around till now because Euphoria is otherwise a very capable language, hoping you'd change your minds. I was stupid to not just move on. I am just too tired of repeating myself to go on discussing this as if it will eventually happen.
Kat
65. Re: string_exec()
- Posted by jimcbrown (admin) Mar 06, 2015
- 2079 views
Way back when, the solution was dynamic includes, but of course they couldn't be un-included, and there were other issues (like eventually you simply ran out of the encompassing program). Dynamic includes were normal includes, but sprinkled thruout the program, possibly written by the program itself before they were included. I don't recall anyone suggesting those be sandboxed.
Pleading for string execution is like beating a dead horse, and i am weary and feel hopeless. It's not going to happen. I made a mistake sticking with wintel, trying nix, and waiting on Euphoria to grow up. It's that simple.
You talk like this is an easy thing to make happen and we're not doing it to spite you. There are legitimate concerns that a lot of people have in addition to the implementation challenges, which are significant. I'm not your slave and trying to guilt trip me into something like this isn't going to work, either.
Matt
I didn't invalidate other's concerns, except they don't allow what i wish to do, they don't apply in my case. I understand it's significant work, altho i don't understand why, because i did it 30 years ago, and i have asked for it in Euphoria for over a decade. And there's not any guilt trip, you all decided not to do string execution back when i was still using win95, and i hung around till now because Euphoria is otherwise a very capable language, hoping you'd change your minds. I was stupid to not just move on. I am just too tired of repeating myself to go on discussing this as if it will eventually happen.
Kat
Please take any venting off the forum.
66. Re: string_exec()
- Posted by dcuny Mar 06, 2015
- 2255 views
I'm not your slave and trying to guilt trip me into something like this isn't going to work, either.
Kat's comments don't seem to be targeted to anyone specifically.
Matthew's are, and seem to me as both "putting down" and "showing disrespect".
So why is Kat the one being slapped with a Code of Conduct violation here?
In any event, I'm still unclear on the details of the request. So I'm going to try to get clarification again.
Code that is outside of routine definitions effectively goes into a no-name top-level routine. You can think of:
integer a function foo() return 100 end function a=foo()
As effectively:
function foo() return 100 end function procedure __top_level__() integer a a = foo end procedure
If you're asking for something like this:
string_exec("integer a\nfunction foo()\n return 100\n end function\n a=foo()")
it raises the question of where the "run me now" code goes. It can't be inserted into existing top level code, since that's a currently executing scope.
Since you're doing two different things - compiling and executing - you need two different steps. For example:
-- compile foo(), and return the routine_id integer foo_id = compile_func("foo", {}, {"return 100"}) -- compile code to run foo() and set a global variable, then execute it call_proc(compile_proc("run_foo", {"a=foo()"}))
Does this approach make sense?
- David
67. Re: string_exec()
- Posted by jimcbrown (admin) Mar 06, 2015
- 2050 views
I'm not your slave and trying to guilt trip me into something like this isn't going to work, either.
Kat's comments don't seem to be targeted to anyone specifically.
Agreed.
Matthew's are, and seem to me as both "putting down" and "showing disrespect".
Hmm. Not sure I'd agree with you here. If those statements were directed at me, and Mmattlewis was trying to protest (to me) against being forced into working on coding something that he really didn't want to do, I wouldn't feel that way.
So why is Kat the one being slapped with a Code of Conduct violation here?
katsmeow has not been slapped with a CodeOfConduct violation.
In any event, I'm still unclear on the details of the request.
I was under the impression that what katsmeow really wanted was dynamic/self-modifying code + REPL. OTOH, the entire kitchen sink. Which is something I support.
So I'm going to try to get clarification again.
I think that maybe it is time to get something a whole lot more formal. Perhaps something along the lines of what's described here: http://www.bradapp.com/docs/sdd.html
68. Re: string_exec()
- Posted by mattlewis (admin) Mar 06, 2015
- 2045 views
I'm not your slave and trying to guilt trip me into something like this isn't going to work, either.
Kat's comments don't seem to be targeted to anyone specifically.
Matthew's are, and seem to me as both "putting down" and "showing disrespect".
So why is Kat the one being slapped with a Code of Conduct violation here?
Sorry, my post was partly responding to posts and emails over the years. I actually toned it down quite a bit from what I wanted to say. She has frequently leveled complaints at the developers (of which I am one) about ignoring her or leading her along and just to trash her or her code. Whatever she was trying to do, that was how it read to me, and I responded.
Personally, I think we've gone a bit overboard with the code of conduct thing.
Matt
69. Re: string_exec()
- Posted by dcuny Mar 06, 2015
- 2022 views
I was under the impression that what katsmeow really wanted was dynamic/self-modifying code + REPL. OTOH, the entire kitchen sink. Which is something I support.
I got that impression as well. I'm just trying to figure out how the mechanics of that would work.
There are a couple bits that aren't clear to me. The where does "top level" code is one of them.
The other is how existing code gets to know about the new routines, since none of it was visible at the time of compilation.
If the request is allowing self-modifying code, what exactly does that mean? Can a routine be redefined?
I think what I've suggested answers both of those questions, but doesn't answer the question of how to add top-level variables.
I'm just trying to understand what's being ask for.
- David
70. Re: string_exec()
- Posted by jimcbrown (admin) Mar 06, 2015
- 2012 views
I'm not your slave and trying to guilt trip me into something like this isn't going to work, either.
Kat's comments don't seem to be targeted to anyone specifically.
Matthew's are, and seem to me as both "putting down" and "showing disrespect".
So why is Kat the one being slapped with a Code of Conduct violation here?
Sorry, my post was partly responding to posts and emails over the years. I actually toned it down quite a bit from what I wanted to say. She has frequently leveled complaints at the developers (of which I am one) about ignoring her or leading her along and just to trash her or her code. Whatever she was trying to do, that was how it read to me, and I responded.
I think that if any interested parties want to discuss this topic further, it should be taken off the forum. Please use openeuphoria@gmail.com for further discussion, if that is desired.
Personally, I think we've gone a bit overboard with the code of conduct thing.
Matt
Perhaps. I'd rather err on the side of being too polite rather than the opposite, myself.
71. Re: string_exec()
- Posted by jimcbrown (admin) Mar 06, 2015
- 2048 views
I was under the impression that what katsmeow really wanted was dynamic/self-modifying code + REPL. OTOH, the entire kitchen sink. Which is something I support.
I got that impression as well. I'm just trying to figure out how the mechanics of that would work.
I'm just trying to understand what's being ask for.
Logical.
There are a couple bits that aren't clear to me. The where does "top level" code is one of them.
What I had envisioned includes the ability to modify true top-level code and then re-execute it.
So top level code would continue to live in the top level, with no special scoping or protection from pre-existing top level code.
Ironically, top level code is the easiest to deal with in a BCEL library since there's fewer internal data structures that need to be rearranged to deal with it (as compared to creating a whole new routine, for example).
The other is how existing code gets to know about the new routines, since none of it was visible at the time of compilation.
This is hard to do since in addition to IL itself we'd have to update the internal data structures that manage routines. But the goal is to allow one to write whole new routines on the fly and call them.
If the request is allowing self-modifying code, what exactly does that mean? Can a routine be redefined?
Yes.
I think what I've suggested answers both of those questions, but doesn't answer the question of how to add top-level variables.
This also requires modifying internal data structures, but it's probably a little easier than deal with routines.
72. Re: string_exec()
- Posted by dcuny Mar 06, 2015
- 2020 views
What I had envisioned includes the ability to modify true top-level code and then re-execute it.
My concern is that you'll effectively be in the middle of the top-level code:
-- this string_exec(...) -- that
If I understand your scheme, it would effectively become:
-- this string_exec(...) -- code generated by string_exec() <-- code begins re-executing here -- that
Is that correct?
The other is how existing code gets to know about the new routines, since none of it was visible at the time of compilation.
This is hard to do since in addition to IL itself we'd have to update the internal data structures that manage routines. But the goal is to allow one to write whole new routines on the fly and call them.
I believe using the routine_id mechanism solves this issue a bit more cleanly... although I'll confess having a love/hate relationship with routine_id.
If the request is allowing self-modifying code, what exactly does that mean? Can a routine be redefined?
Yes.
It seems you can get the same result via routine_id, at a much cheaper cost.
- David
73. Re: string_exec()
- Posted by katsmeow Mar 06, 2015
- 1991 views
Can a routine be redefined?
Yes.
It seems you can get the same result via routine_id, at a much cheaper cost.
- David
Which is why i suggested it be converted to IL code or compiled or whatever the wizardry du jour is, and the result inlined (or gosub'd) where it was called.
Pronounce "gosub" like "routine_id". Gosub/routine_id means the block of memory where the eval/exec takes place does not need to displace existing code (see http://openeuphoria.org/forum/m/127273.wc), or be rearranged if a max size is allocated (altho it could be alloc'd and free'd, perhaps), and (almost) any header info (such as parameters restricting scope to the location it gosub'd from) can be static at compile time. This is exactly how i did it (except i emulated the gosub with pointer swaps instead of using the stack, for speed reasons).
Kat