OpenEuphoria

1. string_exec()

• Posted by petelomax Feb 28, 2015
• 2530 views
• Last edited Mar 01, 2015

An old idea, but I thought it is about time I at least got a start on some design.

Implementation may be some time off, this is all just some preliminary musings, and not particularly well thought through at that, but as I'm currently doing some design work on the optable (not that you need to know or care what that is), now seems as good a time as any to consider this. I haven't seen any examples that make me think "you know what, that's a really good idea", so now is your chance.

Obviously this has something of a Phix bias that may need ironing out. It must (somehow) include a complete copy of the compiler/interpreter, or re-use the one already running. I imagine OpenEuphoria might use eu.ex, the hll interpreter.

Firstly, there is no security model whatsoever. If you put this on a web server and let visitors type in and execute code, you may as well just give them the root password. However, if this is an app they have downloaded and installed locally, it is, literally, the same deal as installing a compiler. All I am really saying is that I have no plans to go down the Java route of 12 security patches a day from now until the end of time; if someone else wants that then they can jolly well do it themselves! That said, I would support an optional "draconian safe mode" whereby such things as file i/o, system_exec, inline assembly, etc are all completely prohibited inside string_exec() - if someone else takes the time to carefully sift through the manual and compose a comprehensive list of all potential dangers.

Let's begin simple:

integer i = 5 
    ctrl = string_exec(ctrl,"i+=1") 

While (on Phix) when interpreted there might be a few salvage-able bits of lookup and the possibility of extending the symbol table, when compiled we have:

• original symbol table (damaged, fixed size, non-extendable, no lookup)
• static data block (fixed size, non-extendable)
• executable code (fixed size, non-extendable)

So during string_exec(), we will need:

• a completely rebuilt symbol table, extendable, with rebuilt lookup
• original and new static data blocks, with index ranges (lists of)
• original and new executable code blocks (lists of)

Such things could be preserved in ctrl¹ so that subsequent invocations are not quite so expensive, even if that might only be half a second per call to string_exec, and for reasons that should become clear soon.

Question: Are we hoping for something like this?

integer i = 5 
    ctrl = string_exec(ctrl,"integer j\n"& 
                            "function add1(integer i)"& 
                            "   return i+1"& 
                            "end function") 
    ctrl = string_exec(ctrl,"j = add1(i)") 

In other words, are the rebuilt and extended symbol table, new data, and new code all available for use in subsequent invocations of string_exec()? Or should the symbol table be reset for each invocation and any temporary code/data discarded? Or do we need to cater for both? [Examples to backup any claims, please.]

Next question: What sort of context are we looking for?

procedure p() 
integer i=5 
    ctrl = string_exec(ctrl,"i+=1") 
end procedure 

Should scope be as per EOF of main file or at the point of invoking string_exec? The latter is much harder - and if there are multiple string_exec calls, we might need different scopes for each. I would need some blinding examples to convince me that is worthwhile.

What error messages (compile-time and run-time) if any do we really need?

What sort of clean-up (deallocating temps, freeing allocated memory, closing files, etc) do we really need? As per the add1 example above, we could go too far and trash something we might want later.

Is there any need for expression results?

integer i = 5, j 
    j = string_expr(ctrl,"i") 
-- (or maybe 
--  ctrl = string_expr(ctrl,"i") 
--  j = ctrl[RESULT]) 

Note that you couldn't define any vars/funcs, unless you made "return <expr>" at top level legal in string expressions. Again, I'd need convincing examples.

We probably ought to be able to "nest" string evaluation, though I hope someone can provide a much better example than this:

    j = string_expr(ctrl,"string_expr(ctrl,\"i\")") 

I can see a use for "standalone" string evaluation, that starts with the basic "empty" symbol table², such as:

    a = string_expr(NULL,"5+4/3") 

which might be quite useful in say a calculator, anything else? (Obviously that assumes support for expression results, but we could achieve equivalent functionality via something like "res=5+4/3")

Also, it is not particularly clear what command_line() should return when it is invoked from inside a string_exec().

This is far from over, but I think that's probably enough for now, plus I just spotted a major flaw in nested opInterp/thread safety that I must go and fix asap.

Please try to provide examples that I have a vague chance of understanding

Pete

¹ the content and structure of ctrl is deliberately left vague and undefined.
² maybe: string_expr(NULL,...)=>empty; string_expr({},...)=>rebuild, or vice versa.

3. Re: string_exec()

• Posted by Ekhnat0n Feb 28, 2015
• 2504 views

Great idea, but dangerous if not really well safe-guarded against any hacking possibilities.
Would be quite hard to write that all in assembler though
as you know I am currently working on a basic set
of commands to do general things like
Do......while
Loop....Until
If....Then===== Elsif======(etc) Endif
Case x .... y..... etc Endcase
Put {screen/file/mem} & Get {ditto}
all in native ASM for 68K, 8086,8086-64 and ARM

5. Re: string_exec()

• Posted by jimcbrown (admin) Feb 28, 2015
• 2505 views

No problem. It should suffice to talk about the structure of eu.ex and execute.e

So basically, if a NFI occurs, but it happens to a statement that's usable inside a string_eval() call, then replace the line with a string_eval() call to the original statement.

So, assuming that division by zero is an NFI, when we execute

? 1/0 

and hit the NFI, we rollback and replace it with

string_eval("? 1/0") 

Presumably, calls to string_eval() that throw an error will not crash the main program, but only stop execution of the eval'd code and set some kind of error flag that can be checked later.

In short, you are suggesting that we use string_eval() to implement "ON ERROR GOTO NEXT" error handling behavior, with perhaps some minor differences.

The basic concept is sound.

     .timer 1 %defflooddelay %temptext 

Ok, so if code inside of a string_eval() string raises an error, we treat it exactly the same as if it had happened in the top-level scope. E.g. If we have try-catches, then we'd throw the exception from string_eval() up to the caller, and so on.

If we have try-catches or some other sane error handling method, this works. Right now, due to the issues dcuny raised previously in his thread about try/catch, I believe that string_eval() should not do this. I do not think that an error in string_eval() should shut down the entire program and generate an ex.err file. But once we implement better error handling, then this can be changed too.

7. Re: string_exec()

• Posted by petelomax Mar 01, 2015
• 2440 views

9. Re: string_exec()

• Posted by petelomax Mar 01, 2015
• 2477 views

 
So, assuming that division by zero is an NFI, when we execute 
 
? 1/0 
 
and hit the NFI, we rollback and replace it with 
 
string_eval("? 1/0") 
 

Well, presumably it is an NFI because it is inside a string_eval, so if we rollback (whatever that is) and replace it with a string_eval, it will cause an NFI which presumably we will rollback (whatever that is) and replace with a string_eval, which will cause an NFI which we can replace with a string_eval that causes an NFI that we replace with a string evalthatcausesanNFIthatwereplacewithastringevalthat...

What is the point?

Pete

11. Re: string_exec()

• Posted by dcuny Mar 01, 2015
• 2415 views

My assumption was that the string execution would be implemented via the current Euphoria interpreter. The context would be all the information the Euphoria-to-opcode program generates.

If the context is re-entrant and the data structure can be appended to, that would be awesome.

But I really don't know enough about the compiler to offer any suggestions here.

I've been kicking around the idea of eventually writing my own "assembler" for Euphoria, which would look like:

Euphoria source -> assembly -> context (opcodes + dictionary) 

... but like everyone else, I haven't gotten around to it yet.

- David

13. Re: string_exec()

• Posted by jimcbrown (admin) Mar 01, 2015
• 2397 views

Huh? Which one should it be?

15. Re: string_exec()

• Posted by jimcbrown (admin) Mar 01, 2015
• 2382 views

Yes.

No.

Only if it is misread/misheard. But unfortunately that type of situation is quite common.

17. Re: string_exec()

• Posted by jimcbrown (admin) Mar 01, 2015
• 2376 views

No.

19. Re: string_exec()

• Posted by jimcbrown (admin) Mar 01, 2015
• 2366 views

False premise. By itself, that phrase causes a fatal error.

<0132>:: Syntax error - expected to see possibly '(', not the end of file 
<end-of-file> 
^ 

Anyways, statements are either fatal or not. But there's no way to flag one kind into the other.

21. Re: string_exec()

• Posted by jimcbrown (admin) Mar 01, 2015
• 2373 views

I respect your right and choice to withdraw from this conversation. In general though, I suggest that - unless someone is requesting that one posts a response - the best way to withdraw is simply not to post a reply at all.

A friendly reminder: Putting words in my mouth is a CodeOfConduct violation...

It's definitely possible, since other languages exist that do it.

23. Re: string_exec()

• Posted by dcuny Mar 01, 2015
• 2412 views

I don't see why you'd want string_eval to throw an exception unless there was a mechanism to catch it with.

Even then, the exception would have happened within the context, not the caller.

The point of returning a result code is to interrogate the result of string_eval.

Why would you want string_eval throw a fatal exception in the calling routine? What's the benefit?

- David

25. Re: string_exec()

• Posted by mattlewis (admin) Mar 01, 2015
• 2367 views

As I look at it, we're talking about calling some code. The only difference here is that the code wasn't parsed, etc, at the beginning. So the string_eval() call is just another part of the call stack, so (assuming exceptions have been implemented) an exception should bubble up to the caller like any other function call.

Unless we're talking about this as something sandboxed away from the current program. I don't think that's what Kat wants, since I believe she wants to interact with the data and code of the program that's executing the dynamic code.

Matt

27. Re: string_exec()

• Posted by andi49 Mar 01, 2015
• 2387 views

From my point of view this is the only way to implement!
Eval in the current program is something i do not want to even think about.
(eval needs to live in a Sandbox and nowhere else!)

Thank you

Andreas

Forked into: Multiple interpreter instances

29. Re: string_exec()

• Posted by mattlewis (admin) Mar 01, 2015
• 2404 views

Absolutely. You can use ooeu that way. I used that for wxedb. But that's not what Kat's talking about when she asks for an eval function.

Matt

31. Re: string_exec()

• Posted by mattlewis (admin) Mar 01, 2015
• 2387 views

There absolutely is a point. You can provide integration points where the internal code can interface with the main program. Think of a plugin, or user scripts in a game. There are many reasons why security could still be an issue there. You might also want to have multiple contexts that do not interfere with each other.

Here are the oeu docs on embedding an interpreter inside your program:

http://ooeu.sourceforge.net/docs/EMBEDDING.htm

Matt

33. Re: string_exec()

• Posted by jimcbrown (admin) Mar 02, 2015
• 2420 views

Understood, but I agree with dcuny here:

35. Re: string_exec()

• Posted by jimcbrown (admin) Mar 02, 2015
• 2388 views

Which is why I haven't advocated OOEU as the official new eueval library.

No matter which way you look at it, a new implementation is going to be necessary.

37. Re: string_exec()

• Posted by petelomax Mar 04, 2015
• 2244 views

Not that any of this is likely to happen any time soon, but what I've gathered so far:

The notion of extendable, re-usable "contexts" is getting more and more appealing to me.

Fatal errors should(must) terminate (in) the supplied context rather than the host context.
A context contains enough (error) information to determine precisely what just happened, eg:

code = "a = 1\n"& 
       "b = 2/0\n"& 
       "c = 3" 
context = string_exec(context,code) 

Somewhere in context (I am implying the above is not the first time it is used), there should now be one of:

• line 1, variable a has not been defined
• line 2, variable b has not been defined
• line 3, variable c has not been defined
• line 2, attempt to divide by zero

Obviously, if code contains no '\n' then everything gets reported on line 1, and it is left as a programming challenge to map whatever lines numbers you get back to whatever source code you threw at string_exec, but there will be reasonable clues, normally including a <=80-character code snippet.

Under no circumstances will c have been set to 3, and in that sense it is still a fatal error, but there is a separate outer context still running that can trap it.

Perhaps string_exec(or_all({NO_MSG,NO_ERRFILE}),code) or WITH_MSG,WITH_ERRFILE or even WITH_FATAL or something of that ilk could also be allowed, as an alternative to the initial string_exec(NULL,code). Obviously any such flags would be preserved within the returned context so they would not need to (and in fact could not) be re-specified on every call, and we can decide on the default settings so that in most cases no flags are required, later.

I am also thinking that should you pass a context with any non-zero error information back to string_exec then it should cause a fatal (host) error, because you clearly just completely ignored a problem. Or maybe once you get a fatal error, that context is forever non-reusable.

Migration
=======

Instead of (my initial ideas of) rebuilding a context from some previous (static) compilation, dynamic programming should be achieved by migrating more and more of the host app into the naturally dynamic playground of string_exec().

Instead of string_exec(contextX,...) being able to "magically" refer to data and routines declared in the hosting application, anything and everything needed should have previously been migrated into "contextX".

Scope issues vanish because what you are effectively doing is interperting some source, and later optionally extending that source code with some more code.

Multiple contexts can be declared as completely isolated from one another, or if you prefer you can run everything in a single context. You can save a "snapshot" or continuously extend a context as you please, eg:

context1 = string_exec(NULL,"integer a") 
context2 = string_exec(context1,"integer b") 
trash = string_exec(context1,"b = 3")       -- error, b has not been defined 
trash = string_exec(context2,"integer b")   -- error, b has already been defined 
trash = string_exec(context2,"b = 4")       -- fine 
trash = string_exec(context1,"a = 5")       -- fine 
trash = string_exec(context2,"b = a")       -- sets b to 5 

(I've used "trash" for clarity only, in general that sort of thing would be a terrible idea.)
The last line works because a context does not contain data but pointers to data blocks. Or we might want a WITH_CLONE flag to ensure that is not the case. It would make sense to automatically give contexts some (private) delete-routine-type and extra-reference-count-style handling. There would also no doubt be issues with shared or overlapping contexts being used by multiple threads, that are probably not very productive to get into just yet, given that we ain't got threads yet.

I am undecided whether lines 3 and 4 should scupper any attempts to carry on. At some level you must have a "clean compile", and it is not yet clear whether line 3 should scupper lines 4/5, because context2 is a derivative/extension of the now-invalidated context1 (and we did not specify WITH_CLONE anywhere).

Going somewhat beyond a 5-line example, a natural division might be to write the (largely static) gui logic as a host and the "business logic" in an entirely separate and dynamic string_exec context, obviously with an appropriate mechanism to retrieve items for display.

The one big obvious and compelling example that I somehow completely forgot about is a REPL.

Pete

39. Re: string_exec()

• Posted by jimcbrown (admin) Mar 04, 2015
• 2233 views

code = "a = 1\n"& 
       "b = 2/0\n"& 
       "c = 3" 
context = string_exec(context,code) 

In the caller of string_exec() you mean? In that case, IIUC petelomax correctly, it wouldn't make a difference in his implementation, since the context is still different - so you'd still get that error since variable a was declared in the wrong context.

atom b = 9 
b += 1 

No, not in petelomax's implementation, because the context is different.

Edit: well, not directly of course. You probably would be able to do something like "set_string_exec_variable(context, "b", 10)" or "get_string_exec_variable(context, "b")" though.

41. Re: string_exec()

• Posted by ryanj Mar 04, 2015
• 2197 views

I still do not understand why string_exec() would be useful. Would someone please show a code example that demonstrates what string_exec() can do that can't currently be done in Euphoria, or that would make something easier to do?

43. Re: string_exec()

• Posted by dcuny Mar 05, 2015
• 2165 views

It's pretty easy to write a string_exec() routine that does this:

string_exec("5+4") 

Since Euphoria can see it's own variables and values (trace does this), it should be straight forward to do this as well:

string_exec("5+4-myVariable") 

And since Euphoria can see user defined functions, it's not hard to extend string_exec to be able to this as well:

string_exec("5+4-myRoutine(myVariable)") 

Is this what you're asking for? If so, it's a lot simpler than the solutions that have been proposed.

- David

45. Re: string_exec()

• Posted by katsmeow Mar 05, 2015
• 2213 views

string_exec("5+4") 

string_exec("5+4-myVariable") 

Well, except the programmer cannot see the var table. If you ask for the value of myVariable, it's more likely to crash because the var doesn't exist.

string_exec("5+4-myRoutine(myVariable)") 

You can of course put anything in the string, that doesn't mean it gets converted to IL code and executes in the scope of where it was called.

Show us. I'll pause taking Tiggr apart for a lil while, to see what you come up with.

Kat

47. Re: string_exec()

• Posted by petelomax Mar 05, 2015
• 2168 views

The only half-decent concrete example I've got is a REPL.

To be honest, I was expecting more interest, and examples...

My current thinking is that sting_exec can only see the things it has been explicitly told about. The reference-counted nature of the language should make it easy/fast to shunt even big data into and out of workspaces as needed.

One specific case I mused on the other day was that if it automatically granted full access to the underlying data, then string_exec("symtab=0") would guarantee instant death with no error, no warning, and no diagnostics. Since the presence of string_exec means a complete embedded copy of the compiler/interpreter, literally hundreds of other identifiers that could cause strange and highly intermittent bugs might also be exposed. While I probably/hopefully could avoid such issues, I took that level of potential devastation as fair warning.

Indeed, from a programming language perspective such things have zero access to the application's data space, except for some user-defined bits in a handful of specific tables, that it doesn't even know the names of.

I've gone with "context" as in context = string_exec(context,code). The (as yet loosely defined) context would contain extend-able symbol and lookup tables, code and data block(s), result codes, and anything else needed.

One other thing I'd like to mention is Matt's embed_routine idea. I've no real problem with that method of granting selected access to the underlying application's data, except for potential confusion if it plays with a different "mytable" to the one in the string_exec workspace. Hence I'm somewhat hesitant on that.

Pete

49. Re: string_exec()

• Posted by dcuny Mar 05, 2015
• 2167 views

As a follow up, I'm not sure I understand why you need to see Euphoria's variables. For example:

a = string_exec("a+b") 

The code that builds the string "a+b" obviously thinks there are variables called a and b, because it built a string expression containing those variables. But how does the code know about them?

Put a different way, if string_exec() were able to used named variables, would that be enough? For example:

sequence context = {{"a", 100}, {"b", 200}} 
string_exec(context, "myRoutine(a+b)+sin(a)") 

Would something like that satisfy your needs? That would be even simpler to write, and it could be written in pure Euphoria. I could cannibalize the parser I've written for Py to do this.

The question then becomes what's the best way to pass values of variables to string_exec. For something like a spreadsheet, the context would be pretty clunky. Passing the routine_id of a user-defined function that would resolve variables would be a more flexible approach, along the lines of:

sequence context = {{"a", 100}, {"b", 200}} 
 
-- returns {value, success_flag} 
function lookup_variable(sequence varName) 
   for i = 1 to length(context) 
      if equal(context[i][1], varName) then 
         -- return value and success flag 
         return {context[i][2], 1} 
      end if 
   end for 
   -- return bogus value and failure flag 
   return {0, 0} 
end function 
 
string_exec(routine_id("lookup_variable"), "myRoutine(a+b)+sin(a)") 

Or the function lookup_variable() could be supplied, and you'd just recode it to your needs.

Would that work?

- David

51. Re: string_exec()

• Posted by dcuny Mar 05, 2015
• 2101 views

I suspect that for Kat's use case, it's probably not necessary anyway (see prior post).

- David

53. Re: string_exec()

• Posted by katsmeow Mar 05, 2015
• 2104 views

a = string_exec("a+b") 

It might not know about them. Do you have the variable TheSecondCowThatJoeBought?

What if you were in the 3rd grade, and i offered to teach you a new trick: how to calculate the average age of Joe's cow herd, but you are unable to learn new procedures or functions or variable names?

Kat

55. Re: string_exec()

• Posted by mattlewis (admin) Mar 05, 2015
• 2098 views

string_exec("5+4-myVariable") 

Right, but that's using all new stuff, not trying to look stuff up. As I said, my preference is to keep that stuff sandboxed but allow integration points to be defined.

Matt

57. Re: string_exec()

• Posted by mattlewis (admin) Mar 05, 2015
• 2090 views

I guess it depends on what you're doing. I assume, however, that the point of running some dynamic code is to have it affect your program. If it's completely isolated from your program it doesn't seem very useful.

Matt

59. Re: string_exec()

• Posted by jimcbrown (admin) Mar 05, 2015
• 2101 views

I don't think so. Doing what you want via string_exec() would almost seem to require that string_exec() perform dynamic runtime self-modifying bytecode manipulation anyways. Which certainly isn't out of the question - but if you're going to be manipulating bytecode anyways, why not just go straight to the source instead of dealing with an extra layer?

Another point is that implementing a sandbox'd string_exec() or eueval4.1 is a lot easier than implementing an unsandbox'd string execution mechanism via self-modifying bytecode.

Edit: Although there may be no grey area, I think that there is room for both concepts to co-exist side-by-side (at least IMVHO).

61. Re: string_exec()

• Posted by katsmeow Mar 05, 2015
• 2060 views

I said "string execution". On an OE forum, i expected everyone would understand the string was of OE code. Any OE code. I intentially did not restrict the code to certain subsets of OE.

My solution doesn't need to know about it's variables any more than the encompassing program does (which is a different thread, and i think it does). The way Matt, Derek, petelomax, etc see sandboxed string execution, one must know a var exists and be able to copy it into the sandbox, and then know it exists inside the sandbox and have code already written to copy it out of the sandbox.

Way back when, the solution was dynamic includes, but of course they couldn't be un-included, and there were other issues (like eventually you simply ran out of the encompassing program). Dynamic includes were normal includes, but sprinkled thruout the program, possibly written by the program itself before they were included. I don't recall anyone suggesting those be sandboxed.

Pleading for string execution is like beating a dead horse, and i am weary and feel hopeless. It's not going to happen. I made a mistake sticking with wintel, trying nix, and waiting on Euphoria to grow up. It's that simple.

Kat

63. Re: string_exec()

• Posted by mattlewis (admin) Mar 05, 2015
• 2060 views

You talk like this is an easy thing to make happen and we're not doing it to spite you. There are legitimate concerns that a lot of people have in addition to the implementation challenges, which are significant. I'm not your slave and trying to guilt trip me into something like this isn't going to work, either.

Matt

65. Re: string_exec()

• Posted by jimcbrown (admin) Mar 06, 2015
• 2066 views

Please take any venting off the forum.

67. Re: string_exec()

• Posted by jimcbrown (admin) Mar 06, 2015
• 2036 views

Agreed.

Hmm. Not sure I'd agree with you here. If those statements were directed at me, and Mmattlewis was trying to protest (to me) against being forced into working on coding something that he really didn't want to do, I wouldn't feel that way.

katsmeow has not been slapped with a CodeOfConduct violation.

I was under the impression that what katsmeow really wanted was dynamic/self-modifying code + REPL. OTOH, the entire kitchen sink. Which is something I support.

I think that maybe it is time to get something a whole lot more formal. Perhaps something along the lines of what's described here: http://www.bradapp.com/docs/sdd.html

69. Re: string_exec()

• Posted by dcuny Mar 06, 2015
• 2009 views

I got that impression as well. I'm just trying to figure out how the mechanics of that would work.

There are a couple bits that aren't clear to me. The where does "top level" code is one of them.

The other is how existing code gets to know about the new routines, since none of it was visible at the time of compilation.

If the request is allowing self-modifying code, what exactly does that mean? Can a routine be redefined?

I think what I've suggested answers both of those questions, but doesn't answer the question of how to add top-level variables.

I'm just trying to understand what's being ask for.

- David

71. Re: string_exec()

• Posted by jimcbrown (admin) Mar 06, 2015
• 2033 views

Logical.

What I had envisioned includes the ability to modify true top-level code and then re-execute it.

So top level code would continue to live in the top level, with no special scoping or protection from pre-existing top level code.

Ironically, top level code is the easiest to deal with in a BCEL library since there's fewer internal data structures that need to be rearranged to deal with it (as compared to creating a whole new routine, for example).

This is hard to do since in addition to IL itself we'd have to update the internal data structures that manage routines. But the goal is to allow one to write whole new routines on the fly and call them.

Yes.

This also requires modifying internal data structures, but it's probably a little easier than deal with routines.

73. Re: string_exec()

• Posted by katsmeow Mar 06, 2015
• 1978 views

Pronounce "gosub" like "routine_id". Gosub/routine_id means the block of memory where the eval/exec takes place does not need to displace existing code (see http://openeuphoria.org/forum/m/127273.wc), or be rearranged if a max size is allocated (altho it could be alloc'd and free'd, perhaps), and (almost) any header info (such as parameters restricting scope to the location it gosub'd from) can be static at compile time. This is exactly how i did it (except i emulated the gosub with pointer swaps instead of using the stack, for speed reasons).

Kat