1. Adding a Builtin to EUPHORIA

This mystifies me but I have heard this before. There is no compelling need to add a EUPHORIA builtin when you are just adding routines. Regex is in the backend, but the only reasons I remember given is to get rid of the need of adding dll files to the distribution of EUPHORIA and for speed.

Regex can be compiled as a dll and then added via the dll interface. As developers we could have done it that way. It would have been easier and less bug prone. If dll is too slow we ought to work out how we can make it faster.

Shawn Pringle

Forked from Re: Future direction and purpose of Euphoria?

BRyan said...

I think that there would be more contributions to Euphoria if some knowledgeable person

would document in "complete detailed" how to add a built-in "euphoria coded" function to the

source code.

This is one of the biggest stumbling blocks for the average user to contribute ideas.

The only documentation for adding to the source is still in a 2.x document which was

included by Rob years ago.

new topic     » topic index » view message » categorize

2. Re: Adding a Builtin to EUPHORIA

SDPringle said...

This mystifies me but I have heard this before. There is no compelling need to add a EUPHORIA builtin when you are just adding routines.

Depends on the nature of the routine. I think BRyan is trying to add something like a variable_id() (routine_id() but for variables), or something.

SDPringle said...

Regex is in the backend, but the only reasons I remember given is to get rid of the need of adding dll files to the distribution of EUPHORIA and for speed.

It doesn't speed things up and adding a shared library is not a big deal. The real reason was DOS support.

Since DOS was dropped, there no longer is any compelling reason to include regex the way we do in the source code. Using a shared library is actually better, as it makes it possible to upgrade the PCRE component indepedently.

SDPringle said...

Regex can be compiled as a dll and then added via the dll interface. As developers we could have done it that way. It would have been easier and less bug prone. If dll is too slow we ought to work out how we can make it faster.

Agreed. There's no backwards compatibility concerns in doing so either, since we can hide all that in the stdlib.

new topic     » goto parent     » topic index » view message » categorize

3. Re: Adding a Builtin to EUPHORIA

SDPringle said...

Regex is in the backend, but the only reasons I remember given is to get rid of the need of adding dll files to the distribution of EUPHORIA and for speed.

Regex can be compiled as a dll and then added via the dll interface. As developers we could have done it that way. It would have been easier and less bug prone. If dll is too slow we ought to work out how we can make it faster.

"The only reasons" you mention are huge, though. It might have been less bug prone. It definitely would have been slower. Basically, having regexes is pretty much an entry level requirement for a language. Requiring a separate library to be distributed would be crazy.

Matt

new topic     » goto parent     » topic index » view message » categorize

4. Re: Adding a Builtin to EUPHORIA

mattlewis said...
SDPringle said...

Regex is in the backend, but the only reasons I remember given is to get rid of the need of adding dll files to the distribution of EUPHORIA and for speed.

Regex can be compiled as a dll and then added via the dll interface. As developers we could have done it that way. It would have been easier and less bug prone. If dll is too slow we ought to work out how we can make it faster.

"The only reasons" you mention are huge, though. It might have been less bug prone. It definitely would have been slower.

I don't think it would have been that much slower. It's how we handle a lot of the stuff in the stdlib, like msgboxes and mmap.

mattlewis said...

Basically, having regexes is pretty much an entry level requirement for a language.

Agreed.

mattlewis said...

Requiring a separate library to be distributed would be crazy.

Agreed, if it had to be distributed separately. I don't see it being a big deal to have to ship the one PCRE library with Euphoria though. You ever see how many dlls a Windoze installation of Python comes with?

new topic     » goto parent     » topic index » view message » categorize

5. Re: Adding a Builtin to EUPHORIA

jimcbrown said...

I don't think [regex] would have been that much slower. It's how we handle a lot of the stuff in the stdlib, like msgboxes and mmap.

There would be *a lot* more manual memory management. And not just the stuff we have to do to convert a sequence into a C string, but a lot of the structures and things that get passed around. Stuff that goes right on the stack in C (very fast) would be allocated on the heap. For one off stuff, it's not a big deal, but regexes are often used (I know I do) for processing lots of data, over and over.

jimcbrown said...
mattlewis said...

Requiring a separate library to be distributed would be crazy.

Agreed, if it had to be distributed separately. I don't see it being a big deal to have to ship the one PCRE library with Euphoria though. You ever see how many dlls a Windoze installation of Python comes with?

Except one of the selling points of euphoria is how easy it is to distribute stuff. You can easily put it all together into a single executable (plus whatever else you add, of course). But requiring an additional runtime to go along with it is a dangerous and painful path (MSVCRT, VBRUN, etc).

Matt

new topic     » goto parent     » topic index » view message » categorize

6. Re: Adding a Builtin to EUPHORIA

mattlewis said...
jimcbrown said...

I don't think [regex] would have been that much slower. It's how we handle a lot of the stuff in the stdlib, like msgboxes and mmap.

There would be *a lot* more manual memory management. And not just the stuff we have to do to convert a sequence into a C string, but a lot of the structures and things that get passed around. Stuff that goes right on the stack in C (very fast) would be allocated on the heap. For one off stuff, it's not a big deal, but regexes are often used (I know I do) for processing lots of data, over and over.

I don't understand why the existing type system (with regexes being a unique type) wouldn't work with PCRE being a shared library. Jeremy easily went back and forth on this in the code when we were debating this issue.

mattlewis said...
jimcbrown said...
mattlewis said...

Requiring a separate library to be distributed would be crazy.

Agreed, if it had to be distributed separately. I don't see it being a big deal to have to ship the one PCRE library with Euphoria though. You ever see how many dlls a Windoze installation of Python comes with?

Except one of the selling points of euphoria is how easy it is to distribute stuff. You can easily put it all together into a single executable (plus whatever else you add, of course). But requiring an additional runtime to go along with it is a dangerous and painful path (MSVCRT, VBRUN, etc).

Matt

We can still do that. Just carry around both the shared and static versions of the PCRE library. If py2exe can turn a Python app (and all its required runtimes) into a single executable, I don't see why we'd have so much trouble with just one library.

new topic     » goto parent     » topic index » view message » categorize

7. Re: Adding a Builtin to EUPHORIA

jimcbrown said...
mattlewis said...
jimcbrown said...

I don't think [regex] would have been that much slower. It's how we handle a lot of the stuff in the stdlib, like msgboxes and mmap.

There would be *a lot* more manual memory management. And not just the stuff we have to do to convert a sequence into a C string, but a lot of the structures and things that get passed around. Stuff that goes right on the stack in C (very fast) would be allocated on the heap. For one off stuff, it's not a big deal, but regexes are often used (I know I do) for processing lots of data, over and over.

I don't understand why the existing type system (with regexes being a unique type) wouldn't work with PCRE being a shared library. Jeremy easily went back and forth on this in the code when we were debating this issue.

Oh, yeah, it would work to put our be_pcre.c code into the library along with the actual PCRE code. I got the impression that Shawn was talking about having a euphoria wrapper for a normal PCRE library, though.

jimcbrown said...

We can still do that. Just carry around both the shared and static versions of the PCRE library. If py2exe can turn a Python app (and all its required runtimes) into a single executable, I don't see why we'd have so much trouble with just one library.

This sounds like a completely losing proposition to me. We've just complicated our build and distribution (and probably users' build and distribution of their own programs) for pretty much no gain.

Matt

new topic     » goto parent     » topic index » view message » categorize

8. Re: Adding a Builtin to EUPHORIA

mattlewis said...
jimcbrown said...
mattlewis said...
jimcbrown said...

I don't think [regex] would have been that much slower. It's how we handle a lot of the stuff in the stdlib, like msgboxes and mmap.

There would be *a lot* more manual memory management. And not just the stuff we have to do to convert a sequence into a C string, but a lot of the structures and things that get passed around. Stuff that goes right on the stack in C (very fast) would be allocated on the heap. For one off stuff, it's not a big deal, but regexes are often used (I know I do) for processing lots of data, over and over.

I don't understand why the existing type system (with regexes being a unique type) wouldn't work with PCRE being a shared library. Jeremy easily went back and forth on this in the code when we were debating this issue.

Oh, yeah, it would work to put our be_pcre.c code into the library along with the actual PCRE code. I got the impression that Shawn was talking about having a euphoria wrapper for a normal PCRE library, though.

Oh, I think what Jeremy did used a normal PCRE library with a shim. Anyways, I think we're on the same page here now.

mattlewis said...
jimcbrown said...

We can still do that. Just carry around both the shared and static versions of the PCRE library. If py2exe can turn a Python app (and all its required runtimes) into a single executable, I don't see why we'd have so much trouble with just one library.

This sounds like a completely losing proposition to me. We've just complicated our build and distribution (and probably users' build and distribution of their own programs) for pretty much no gain.

Matt

Running py2exe is not complicated. We're talking about an extra flag to euc and bind, and maybe a few extra files in the bin directory. If done right, it doesn't have to be complicated. (For backwards copmatibility, euc and bind would probably default to sticking the static PCRE into the final executable, and you'd have flags to use a shared version or to kick regex out altogether.)

new topic     » goto parent     » topic index » view message » categorize

9. Re: Adding a Builtin to EUPHORIA

Back to BRyan's topic:

BRyan said...

I think that there would be more contributions to Euphoria if some knowledgeable person

would document in "complete detailed" how to add a built-in "euphoria coded" function to the

source code.

This is one of the biggest stumbling blocks for the average user to contribute ideas.

The only documentation for adding to the source is still in a 2.x document which was

included by Rob years ago.

It's true that the documentation hasn't kept completely up with the code, and that's unfortunate. Largely from memory, here's what's required (maybe we can flesh this out and get it into the docs, or at least on the wiki):

  • Add the new name to reswords.e, opnames.e, opnames.h, keylist.e, redef.e, include/euphoria/keywords.e
  • shift.e: add it into op_info, with metadata about the opcode for your built-in
  • Add a new case in emit.e, or add the handling for it to an existing, compatible case (i.e., functions with the same sort of parameters)
  • be_execute.c:
    • add to appropriate place in code_set_pointers()
    • add to localjumptab
    • add case for execution to do_exec(), which may simply be a call to an implementation in be_runtime.c (which translated code could use, as well)
  • compile.e
    • add handler for translating
    • put call to handler in init_opcodes()
  • execute.e
    • add wrapper for execution
    • put call to wrapper in do_exec()
  • dis.e: add handler for your new function (it can probably use one of the generic formatters, unary, punary, binary, pbinary, etc)
  • Add documentation for your new routine in appropriate place (either a related std library file or a separate documentation file)

Adding a keyword or other new language feature requires changes to parsing and possibly scanning, inlining and forward referencing, potentially in addition to all of the above. This is typically more difficult than adding a built-in routine. Machine procs/funcs are a bit simpler.

Matt

new topic     » goto parent     » topic index » view message » categorize

Search



Quick Links

User menu

Not signed in.

Misc Menu