Re: Multiple interpreter instances
- Posted by ghaberek (admin) Mar 01, 2015
- 3335 views
I have had an idea several years ago. What if the euphoria "backend" was able to run in multiple instances, each instance in a different thread? I need to study the internal working of OE more, but i believe it would require converting all global state data to be put inside arrays/sequences and add a manager that can create and destroy instances, each with it's own index/id number, and a sharing interface which all instances can access, with some sort of semaphore mechanism. This would open up some interesting possibilities:
- Multi-threading and blocking I/O would be possible, because a new thread would actually be a different interpreter, completely independent.
- Running scripts or generating code on-the-fly would be possible, by simply starting a new interpreter instance.
- Processing data with multiple cores would be possible, by starting multiple instances of the same code (library or routine) on multiple cores.
- By adding a few built-in routines, a "interpreter instance" or "thread" manager could be accessed to start, stop, or monitor threads, as well as share data between them in a thread-safe way.
- A eui.dll or eui.so could be linked to a main program to execute euphoria code, and share data with that program using the same system that shares data between interpreter instances.
What do you think?
I've considered this project a few times and I think it is entirely possible. What you need at the basic level is something like memcached for shared data storage and then some sort of message-passing interface like ZeroMQ to coordinate all of the nodes. These pieces could be written in Euphoria or you could use those libraries directly. The best part is that such a system wouldn't really care what a "node" looks like, be it a thread, CPU, or another machine entirely. You could probabyl scale out to hundreds of nodes quite easily. Another option would be to use Gearman to do a lot of the heavy lifting. It is already designed for such things.
-Greg
