OpenEuphoria

2. Re: ver 4 source preproc..e ?

• Posted by jeremy (admin) Aug 03, 2009
• 4701 views
• Last edited Aug 04, 2009

Bernie,

preproc.e is a new feature that was added this weekend. I still need to write up a section in the manual about it. It will be a fairly detailed section, probably a mini-guide. It is a hook into Euphoria that allows users to provide a pre-processor of their own depending on file extension. Anyone can write a pre-processor to do anything they want. I have included two demo's of useful pre-processors in the demo/preproc directory but for ease of of the example, let's say that you wanted to create a pre-processor that prints the filename of every file processed by euphoria. Let's write that pre-processor right here. I am sure I'll incorporate this into the docs, so it is not wasted time at all...

First, for your pre-processor to work with Euphoria it must adhere to 3 specifications.

1. It must accept as the first parameter the filename to process.
2. It must accept as the second parameter the filename to write the post-processed file to.
3. It must exit with a non-zero error code if the pre-processing failed, which tells Euphoria to cease the parsing, just as though a syntax error has occurred.

If your script/program meets that criteria, then it can be used as a pre-processor. So let's write our pre-processor.

-- fname_preproc.ex 
-- Print the filename when the file is included 
 
include std/io.e 
 
sequence cmds = command_line() 
sequence input_filename = cmds[3] 
sequence output_filename = cmds[4] 
 
sequence content = read_file(input_filename) 
content = "puts(1, \"Filename: " & input_filename & "\")\n" & content 
 
write_file(output_filename, content) 
abort(0) 

So, this program simply appends a line of code to the top of every file processed that reads puts(1, "Filename: xyz.e\n").

Now, let's create a sample program so we can see this work.

-- hello.e 
public procedure say_hello() 
    puts(1, "Hello, World!\n") 
end procedure 
 
-- main.ex 
include hello.e 
 
say_hello() 

So, we have now have three files. fname_preproc.ex which is our pre-processor and two program files, hello.e and main.ex. Let's run it with Euphoria and the new pre-processor interface. Watch the output.

C:\euphoria> eui -p e,ex:fname_preproc.ex main.ex 
Filename: main.ex 
Filename: hello.e 
Hello, World! 

So, how did this work? First, let me explain the -p parameter to Euphoria. It allows the defining of one or more pre-processors. It's syntax is easy. The first section is a comma delimited list of extensions that should be associated with the pre-processor. In our case, it is normal Euphoria files, e and ex. The second section is the command to run. This can be any command, for instance, it can be a batch file, a perl script, an executable or a Euphoria file. If the pre-processor's extension is .ex, then it is automatically called with eui, there is no reason to specify eui.

So, we have just told Euphoria, when it sees a .e or .ex file, before it ever opens the file to read it, it should send it through the command fname_preproc.ex. The first file that Euphoria sees that matches an extension list is main.ex. Thus, it calls fname_preproc.ex with an input filename (main.ex) and an output filename (main_post.ex ... we will make it DOS compliant before final release). fname_preproc.ex reads main.ex into a sequence and writes it back out, but before doing so, it add's the puts() line. Thus, main.ex after pre-processing looks like

puts(1, "Filename: main.ex\n") 
include hello.e 
 
say_hello() 

The same thing happens for hello.e

puts(1, "Filename: hello.e\n") 
public procedure say_hello() 
    puts(1, "Hello, World!\n") 
end procedure 

Now, Euphoria does not execute the original sources but instead the post-processed sources, thus it prints the filename's of each file processed.

Ok, let's go on to a litte more complicated, but not really, example. Literate Programming. Literate Programming simply means that documentation is primary and code is actually secondary. You can read more about Literate Programming on http://literateprogramming.com/

So, here is an example source file for Euphoria Literate Programming

This is a traditional Hello World program except that it is done using 
the Literate Programming style. The filename is hello.lex, i.e. 
Literate Euphoria. 
 
We do not want to repeat ourself later on, so let's create a function 
that will say hello to the world or to someone specified as the first 
parameter of the function. 
 
<eucode> 
procedure say_hello(to_who="World") 
    printf(1, "Hello, %s!\n", { to_who }) 
end procedure 
</eucode> 
 
Now that we have our function, let's start by saying hello to the world. 
 
<eucode> 
say_hello() 
</eucode> 
 
Now, let's spice things up a bit and say hello 3 times to the person 
specified on the command line. 
 
<eucode> 
sequence cmds = command_line() 
 
for i = 1 to 10 do 
    say_hello(cmds[3]) 
end for 
</eucode> 
 
There you have it! This is an example of literate programming with 
Euphoria. 

Ok, great! Now, how in the world will this work with Euphoria? Well, it's cake with the new pre-processor interface. I am not going to create the code here for the literate programming style pre-processor, instead it is included as a pre-processor demo with Euphoria. I would encourage you to view the code before continuing to read this message. The code can be read online at http://rapideuphoria.svn.sourceforge.net/viewvc/rapideuphoria/trunk/demo/preproc/literate.ex?revision=2327&view=markup

Now, all we have to do is

C:\Euphoria> eui -p lex:literate.ex hello.lex Bernie 
Hello, World! 
Hello, Bernie! 
Hello, Bernie! 
Hello, Bernie! 

Now, who wants to type in -p lex:literate.ex all the time? No one I am sure, so, create a file in %EUDIR% named euinc.conf and add this line

-p lex,le:literate.ex 

Now that will be appended to each and every invocation of eui or euc and if you run or include a lex or le file, it will be handled properly.

C:\Euphoria> eui hello.lex Jeremy 
Hello, World! 
Hello, Jeremy! 
Hello, Jeremy! 
Hello, Jeremy! 

I hope you are seeing the benefit by now, but let me briefly introduce two other pre-processors. One is also a demo included in the demo/preproc directory. It's name is etml. It stands for Euphoria Text Markup Language.

<!-- Hello ETML World: hello.etml --> 
<html> 
  <head><title><%= title %></title></head> 
  <body> 
    <% for i = 1 to 3 do %> 
      <p>Hello, <%= name %>!</p> 
    <% end for %> 
  </body> 
</html> 

Now, let's create a program to use our new ETML template file:

-- hello.ex 
include std/map.e as m 
include hello.etml as hello -- Yes, include the file directly!!! 
 
m:map data = m:new() 
m:put(data, "title", "Hello, World!") 
m:put(data, "name", "Bernie") 
 
puts(1, hello:template(data) & "\n") 

That's it! The etml preprocessor is also available for reading online: http://rapideuphoria.svn.sourceforge.net/viewvc/rapideuphoria/trunk/demo/preproc/etml.ex?revision=2327&view=markup It's slightly more complex than literate.ex, but not much. How to run it?

C:\Euphoria> eui -p etml:etml.ex hello.ex 
<html> 
  <head><title>Hello, World!</title></head> 
  <body> 
    <p>Hello, Bernie!</p> 
    <p>Hello, Bernie!</p> 
    <p>Hello, Bernie!</p> 
  </body> 
</html> 

Now, one more example. Many know of the dot preprocessor written by David Cuny. It provides many additions on top of the normal Euphoria syntax. A while ago, I spoke with David and asked for permission to update dot for 4.0 and release it as dot4. It also is available in The Archive. However, since the addition of the pre-processor interface into Euphoria, I have modified it and new versions are available in my SVN repository, http://jeremy.cowgar.com/svn/dot4/trunk

It allows for all sorts of neat Euphoria additions such as structured sequence access, dot function calling, design by contract, one line if statements, ease of routine_id access and much more. An example program is in order

-- example.dex (DOT euphoria file) 
include std/io.e 
include std/text.e 
 
enum first_name, last_name, age 
sequence people = { 
  { "John", "Doe", 38 }, 
  { "Jane", "Smith", 44 } 
} 
 
procedure greet_person(sequence person) 
  -- Design by Contract 
  ?? length(person) = 3 -> "Invalid person sent to greet_person" 
   
  -- DOT access 
  STDOUT.printf("Hello, %s!\n", { person.first_name }) 
   
  -- One line if statement 
  ?? person.age > 18 : puts(1, "Person is an adult\n") 
end procedure 
 
-- More examples: 
for i = 1 to people.length() do 
  call_proc(@greet_person, { people.i }) 
  STDOUT.printf("%s is %d years old\n", { people.i.age }) 
end for 

So, this contains many statements that are not Euphoria compliant, however, with the pre-processor, that's no problem.

C:\Euphoria> eui -p de,dex:dot4.ex example.dex 

Of course, you can add -p de,dex:dot4.ex to your euinc.conf file as well to avoid typing it in all the time.

Now, one last thing to mention. Everyone is going to say, SPEED? Where's the SPEED? This is where Euphoria does a few tricks...

1. If no pre-processors are defined, then no checks are ever done for a pre-processor thus, those not using a pre-processor are not pentalized.
2. When a file is pre-processed, Euphoria caches the output into a file with the suffix _post addeded (meaning post processed). Then, the original file is only sent to the pre-processor if it is newer than the cached file. Thus, your first execution is slightly delayed, however, subsequent executions occur at full speed with no slow down what so ever.
3. Your pre-processor can also be compiled as a DLL avoiding the need for Euphoria to even spawn a new process to process your file for the first time, thus, first time executions can be at almost full speed (there is a slight delay because your pre-processor does parse the code and write the output before Euphoria works on it, however, in my tests with dot4, literate and etml, these delays are measured in micro seconds).

So, what will this do for Euphoria? Wow! Look at the opportunities! For instance, some popular programs right now use Kanarie which is a templating system for web applications. With the addition of the pre-processor and ETML, benchmarks (talked about on the dev list) already show that ETML with the pre-processor is more than 13 times faster! Literate programming has never been easier! DOT4 adds a whole host of features to Euphoria. Other pre-processors will begin to appear pushing the evelope of what can be done with Euphoria. Many of these pre-processors I am sure will add functionality to Euphoria that will either become very popular or quickly dropped by the way side. The pre-processor is the way for future functionality to be tested before ever being added to Euphoria. It's the way for every day users of Euphoria to create new syntax, test it on the users and core Euphoria developers picking these up for inclusion directly into Euphoria.

With the new pre-processor interface, Euphoria has become the most versitle language ever. Ok, I cannot prove that, but I'll bet you show off literate.ex to a few Literate Programmers and they will be in love with how easy this task is to perform with Euphoria. Web programmers will love how fast ETML is (it's translated directly into Euphoria code!). The list can go on and on.

Let us know what you think. I know this has been a long message, but hopefully it gives an idea for what the pre-processor is and what it's capable of.

Jeremy

4. Re: ver 4 source preproc..e ?

• Posted by SDPringle Aug 06, 2009
• 4608 views
• Last edited Aug 07, 2009

I like it. C macros anyone?

6. Re: ver 4 source preproc..e ?

• Posted by SDPringle Aug 07, 2009
• 4588 views

Sometimes I would like to do something like:

C wouldn't make you retype these things twice.. constant Monday = 1, Tuesday = 2, ... constant WeekDayNames = { "Monday", "Tuesday", ...

puts(1, WeekDayNames[Monday]) Prints Monday.

I wouldn't really use the C macro processor to do function inlining. I would like to see a method of only having to type the days of the week once in EUPHORIA. The C macro preprocessor allows you to do this. Until named enums and a function for converting values back to literals exist in EUPHORIA that may be what some people may do. I can't remember how they do that though.

Shawn

8. Re: ver 4 source preproc..e ?

• Posted by jeremy (admin) Aug 11, 2009
• 4513 views

I figured there were some languages that provided that level of pre-processor, I just didn't know which ones.

About one pre-processor to another... I was thinking of a few ways of solving that. Obviously I am not going to be able to solve everything, as you say. For example in dot4, you can reference a routine via @routine_name, however, in ETML, @routine_name will get a var from the parameters to the template. So, running one or the other first will goof up the next.

I'm up for suggestions, but I'm afraid that as you say, some pre-processors will just not be compatable with each other. That, however, does not take away the importance of the pre-processor. For instance, the idea of including a .h file directly and it's translated into Euphoria code for accessing the C functions contained therein, or eSQL, which allows you to include a .sql file generating common crud/finder methods for the tables/indexes found in the SQL file.

Anyway, I enjoyed your nice message for a change! It was helpful and not derogatory.

Jeremy

10. Re: ver 4 source preproc..e ?

• Posted by useless Aug 11, 2009
• 4539 views

Is this not akin to php as a preprocessor to apache?

useless

12. Re: ver 4 source preproc..e ?

• Posted by useless Aug 11, 2009
• 4478 views

Hmm, ok, i didn't think of Apache and PHP like that. I see Apache as a application that executes it's prebuilt code, and PHP is an extension to add user commands to it.

useless, as always.

14. Re: ver 4 source preproc..e ?

• Posted by useless Aug 11, 2009
• 4473 views

If Apache executed html, then we wouldn't need PHP. If Eu executed "function blarque()" then we wouldn't need a preprocessor. Apache is precompiled to merely to do a good job of sending the html out, you cannot tell Apache to do anything in the html.

useless

16. Re: ver 4 source preproc..e ?

• Posted by useless Aug 11, 2009
• 4486 views

That's a common occurance. So common, it tests reality and sanity. Would a preprocessor have helped our communication, or is there something else amiss?

useless