1. Long start-up time of a program with include std/
- Posted by Insolor Feb 28, 2012
- 2468 views
Hi to everyone.
I have written a test program to measure a start-up time of a program in Euphoria:
include std/console.e include std/math.e function runit(sequence s, integer n) sequence test = repeat(0,n+2) atom t,d for i = 1 to n do t = time() system("eui " & s, 2) test[i] = time()-t test[n+1] += test[i] end for test[n+1] /= n for i = 1 to n do d = abs(test[n+1]-test[i]) if d>test[n+2] then test[n+2] = d end if end for return test end function constant header = " #1 #2 #3 average delta\n" constant format = "%1.5f %1.5f %1.5f %1.5f %1.5f\n" puts(1,"Running an empty program:\n") puts(1,header) printf(1,format,runit("NUL",3)) system("echo include NUL > nulinc.exw") puts(1,"\n") puts(1,"Running a program that includes an empty file:\n") puts(1,header) printf(1,format,runit("nulinc",3)) system("echo include std/console.e > stdcon.exw") puts(1,"\n") puts(1,"Running a program with an only 'include std/console.e' line:\n") puts(1,header) printf(1,format,runit("stdcon",3)) any_key()
On ASUS notebook with 1.73GHz CPU and 0.98 GB RAM, Win XP SP3, I have got these results:
Running an empty program: #1 #2 #3 average delta 0.25000 0.29700 0.21900 0.25533 0.04167 Running a program that includes an empty file: #1 #2 #3 average delta 0.26600 0.28100 0.25000 0.26567 0.01567 Running a program with an only 'include std/console.e' line: #1 #2 #3 average delta 1.98400 2.03200 2.15600 2.05733 0.09867 Press Any Key to continue...
Why the start-up time of the third program is so long? I guess it is because of the new advanced preprocessor features, used in the standard includes, am I right? Is there any workaround?
P.S. eui version is
Euphoria Interpreter v4.0.3 development Windows, Using Managed Memory Revision Date: 2011-06-24 13:43:18, Id: 4985:26b65e77e3d9
2. Re: Long start-up time of a program with include std/
- Posted by _tom (admin) Feb 29, 2012
- 2419 views
The std/console.e file adds many lines of code to your program; its includes have more includes amounting to about half of the entire standard library.
After compiling (euc) or binding, unused lines of library code are not included in your program.
3. Re: Long start-up time of a program with include std/
- Posted by Insolor Feb 29, 2012
- 2382 views
Anyway, IMO 2 seconds long startup time is TOO much for interpretable language. There is something to be optimized: the parser, or the library, or both.
4. Re: Long start-up time of a program with include std/
- Posted by DerekParnell (admin) Feb 29, 2012
- 2383 views
Why the start-up time of the third program is so long? I guess it is because of the new advanced preprocessor features, used in the standard includes, am I right? Is there any workaround?
It has nothing to do with preprocessor functionality.
Eu4 parses every line of code in the application, converting them to IL (intermediate language) before executing the first line it has to. It does this for a few reasons, including
- Ensuring that the program will not run if there is any coding mistake in it,
- Ensuring that all forward references can be resolved.
The included file std/console.e is a large file that in turn includes other standard library files, which include still other files and so on. This means that when using the std/console.e module, your application will parse and convert many thousands of lines of code ... in case your application might need them.
The work around is to create your own version of console.e that only contains the things that your application actually needs. Note that when you bind or translate-to-C, the resulting application not only removes the parse-convert steps, but also strips out routines that your applications will not need.
5. Re: Long start-up time of a program with include std/
- Posted by Insolor Mar 01, 2012
- 2357 views
Ok, now I see that the problem is in the library.
6. Re: Long start-up time of a program with include std/
- Posted by AndySerpa Mar 02, 2012
- 2282 views
I'm pretty sure there is actually something wrong with the last released version or two in regards to startup. There are definitely memory leaks and other weirdness (freeze-ups on crashes, and some other stuff), although I haven't been able to nail it down to any easily demonstrated errors since they only seem to occur with large complicated programs.
But I have been using a lot of virtual machines the last year or so (either the Microsoft one or VirtualBox, both running Windows XP), and when I run Eu on them with the newest versions the startup times are insanely long, like 30 seconds to a minute. (This actually is easily demonstrated as it applies to any Eu program on a Windows VM.) But the exact same program using version 4.0.1 (and maybe 4.0.2, can't remember) starts up more or less instantly, so it isn't a matter of just lots to parse or whatever. Why the VM brings out the problem to such an extent, I don't know, but like I said I have documented several other weird things with 4.0.3 (not on a VM). They always seem to involve programs with lots/complicated includes. A couple of times I've had programs persistently crash on start up giving an error that doesn't make sense (like crashing on a constant declaration) and "fixed" it just by entering a do-nothing line like "constant dummyvar = 1". I had to do this with one of the standard includes. I've also observed memory leaks and eu programs corrupting the console after they've been run. Often times a program will freeze on a crash. All of my programs are console-based Windows programs. Have any memory bugs been fixed in the current development versions?
7. Re: Long start-up time of a program with include std/
- Posted by mattlewis (admin) Mar 02, 2012
- 2282 views
A couple of times I've had programs persistently crash on start up giving an error that doesn't make sense (like crashing on a constant declaration) and "fixed" it just by entering a do-nothing line like "constant dummyvar = 1". I had to do this with one of the standard includes.
If you come across this again, or can still reproduce the crash, please try using either eu.ex or int.ex to interpret the program. This should result in a standard euphoria error message when it happens, rather than the translated C code crashing. Then make sure you write a ticket with the results. 
I've also observed memory leaks and eu programs corrupting the console after they've been run. Often times a program will freeze on a crash. All of my programs are console-based Windows programs. Have any memory bugs been fixed in the current development versions?
I think that a few things that could have caused memory leaks have been cleaned up, in particular some cases where we weren't properly disposing of temps. It's possible that there were others, but I'm just going by memory.
Can you be more specific as to how the console was corrupted?
Matt
8. Re: Long start-up time of a program with include std/
- Posted by AndySerpa Mar 02, 2012
- 2179 views
[quote mattlewis]
I think that a few things that could have caused memory leaks have been cleaned up, in particular some cases where we weren't properly disposing of temps. It's possible that there were others, but I'm just going by memory.
Can you be more specific as to how the console was corrupted?
First, the memory leaks. I've got an "eval" library that evaluates string expressions, i.e. the string "2+2" will return 4, but it can also do Euphoric stuff and return sequences so it isn't only for math. I've had this library back to Eu 2.5 or so, I use it a lot. I noticed in the newer versions of Eu it causes memory leaks, but only when returning sequence values. As you can imagine, a string eval library uses indirect calls by routine_id quite a bit. I suspect that's where whatever leak is occuring is occuring, something to do with indirect calls (which could be nested several levels) where sequence values are involved. I have gone over it with a fine-tooth comb and am fairly confident the library is sound and the problem is within Eu. It really has to be since it is a pure Eu library with no low-level routines, it has no globals that could just be getting bigger and if it crashes it usually hangs instead of a proper Eu crash.
Anyway, I've got a few programs that rely heavily on those routines and I so I just live with the memory leaks. But they need to process a ton of data so instead of making a big loop I end up restarting the program a lot to "reset" the memory problems. Since this is a Windows console program, it is constantly popping up a new console (I usually just run with the normal interpreter) with its auto-restart of itself, which gets in the way of whatever else I'm doing when I really want it to be in the background. So to avoid this, I have sometimes open a console first, and then run it from the prompt with the console interpreter to keep all the text output confined to the one window, and then when it restarts itself it still remains in that same console. So...what happens when it finishes (even if you just run it once) with one of these leaky programs where it goes back to the prompt is that the prompt is corrupted, I can't type properly in it anymore or it ignores letters, for instance if I type "dir" it will say "ir" not a command or something like that. That's what I mean about a corrupted console.
And what's with the VMs and super-long start times on newer Eu's anyway? There is definitely something gone wrong. I've been living with these problems for at least a year so I've forgotten some of the details, just hoping you'll figure it out at some point. I can't remember if the first version to do that was 4.0.3 or 4.0.2. I can check. For a while I kept the older version on my VMs so they would go fast, but eventually it was becoming problematic for them not to be synced up with my main machine, so now I bind the programs from the main machine and copy them to the VMs to run. (Bound programs start right up.)
9. v4Re: Long start-up time of a program with include std/
- Posted by mattlewis (admin) Mar 03, 2012
- 2142 views
First, the memory leaks. I've got an "eval" library that evaluates string expressions, i.e. the string "2+2" will return 4, but it can also do Euphoric stuff and return sequences so it isn't only for math. I've had this library back to Eu 2.5 or so, I use it a lot. I noticed in the newer versions of Eu it causes memory leaks, but only when returning sequence values. As you can imagine, a string eval library uses indirect calls by routine_id quite a bit. I suspect that's where whatever leak is occuring is occuring, something to do with indirect calls (which could be nested several levels) where sequence values are involved. I have gone over it with a fine-tooth comb and am fairly confident the library is sound and the problem is within Eu. It really has to be since it is a pure Eu library with no low-level routines, it has no globals that could just be getting bigger and if it crashes it usually hangs instead of a proper Eu crash.
Is the library posted somewhere? If not, would you mind sending it to me? Once I can reproduce the leaks, I should be able to fix the memory leaks.
Anyway, I've got a few programs that rely heavily on those routines and I so I just live with the memory leaks. But they need to process a ton of data so instead of making a big loop I end up restarting the program a lot to "reset" the memory problems. Since this is a Windows console program, it is constantly popping up a new console (I usually just run with the normal interpreter) with its auto-restart of itself, which gets in the way of whatever else I'm doing when I really want it to be in the background. So to avoid this, I have sometimes open a console first, and then run it from the prompt with the console interpreter to keep all the text output confined to the one window, and then when it restarts itself it still remains in that same console. So...what happens when it finishes (even if you just run it once) with one of these leaky programs where it goes back to the prompt is that the prompt is corrupted, I can't type properly in it anymore or it ignores letters, for instance if I type "dir" it will say "ir" not a command or something like that. That's what I mean about a corrupted console.
That's very strange. I have no idea what might be causing that.
And what's with the VMs and super-long start times on newer Eu's anyway? There is definitely something gone wrong. I've been living with these problems for at least a year so I've forgotten some of the details, just hoping you'll figure it out at some point. I can't remember if the first version to do that was 4.0.3 or 4.0.2. I can check. For a while I kept the older version on my VMs so they would go fast, but eventually it was becoming problematic for them not to be synced up with my main machine, so now I bind the programs from the main machine and copy them to the VMs to run. (Bound programs start right up.)
That's very strange, too.
Matt
10. v4Re: Long start-up time of a program with include std/
- Posted by AndySerpa Mar 03, 2012
- 2145 views
I have to double-check everything I've said it has been so long since I tried to pin these things down. Where can I send you stuff?
11. v4Re: Long start-up time of a program with include std/
- Posted by mattlewis (admin) Mar 06, 2012
- 2006 views
I have to double-check everything I've said it has been so long since I tried to pin these things down. Where can I send you stuff?
Andy, I tried sending an email, but perhaps I have a bad address. You can reach me with an openeuphoria.org email address using my user name (mattlewis).
Matt
12. v4Re: Long start-up time of a program with include std/
- Posted by AndySerpa Mar 10, 2012
- 1817 views
I have to double-check everything I've said it has been so long since I tried to pin these things down. Where can I send you stuff?
Andy, I tried sending an email, but perhaps I have a bad address. You can reach me with an openeuphoria.org email address using my user name (mattlewis).
Matt
I got it. Just busy.
13. Re: Long start-up time of a program with include std/
- Posted by mattlewis (admin) Mar 11, 2012
- 1689 views
The std/console.e file adds many lines of code to your program; its includes have more includes amounting to about half of the entire standard library.
After compiling (euc) or binding, unused lines of library code are not included in your program.
I've occasionally thought about this issue. The standard library has gotten much bigger in 4.0, which is a good thing. We've tried to spread stuff out in a logical fashion, but a lot of the more sophisticated routines rely on other parts of the library. For instance, text routines are kept in std/text.e, and since text is stored as a sequence, it's not surprising that some text routines will be dependent on more generic sequence routines that exist in std/sequence.e.
I think that the solution is to do some dependency analysis in order to minimize the required footprint of any particular file in the standard library. Fortunately, the new scoping rules in 4.0 allow this to be done transparently to the coder who is using the standard library.
Euphoria includes a tool that can help with this, at least as far as documenting the inter-dependencies of the files and their routines. Eudis has options to build these graphs in an html document. This is far from a trivial job, but I think it's probably worth it.
Matt
