1. 4.0 and include statements
- Posted by mattlewis (admin) Jan 07, 2009
- 1004 views
In an effort to say ahead of Rowland, I've been trying to test new builds of euphoria 4.0 against Judith's IDE. It's a very large application, and it really abuses the forward referencing features due to the fact that most files don't have include statements for the files they use.
I think that the basic pattern is that the main file includes nearly everything, and the other files (of which there are many) occasionally include other files. This being eu3.1 based code, everything is declared either locally or globally, so the exposed symbols are visible throughout the app.
In 4.0, when it parses a file, and it resolves a symbol to a global symbol in another file that wasn't included by the current file, it treats this as a forward reference, since there could be a matching symbol somewhere else either in the same file, or in another file that was included, and we don't want the errant global symbol to override the symbol the programmer actually wanted.
So when the IDE is parsed, over the entire course, it ends up with over 60,000 unique forward references (this includes things like type checks). Which really, really slows down the parsing. I went through and added include ide.exw to the files, and the parsing time was drastically reduced. Note, it would have been better to have included the specific files that were used, but this was easy, and I just wanted to see the difference.
The moral of this story is that to play nice with 4.0, you shouldn't have files rely on other files to include the code you rely on. Of course, with public and export scopes, this will be enforced to a certain extent.
Matt
2. Re: 4.0 and include statements
- Posted by euphoric (admin) Jan 07, 2009
- 984 views
The moral of this story is that to play nice with 4.0, you shouldn't have files rely on other files to include the code you rely on. Of course, with public and export scopes, this will be enforced to a certain extent.
Is anybody updating Win32Lib or making a Win32Lib for Eu4.0 lib?
3. Re: 4.0 and include statements
- Posted by RStowasser Jan 08, 2009
- 987 views
So when the IDE is parsed, over the entire course, it ends up with over 60,000 unique forward references (this includes things like type checks). Which really, really slows down the parsing. I went through and added include ide.exw to the files, and the parsing time was drastically reduced. Note, it would have been better to have included the specific files that were used, but this was easy, and I just wanted to see the difference.
The moral of this story is that to play nice with 4.0, you shouldn't have files rely on other files to include the code you rely on. Of course, with public and export scopes, this will be enforced to a certain extent.
Matt
Yes, in the original state IDE has become very slow with Eu 40. The starting Splash Screen shows in about a minute compared to 4 to 5 seconds with Eu 3.1.1 on my system. Translated and compiled IDE starts immediately.
But as long as there is no official Win32lib for Eu40 which also uses some of the new strategies of Euphoria it makes no sense in my opinion to do substantial changes with IDE, apart from changing the include statements which you described. Using the standard library of Eu40 there would be a lot of places in IDE where the new provided functions and procedures could be applied.
Roland
4. Re: 4.0 and include statements
- Posted by RStowasser Jan 18, 2009
- 929 views
In 4.0, when it parses a file, and it resolves a symbol to a global symbol in another file that wasn't included by the current file, it treats this as a forward reference, since there could be a matching symbol somewhere else either in the same file, or in another file that was included, and we don't want the errant global symbol to override the symbol the programmer actually wanted.
So when the IDE is parsed, over the entire course, it ends up with over 60,000 unique forward references (this includes things like type checks). Which really, really slows down the parsing. I went through and added include ide.exw to the files, and the parsing time was drastically reduced. Note, it would have been better to have included the specific files that were used, but this was easy, and I just wanted to see the difference.
I hesitated until now, but I think this topic should be addressed.
If I run the original IDE.exw (I had to change int to integer in some files), using exw rev 1339 takes more than a minute to start the splash screen. With Eu 3.1.1 it takes 4 to 5 seconds. Using my Task-Manager I can see that using Eu 4.0 IDE consumes about 296 MB(!) of RAM. With Eu 3.1.1 it will use about 22 MB.
As an exercise I made a copy of IDE.exw and replaced the include statements of IDE with the real code of the include files, changing the names of redundant symbols and routines or commenting out some of them (like ok, void etc.)
As a result I got a relative identical big IDE in one file which starts as fast as with Eu 3.1.1. (4 to 5 seconds) It uses 20.6 MB of RAM with Eu 4.0 and 21.2 MB with Eu 3.1.1.
There is a similar behaviour with wrapper.e of the EuGtk2 files. With Eu 3.1.1 it will use 4 MB of RAM, with Eu 4.0 rev 1339 it will consume 179.6 MB.
I do not understand why it makes such a difference if I use global symbols with Eu 4.0 and why it will consume so much memory when creating bigger sized applications, no matter if I see them as forward references or not. As the code of Eu 4.0 is far beyond my capabilities I cannot make any suggestion. But is it not possible to treat global symbols in include files without public or export the same way as with Eu 3.1.1?
Roland
5. Re: 4.0 and include statements
- Posted by bernie Jan 18, 2009
- 968 views
In 4.0, when it parses a file, and it resolves a symbol to a global symbol in another file that wasn't included by the current file, it treats this as a forward reference, since there could be a matching symbol somewhere else either in the same file, or in another file that was included, and we don't want the errant global symbol to override the symbol the programmer actually wanted.
So when the IDE is parsed, over the entire course, it ends up with over 60,000 unique forward references (this includes things like type checks). Which really, really slows down the parsing. I went through and added include ide.exw to the files, and the parsing time was drastically reduced. Note, it would have been better to have included the specific files that were used, but this was easy, and I just wanted to see the difference.
Matt:
I still think that it would be a good idea to have some way
for a user to turn off the forward reference parsing for a
include file just like we can turn off type checking.
All the parsing is eating up start up time.
There should be no need to parse forward references in the
standard include files. I thought that was Jeremy's purpose
of having STANDARD include files that did not change.
Bernie
6. Re: 4.0 and include statements
- Posted by mattlewis (admin) Jan 20, 2009
- 887 views
I do not understand why it makes such a difference if I use global symbols with Eu 4.0 and why it will consume so much memory when creating bigger sized applications, no matter if I see them as forward references or not. As the code of Eu 4.0 is far beyond my capabilities I cannot make any suggestion. But is it not possible to treat global symbols in include files without public or export the same way as with Eu 3.1.1?
The issue isn't really the scope of the symbols. You can still use global. The difference is whether files that use the symbols include the file that defined the symbol (or included a file that included the definition, etc). Basically, it's a sloppy programming practice that many (including me!) have indulged, and the new forward referencing (which has many tangible benefits) requires that we wait to resolve un-included global references until later.
The memory bloat is the parser remembering all of the places where this occurred. It's probably possible to improve this (and I'm fairly sure it will happen at some point), but the best solution (which will incur some cost in the transition) is to put in the missing include statements.
Matt
7. Re: 4.0 and include statements
- Posted by mattlewis (admin) Jan 20, 2009
- 918 views
I still think that it would be a good idea to have some way
for a user to turn off the forward reference parsing for a
include file just like we can turn off type checking.
All the parsing is eating up start up time.
There should be no need to parse forward references in the
standard include files. I thought that was Jeremy's purpose
of having STANDARD include files that did not change.
I don't think this has ever been a goal of the standard library. And I don't follow your claim about the need (or lack thereof) for parsing forward references. There is more than the static code to consider. In the old way of doing things, it was possible for the order in which files were included to cause the code to break. Different parts of the standard library use each other, which would have caused some of this instability. But the forward referencing allows the code to be much simpler, and to avoid using routine ids, and other work arounds.
It's certainly technically possible to do this, but I'm not so sure that it's a good idea. You could have some really strange, and difficult to debug issuesone instance of which was the cause of the change in behavior.
Matt
