OpenEuphoria

1. ver 4.0 problem with SVN1222

• Posted by bernie Oct 28, 2008
• 1070 views

Matt:
I have a win32 library code that I am working on.
I have demo programs that are using this library.
When I run a demo program using SVN 1222 the parsing time is something
like a 1.5 MINUTES then the program runs.
This was also true when I ran SVN 1217 but of course it was failing after
the same long parse time.
I ran the same demo with SVN 1205 and SVN 1212 and the parse time is down
to about 12 SECONDS.
So something has changed between SVN 1212 and SVN 1222.

3. Re: ver 4.0 problem with SVN1222

• Posted by robcraig Oct 28, 2008
• 1093 views

Also, on an old machine (e.g. Windows 95/98) you should be careful to remove, or comment out, all "with trace", "with profile" etc. statements, anywhere in the source. These statements cause the source code to be retained by the parser, and surprisingly, that can often (with small RAM machines) cause the parse time to go up quite a lot, when parsing large programs.

Rob

5. Re: ver 4.0 problem with SVN1222

• Posted by bernie Oct 29, 2008
• 1045 views

I don't understand what any of this has to do with running the SAME CODE
with SVN 1212 and SVN 1222.

The start of time for SVN 1212 is taking 12 SECONDS
and SVN 1222 taking 1.5 MINUTES.

7. Re: ver 4.0 problem with SVN1222

• Posted by mattlewis (admin) Oct 29, 2008
• 1026 views

Bernie,

I brought it up because r1212 and r1222 are treating the SAME CODE differently when you use globals without including them. If this is what you are doing (and you haven't answered that yet), have you added "missing" include statements yet? Please let us know how that changes the outcome, because there's nothing else that I can do for you until you either try that, or give me some other information about what you're doing. But those were the main changes to the code in those ranges of revisions.

Matt

9. Re: ver 4.0 problem with SVN1222

• Posted by mattlewis (admin) Oct 29, 2008
• 1012 views

The std library should be fine. I don't think that the 4.0 std library even uses any globals. Everything should be either public or export. Here is the issue [deep breath]:

In pre-4.0 euphoria (maybe pre-3.1, but anyway...), if you tried to use a global without a namespace qualifier, and there was more than one global with that name, you would get an error. In 4.0, the parser tries to be smarter, and to figure out which symbol you really meant to use. It does this by keeping track of which files included which other files.

One way that the earlier behavior could really cause problems was with 3rd party libraries. Imagine that two 3rd party libraries defined a global with the same name. It would be perfectly safe, in your application, to use either library. But if you use both, you might start getting errors in one of those libraries (assuming the author didn't use namespace qualifiers when referencing those globals that he and the other author both defined). Therefore, the only way to get the two libraries to work together would be to go in and edit their code. Note that this could happen, even if your application always used namespace qualifiers.

In 4.0, however, the parser will consider where the symbol is being used and which files were included by the file with the code that references the global. It will choose the global symbol that was included by the third party library over a global declared by code that the file "doesn't know about." Now, the two third party libraries Just Work when you put them together. They operate independently, and don't cause conflicts with each other.

There are two new terms in 4.0: direct include and indirect include. A direct include is when a file has an include statement for a file. So if the line "include bar.e" appears in foo.e, we would say that foo.e directly includes bar.e. Suppose now that bar.e directly includes baz.e. Then we could also say that foo.e indirectly includes baz.e. Also suppose that bat.e is not included by any of these files, directly or indirectly, and that baz.e and bat.e both define a global named X. If, somewhere in foo.e, the code refers to X, the parser will see that there is a connection between foo.e and baz.e, but not bat.e, and use baz.e:X in preference to bat.e:X. And baz.e:X is an included global for foo.e, while bat.e:X is an unincluded global. A common way for this to occur in code is to have the main file include some library like win32lib, and to have other parts of the app included by the main file rely on the main file to include win32lib. Now you've got lots of unincluded global references going on.

Let's complicate things some more. :) We've removed the restriction on forward references. This means that two files can now safely include each other, and use each other's routines and constants and variables [of appropriate scope, of course]. One instance where this happens is between sort.e, sequence.e and text.e. There is code in sequence.e that uses sort(). The order in which files are included affects the order in which their code is actually parsed. This caused lots of headaches if you tried to have mutually dependent files in pre-4.0 euphoria, and lead to use of routine_ids and moving routines before or after include statements, etc. But now that's not necessary.

Now, let's suppose you're working on a complex app that actually happens to include the old sort.e, perhaps because you're porting it from 3.1 to 4.0 (there are other ways that a similar scenario could occur, but this one actually did). And suppose that the old sort.e was included before the new one was. It just so happens that the new sort.e has an extra (optional) parameter, meaning that the two are somewhat incompatible. So if sequence.e is parsed before the new sort.e, the call to sort() gets resolved to the old sort, which is an error, because it tries to pass two parameters to a function that only takes one.

But now we've encountered a situation in which code is being affected (incorrectly!) by code that it knows nothing about. So now, instead of resolving to the unincluded global immediately, it defers doing so until the end, to see if there were other, better choices. If you're using a lot of unincluded globals by design, there will be no better choices, but it will cause a lot of overhead as the parser continues to look for them. At the end of the parsing, it will clean them all up, and the program should run as before.

Each file is only scanned once, just like it's always been. Each time the scanner finishes with a file, it attempts to resolve forward references. In theory, we could wait until the end of the program, but the problem is that local symbols are hidden, so we need to check before we can't see them. One probable future optimization is to make this a bit smarter, and only check those that might refer to local symbols in that file. But it's partly a case of trying to get things correct before optimizing.

Basically, the parser remembers all of the unresolved forward references and periodically tries to resolve them.

Matt

11. Re: ver 4.0 problem with SVN1222

• Posted by mattlewis (admin) Oct 29, 2008
• 1019 views

r1224 does a better job at not trying to resolve things that can't be resolved at intermediate states. I went from parsing Judith's ide in ~30s to ~6s.

Matt

13. Re: ver 4.0 problem with SVN1222

• Posted by bernie Oct 31, 2008
• 1012 views

Matt:
I got my code down to loading and parsing in about 10 seconds.
SVN 1223 didn't make any difference so I started stripping out
and changing code.
I finally discovered that the problem was caused by the way I was
using my includes so I changed many globals to public and redesigned
my include scheme which brought the speed into a livable amount.

I have a suggestion about improving fwd-reference.
Why not have 'with fwd-ref' and 'without fwd-ref' keywords ?
In that way if a user is not using any fwd-ref in a file
they would put the fwd-ref keyword in the file and the fwd-ref scan
would skip that file.

This would increase performance and aid a user in isolating
where fwd-ref scan is spending a lot of time.

Thanks for all your patience
Bernie