OpenEuphoria

1. Packages

• Posted by CChris <christian.cuvier at ?griculture.g?uv.fr> Aug 06, 2007
• 579 views

I just posted at http://oedoc.free.fr/Packages.htm a description for a packaging
system for Euphoria.

It will solve the name clash issues definitively and allow to write cleaner code
by breaking large libraries into smaller subsets. The system was designed so that
not a single piece of code will break. Performancewise, the only place where it
may be noticeable - I didn't check yet - is the processing of an include
statement, hardly a speed bottleneck - the time to open the file and fetch
physical data from hard disk dominates any software processing.

Since I simplified a couple things from my 2.5 eu.ex implementation, more tests
are needed before creating a new development branch with the new system. I hope
to get this done during this month - writing docs in the .htx format explains the
loose timetable.

CChris

3. Re: Packages

• Posted by CChris <christian.cuvier at ?gricul?ure.gouv.fr> Aug 07, 2007
• 560 views

jacques deschênes wrote:
> 
> Your system is too complex, it add a lot of new keywords. Why make it complex
> when one can make simple.

Because simple systems won't work.
They will create issues of their own, out of short-sightedness.

> A few days ago I propose a namespace system similar to what C# use.
> My proprosed system needs only 3 new keywords.
> 
> ref.: <a
> href="http://www.openeuphoria.org/cgi-bin/esearch.exu?fromMonth=6&fromYear=1&toMonth=8&toYear=C&postedBy=Jacques+Desch%EAnes&keywords=namespace">http://www.openeuphoria.org/cgi-bin/esearch.exu?fromMonth=6&fromYear=1&toMonth=8&toYear=C&postedBy=Jacques+Desch%EAnes&keywords=namespace</a>
> 
> In my first post I didn't said that the same namespace identifier could be
> used
> in more than one file as in C#.
> Also I proposed a 2 keywords namespace system, now I add a third one: 'alias'
> 
> alias  namepace1:namespace2:...:namespacen  as alias_name
> this will enable a shorter typing alias_name for long path.
> 

Mine has less typing, and requires less namespaces, so it is simpler. Yours does
not address the problem of existing code which doesn't use namspaces at all.

> tree structure like this are simple system used in many place in information
> system, form exemple: file system, 
> snmp protocol, wbem protocol to name a few.
> 
> regards,
> Jacques Deschênes
> 
> 
> CChris wrote:
> > 
> > 
> > I just posted at <a
> > href="http://oedoc.free.fr/Packages.htm">http://oedoc.free.fr/Packages.htm</a>
> a description</font></i>
> > for a packaging system for Euphoria.
> > 
> > It will solve the name clash issues definitively and allow to write cleaner
> > code by breaking large libraries into smaller subsets. The system was
> > designed
> > so that not a single piece of code will break. Performancewise, the only
> > place
> > where it may be noticeable - I didn't check yet - is the processing of an
> > include
> > statement, hardly a speed bottleneck - the time to open the file and fetch
> > physical
> > data from hard disk dominates any software processing.
> > 
> > Since I simplified a couple things from my 2.5 eu.ex implementation, more
> > tests
> > are needed before creating a new development branch with the new system. I
> > hope
> > to get this done during this month - writing docs in the .htx format
> > explains
> > the loose timetable.
> > 
> > CChris

5. Re: Packages

• Posted by CChris <christian.cuvier at agri?ultu?e.gouv.fr> Aug 07, 2007
• 551 views

jacques deschênes wrote:
> 
> 
> CChris wrote:
> > Because simple systems won't work.
> > They will create issues of their own, out of short-sightedness.
> 
> oh! really, what are the issues with C# namespace system? which is essentially
> what I propose.
> 
> 
> > Mine has less typing, and requires less namespaces, so it is simpler. Yours
> > does not address the problem of existing code which doesn't use namspaces at
> > all. 
> 
> No at all the namespace system I propose, won't brake any legacy code,
> namespace
> qualifier could be only use as needed.
> And the original namespace system will stay valid.
> 

I was not implying that legacy code would break, but that it would keep causing
trouble because of its uncontained identifiers.
In my system, current namespacing is valid and semantically untouched. The
current namespacing works well for multiple file applications, but more poorly
for multiple file libraries. That's why it needs being complemented - not
replaced.


> incluce  libxx.e as xx  -- would stay valid
> 
> or if my proposol is implemented
> 
> include  libxx.e [alias xx] -- no need for an alias here but accepted.
> 
> the if there is a name clash
> [somename:]fctX()
> here "somename" is namespace identifier defined at top level in libxx.e and
> one
> use it only there is a name clash as it now implemented in euphoria.

So you need to edit existing code to make it behave? That's exactly what I
strive to avoid, for obvious reasons explained in the paper.

> But with this system a namespace is not limited to a single file, It applies
> to any global identifier defined inside:
> 
> namespace somename
>   ...
> end namespace
>  
> 

What happens when namespaces are nested? Since this is internal to the library
and may change without notice, the user shouldn't have to bother and use
compound, possibly changing, namespaces at all - he shouldn't have to use any,
for that matter. For something you may forget - SIMPLICITY OF USE OF THE LIBRARY.

> regards,
> Jacques Deschênes

Additionally, this scheme doesn't appear to address the poroblem of namespace
pollution. Once a library defines a symbol, even if namespaced, you won't be able
to define one of your own and perform an unqualified call. Forcing to use a
namespace on the user is clumsy, and is problematic since namespace identifiers
are not inherited by inclusion.
Extension of global routines, which implies redefining them, doesn't work
either.
Unless you have written something elaborate enough to check for the tricky
details, which are galore.

CChris

7. Re: Packages

• Posted by Matt Lewis <matthewwalkerlewis at gm?i?.com> Aug 07, 2007
• 535 views

CChris wrote:
> 
> 
> I just posted at <a
> href="http://oedoc.free.fr/Packages.htm">http://oedoc.free.fr/Packages.htm</a> a
> description
> for a packaging system for Euphoria.
> 
I'm not sure we need to go as far as this.  It does complicate the language,
and I'm not sure that there isn't a better way.  I've resurrected my 
previous proposal for namespace resolution, and put it up on sourceforge.

The benefit is that it will solve the issue of 3rd party name conflicts
with zero change to existing code, and it does it in a very intuitive and
obvious way.

https://sourceforge.net/tracker/?func=detail&atid=902785&aid=1769335&group_id=182827

Here is the text:

Summary: Improved namespace conflict resolution

Initial Comment:
The current namespace feature cannot solve the problem of conflicting 
global symbols in multiple third party libraries.  The proposal is to 
track the files included within each file, and use that knowledge to 
resolve ambiguous symbols.

The proposed algorithm:

If you try to reference a global symbol without  namespaces, and the 
symbol exists in more than one file, it will attempt to resolve the 
identity of the symbol based on the 'include paths' of the symbols.  
If there is one [and only one] global symbol in the include path, then 
that symbol is used.  An include path is defined as files either 
included directly by a file, or indirectly included by files included 
by the file.

Assume:

-- alice.e
global sequence name
name = "Alice"

-- bob.e
include alice.e
global integer x
x = 1
global procedure bob_name()
	printf(1,"Bob says name = \"%s\"\n",{name}) end procedure

-- chris.e
global sequence name
? x  -- ok, this is dumb, but shows that normal
     -- Eu resolution is still in effect name = "Chris"

-- diane.e
include chris.e
global procedure diane_name()
	printf(1,"Diane says name = \"%s\"\n",{name}) end procedure

-- erin.ex
include bob.e
include diane.e
include alice.e as a
include chris.e as c
bob_name()
diane_name()
printf(1,"a:name = \"%s\" c:name = \"%s\"\n",{a:name,c:name})


Running erin.ex should give:
  Bob says name = "Alice"
  Diane says name = "Chris"
  a:name = "Alice" c:name = "Chris"

9. Re: Packages

• Posted by CChris <christian.cuvier at agriculture??ouv.fr> Aug 07, 2007
• 535 views

Jason Gade wrote:
> 
> CChris wrote:
> > 
> > jacques deschênes wrote:
> > > 
> > > Your system is too complex, it add a lot of new keywords. Why make it
> > > complex
> > > when one can make simple.
> > 
> > Because simple systems won't work.
> > They will create issues of their own, out of short-sightedness.
> 
> Surely you're joking?
> 
> And in a different thread:
> 
> > We seem to live on different planets.
> 
> True.
> 

I agree. But some aspect of your planet are causing serious trouble to my
coding.

> Here's my proposal, one new keyword, but I don't know if it is possible:
> import <lib> [as <namespace>]
> 
> This works just like include but globals included or imported by imported
> files
> are no longer visible. It acts as a block to global namespace pollution.
> 
> Basically it only imports symbols defined in <lib>, not symbols defined in
> <lib>'s
> includes or imported files. Those go... somewhere else.
> 

in my_app.exw:
import my_lib.e

in my_lib.e
include get.e

Can my_app.exw access routines in get.e?

If so:

in my_lib.e:
global function my_func()
import my_utils.e

in my_utils.e:
x=my_func()

1/ Is the above call valid?
2/ What happens if my_app.exw also defines a global my_func(), and some other
file calls that global?

Surely you're joking.

CChris

> I'm still studying to the code to figure out if it is possible and/or how to
> do it. I've got a couple of ideas that I should write down, but I'll figure
> it out. If it's possible.
> 
> It shouldn't break any code that doesn't use "import" as a variable name.
> 
> --
> "Any programming problem can be solved by adding a level of indirection."
> --anonymous
> "Any performance problem can be solved by removing a level of indirection."
> --M. Haertel
> "Premature optimization is the root of all evil in programming."
> --C.A.R. Hoare
> j.

11. Re: Packages

• Posted by Jason Gade <jaygade at yahoo.?om> Aug 07, 2007
• 532 views

CChris wrote:
> 
> Jason Gade wrote:
> > 
> > CChris wrote:
> > > 
> > > jacques deschênes wrote:
> > > > 
> > > > Your system is too complex, it add a lot of new keywords. Why make it
> > > > complex
> > > > when one can make simple.
> > > 
> > > Because simple systems won't work.
> > > They will create issues of their own, out of short-sightedness.
> > 
> > Surely you're joking?
> > 
> > And in a different thread:
> > 
> > > We seem to live on different planets.
> > 
> > True.
> > 
> 
> I agree. But some aspect of your planet are causing serious trouble to my
> coding.
> 
> > Here's my proposal, one new keyword, but I don't know if it is possible:
> > import <lib> [as <namespace>]
> > 
> > This works just like include but globals included or imported by imported
> > files
> > are no longer visible. It acts as a block to global namespace pollution.
> > 
> > Basically it only imports symbols defined in <lib>, not symbols defined in
> > <lib>'s
> > includes or imported files. Those go... somewhere else.
> > 
> 
> in my_app.exw:
> import my_lib.e
> 
> in my_lib.e
> include get.e
> 
> Can my_app.exw access routines in get.e?
> 
> If so:
> 
> in my_lib.e:
> global function my_func()
> import my_utils.e
> 
> in my_utils.e:
> x=my_func()
> 
> 1/ Is the above call valid?

On further consideration this should break. Include should treat files as text,
import should treat files as modules and work one way.

> 2/ What happens if my_app.exw also defines a global my_func(), and some other
> file calls that global?

It depends if the other files are linked with include or import.

--
"Any programming problem can be solved by adding a level of indirection."
--anonymous
"Any performance problem can be solved by removing a level of indirection."
--M. Haertel
"Premature optimization is the root of all evil in programming."
--C.A.R. Hoare
j.

13. Re: Packages

• Posted by CChris <christian.cuvier at ag?icultur?.gouv.fr> Aug 07, 2007
• 533 views

Jason Gade wrote:
> 
> CChris wrote:
> > 
> > Jason Gade wrote:
> > > 
> > > CChris wrote:
> > > > 
> > > > jacques deschênes wrote:
> > > > > 
> > > > > Your system is too complex, it add a lot of new keywords. Why make it
> > > > > complex
> > > > > when one can make simple.
> > > > 
> > > > Because simple systems won't work.
> > > > They will create issues of their own, out of short-sightedness.
> > > 
> > > Surely you're joking?
> > > 
> > > And in a different thread:
> > > 
> > > > We seem to live on different planets.
> > > 
> > > True.
> > > 
> > 
> > I agree. But some aspect of your planet are causing serious trouble to my
> > coding.
> > 
> > > Here's my proposal, one new keyword, but I don't know if it is possible:
> > > import <lib> [as <namespace>]
> > > 
> > > This works just like include but globals included or imported by imported
> > > files
> > > are no longer visible. It acts as a block to global namespace pollution.
> > > 
> > > Basically it only imports symbols defined in <lib>, not symbols defined in
> > > <lib>'s
> > > includes or imported files. Those go... somewhere else.
> > > 
> > 
> > in my_app.exw:
> > import my_lib.e
> > 
> > in my_lib.e
> > include get.e
> > 
> > Can my_app.exw access routines in get.e?
> > 
> > If so:
> > 
> > in my_lib.e:
> > global function my_func()
> > import my_utils.e
> > 
> > in my_utils.e:
> > x=my_func()
> > 
> > 1/ Is the above call valid?
> 
> On further consideration this should break. Include should treat files as
> text,
> import should treat files as modules and work one way.
> 

It shouldn't. Submodules need - most of the time - to call some central routine
in the file that includes them. It really would be a terribly useless, harmful
regression if this started to no longer work.

> > 2/ What happens if my_app.exw also defines a global my_func(), and some
> > other
> > file calls that global?
> 
> It depends if the other files are linked with include or import.
> 

Oh, how is this related to goto? And who is Shirley?

Don't you think your approach will make placement of routines even more a
nightmare than it currently is? Not only you have to deal with define_before_use,
but now you'd add define_before_import or somesuch. Cant believe it.

CChris

> --
> "Any programming problem can be solved by adding a level of indirection."
> --anonymous
> "Any performance problem can be solved by removing a level of indirection."
> --M. Haertel
> "Premature optimization is the root of all evil in programming."
> --C.A.R. Hoare
> j.

15. Re: Packages

• Posted by Matt Lewis <matthewwalkerlewis at g?ail.?om> Aug 07, 2007
• 541 views

Jason Gade wrote:
> 
> Hey, Matt. I haven't tested it but does it handle the safe.e issue mentioned
> by Bob Elia?
>
> http://www.openeuphoria.org/cgi-bin/esearch.exu?thread=1&fromMonth=7&fromYear=C&toMonth=9&toYear=C&keywords=%22include+path+problem+</a>[WAS:+STANDARD+EUPHORIA+LIBRARYS]%22
> 
> I can test it later if you don't have time.

No, his issue is based on the way include paths are searched now.  I think 
the answer for him is to copy all of his includes that use safe.e when he
copies that to his local path and renames it machine.e.

Matt

17. Re: Packages

• Posted by CChris <christian.cuvier at agricult?re.go?v.fr> Aug 07, 2007
• 530 views

Jason Gade wrote:
> 
> CChris wrote:
> > 
> > Jason Gade wrote:
> > > 
> > > CChris wrote:
> > > > 
> > > > Jason Gade wrote:
> > > > > 
> > > > > CChris wrote:
> > > > > > 
> > > > > > jacques deschênes wrote:
> > > > > > > 
> > > > > > > Your system is too complex, it add a lot of new keywords. Why make
> > > > > > > it
> > complex</font></i>
> > > > > > > when one can make simple.
> > > > > > 
> > > > > > Because simple systems won't work.
> > > > > > They will create issues of their own, out of short-sightedness.
> > > > > 
> > > > > Surely you're joking?
> > > > > 
> > > > > And in a different thread:
> > > > > 
> > > > > > We seem to live on different planets.
> > > > > 
> > > > > True.
> > > > > 
> > > > 
> > > > I agree. But some aspect of your planet are causing serious trouble to
> > > > my
> coding.</font></i>
> > > > 
> > > > > Here's my proposal, one new keyword, but I don't know if it is
> > > > > possible:
> > > > > import <lib> [as <namespace>]
> > > > > 
> > > > > This works just like include but globals included or imported by
> > > > > imported
> files</font></i>
> > > > > are no longer visible. It acts as a block to global namespace
> > > > > pollution.
> > > > > 
> > > > > Basically it only imports symbols defined in <lib>, not symbols
> > > > > defined in
> <lib>'s</font></i>
> > > > > includes or imported files. Those go... somewhere else.
> > > > > 
> > > > 
> > > > in my_app.exw:
> > > > import my_lib.e
> > > > 
> > > > in my_lib.e
> > > > include get.e
> > > > 
> > > > Can my_app.exw access routines in get.e?
> > > > 
> > > > If so:
> > > > 
> > > > in my_lib.e:
> > > > global function my_func()
> > > > import my_utils.e
> > > > 
> > > > in my_utils.e:
> > > > x=my_func()
> > > > 
> > > > 1/ Is the above call valid?
> > > 
> > > On further consideration this should break. Include should treat files as
> > > text,
> > > import should treat files as modules and work one way.
> > > 
> > 
> > It shouldn't. Submodules need - most of the time - to call some central
> > routine
> > in the file that includes them. It really would be a terribly useless,
> > harmful
> > regression if this started to no longer work.
> 
> Then use include. Don't use import. Isn't that what you say to others when you
> propose a change? Don't use it if you don't like it?
> 

Of course I can. Problems arise when others start using import, because the way
"you" include files has consequences on how "my" globals are seen. goto is pretty
harmless in regard, because, if others use it, it won't impact the code I write.
The quotes are meant to show that use of you or my hasn't anything personal
attached to it.

> > 
> > > > 2/ What happens if my_app.exw also defines a global my_func(), and some
> > > > other
> > > > file calls that global?
> > > 
> > > It depends if the other files are linked with include or import.
> > > 
> > 
> > Oh, how is this related to goto? And who is Shirley?
> 
> Because if this isn't spaghetti code then I don't know what is.
> 
> > 
> > Don't you think your approach will make placement of routines even more a
> > nightmare
> > than it currently is? Not only you have to deal with define_before_use, but
> > now you'd add define_before_import or somesuch. Cant believe it.
> > 
> > CChris
> No idea what this means.
> 

When you use several files in a library, you tend to put routines into them
according to some loose theme classification. That's why you create several
files, right?
The define_before_use rule prevents to do this sometimes, and leads to pretty
unnatural routine placements and a large amount of time figuring out where
exactly in the main file to put some include statement.
In the proposal which you have stated so far (particularly in question 1/), you
are adding extra constraints on this already delicate issue. This is neither
simple nor elegant, but torture.

In contrast, my solution provides more freedom in this respect, not less.
CChris

> --
> "Any programming problem can be solved by adding a level of indirection."
> --anonymous
> "Any performance problem can be solved by removing a level of indirection."
> --M. Haertel
> "Premature optimization is the root of all evil in programming."
> --C.A.R. Hoare
> j.

19. Re: Packages

• Posted by Jason Gade <jaygade at ya?oo.c?m> Aug 07, 2007
• 548 views

CChris wrote:
> 
> Jason Gade wrote:
> > Then use include. Don't use import. Isn't that what you say to others when
> > you
> > propose a change? Don't use it if you don't like it?
> > 
> 
> Of course I can. Problems arise when others start using import, because the
> way "you" include files has consequences on how "my" globals are seen. goto
> is pretty harmless in regard, because, if others use it, it won't impact the
> code I write.

You'll have to give me an example, unless code you write is affected by
implementation details of another file. Which it shouldn't be.

> The quotes are meant to show that use of you or my hasn't anything personal
> attached to it.

Yeah, I'm trying to keep the personal out of it too. I've started writing
several snarky comments and fortunately I've had second thoughts and pulled back
in time.

It isn't that I don't respect you personally -- we just disagree on the "way
things should be". And I realize this.

> > No idea what this means.
> > 
> 
> When you use several files in a library, you tend to put routines into them
> according to some loose theme classification. That's why you create several
> files, right?
> The define_before_use rule prevents to do this sometimes, and leads to pretty
> unnatural routine placements and a large amount of time figuring out where
> exactly
> in the main file to put some include statement.
> In the proposal which you have stated so far (particularly in question 1/),
> you are adding extra constraints on this already delicate issue. This is
> neither
> simple nor elegant, but torture.
> 
> In contrast, my solution provides more freedom in this respect, not less.
> CChris

I still need an example. If I was writing library code then each file would have
a pretty tight theme, not a loose one. And I would try to have the various files
in the library very loosely coupled if at all.

The way things are right now, libraries get mashed into one big file because of
the very problems we are discussing.

--
"Any programming problem can be solved by adding a level of indirection."
--anonymous
"Any performance problem can be solved by removing a level of indirection."
--M. Haertel
"Premature optimization is the root of all evil in programming."
--C.A.R. Hoare
j.

21. Re: Packages

• Posted by CChris <christian.cuvier at ?griculture.gouv?fr> Aug 07, 2007
• 574 views

Matt Lewis wrote:
> 
> CChris wrote:
> > 
> > 
> > This had been discussed at length in the OpenEu group as well as on EuForum.
> > This algorithm may be good as long as the shape of the include tree doesn't
> > change. If it does, strange bugs may arise. Look to Irv Mullins' posts about
> > this issue, in 2004 I believe.
> 
> This doesn't address the issue of paths and include files.  It's a completely
> different problem.

Not sure;  I'd have to see a more detailed proposal to tell.

> 
> > If the main source file defines a global symbol that conflicts with one in
> > the
> > libs, and another file tries to use this symbol, you'll still have a ame
> > clash.
> 
> This is too vague for me to interpret, so I'll assume that you mean that there
> are two, conflicting symbols in the legitimate "include path" of a file.  
> Yes, those symbols will conflict.  This is the point of namespaces.
> 

No, unless you count the main app file as part of the tree, which is true
somehow.
Application A includes a library L which has an internal global symbol. A
doesn't need to know about it, and the symbol is undocumented anyway since it is
not intended for public use. Yet, if A tries to define a global symbol of the
same name, then:
* calls from inside A.e will work, since this symbol will have highest priority
under current rules.
* calls from a file included by A.e, but unrelated to L, will fail.

And, since the symbol is internal to L, there shouldn't be any need for any
namespace qualifier. The symbol may pop in or out from one version of L to the
next.

> The problem is when, say, Jason uses one of my libraries, and he uses one of
> yours.  Individually, they're both fine, but once he includes both, it 
> breaks.  *His* code should have to use namespaces.  That is the whole point.
> The problem that this solves is that neither my library nor your library
> will break because they're both being used.  The interpreter will be able
> to tell which symbol each library really wants, because presumably, neither
> library includes the other.
> 
> > This solution, like import, which had been discussed a lot with different
> > names
> > like local_include, only addresses part of the larger picture.  
> 
> Yes, but I believe that it solves [at least] 95% of the larger picture, 
> and does so in a transparent way that requires no changes to any existing 
> code.

Mine doesn't require any - apart from the convenience exception for the standard
include files; but, as I emphaised in the paper, even that may be avoided.

What is the point of leaving 5% or 1% of the problem purposely unsolved? This I
fail to understand.

> 
> > The supposed added complexity - two top level statements, two options to
> > with/without
> > and two keywords, to be precise - will concern very few people, namely the
> > writers
> > of multiple file libraries, who are the most exposed to the problems.
> > Otherwise,
> > exposure will be nonzero, but pretty minimal. Read the "Who is concerned..."
> > section of the paper. At least the whole problem would be addressed, not
> > just
> > some largish square in the middle of the picture.
> 
> I don't completely agree with you.  As I said, I think that you can solve the
> vast majority of the problem without adding any keywords or syntax.  The
> difference between global and sort of global is just asking for bugs
> as everyone adjusts.

No, it _eliminates_ some bugs, since unexpected assignments are blocked.
And stating explicitly whether an interfile symbol is for public use or not
helps maintaining the code, doesn't it?

> 
> As one of the people concerned, I'm against it.
> 

Too bad.

CChris
> Matt

23. Re: Packages

• Posted by Pete Lomax <petelomax at blu?yo?der.co.uk> Aug 08, 2007
• 536 views

Matt Lewis wrote:
> I've resurrected my previous proposal for namespace resolution

I tend to agree that a very high percentage of the problems CChris is trying to
solve are likely to simply vapourise once your proposal is finished.

One addition:

include win32lib.ew as w32
...
   w32:getOpenFileName()

Currently this fails because getOpenFileName() is defined in w32file.ew.
Although this is not the same a resolving duplicate globals, the namespace
concept should be extended to cover the include subtree, and if it could be done
at the same time... 

FWIW, I would not be against a simple "ringfence" such as

include x.e [exports x,y,z] [as namespace]

ie, limiting "global leakage" to rqd api, but really your proposal needs to be
fully implemented first, BEFORE any further extensions are devised.

Regards,
Pete

25. Re: Packages

• Posted by Pete Lomax <petelomax at bluey?n?er.co.uk> Aug 08, 2007
• 538 views

Jason Gade wrote:
> 
> I was thinking about this. Couldn't your application just use "include 
> w32file.ew as file" ?

Erm yes, but then again here is another of my bugbears:

include w32\win32lib.ew
include w32file.ew as w32f

Assuming that w32\ (which contains the 20+ files that make up win32lib) is not
in EUINC, on 3.1 the above fails:
The include w32file.ew statement inside win32lib.ew works fine but the one in
the main file terminates rather than accept you meant the w32file.ew already
included when it can (no longer) find it.

Admittedly win32lib is a poor example to demonstrate this problem, since almost
everyone with it installed has it in EUINC, but this new behaviour has hampered
my efforts to tidy up project directory structures, and it did sorta work on 2.4.

Regards,
Pete

27. Re: Packages

• Posted by Matt Lewis <matthewwalkerlewis at gm?il.co?> Aug 08, 2007
• 514 views

Pete Lomax wrote:
> 
> Jason Gade wrote:
> > 
> > I was thinking about this. Couldn't your application just use "include 
> > w32file.ew as file" ?
> 
> Erm yes, but then again here is another of my bugbears:
> }}}
<eucode>
> include w32\win32lib.ew
> include w32file.ew as w32f
> </eucode>
{{{

> Assuming that w32\ (which contains the 20+ files that make up win32lib) is not
> in EUINC, on 3.1 the above fails:
> The include w32file.ew statement inside win32lib.ew works fine but the one in
> the main file terminates rather than accept you meant the w32file.ew already
> included when it can (no longer) find it.
> 
> Admittedly win32lib is a poor example to demonstrate this problem, since
> almost
> everyone with it installed has it in EUINC, but this new behaviour has
> hampered
> my efforts to tidy up project directory structures, and it did sorta work on
> 2.4.

I've got all this working in the front end.  The next step is to alter
routine_id in the back end and in the translator to include and utilize
the new information.  That's not quite as easy...especially the 
translator.  But not impossible.

Matt

29. Re: Packages

• Posted by CChris <christian.cuvier at agric?lture.gouv.f?> Aug 08, 2007
• 515 views

Jason Gade wrote:
> 
> Matt Lewis wrote:
> > 
> > Pete Lomax wrote:
> > > 
> > > Jason Gade wrote:
> > > > 
> > > > I was thinking about this. Couldn't your application just use "include 
> > > > w32file.ew as file" ?
> > > 
> > > Erm yes, but then again here is another of my bugbears:
> > > }}}
<eucode>
> > > include w32\win32lib.ew
> > > include w32file.ew as w32f
> > > </eucode>
{{{

> > > Assuming that w32\ (which contains the 20+ files that make up win32lib) is
> > > not
> > > in EUINC, on 3.1 the above fails:
> > > The include w32file.ew statement inside win32lib.ew works fine but the one
> > > in
> > > the main file terminates rather than accept you meant the w32file.ew
> > > already
> > > included when it can (no longer) find it.
> > > 
> > > Admittedly win32lib is a poor example to demonstrate this problem, since
> > > almost
> > > everyone with it installed has it in EUINC, but this new behaviour has
> > > hampered
> > > my efforts to tidy up project directory structures, and it did sorta work
> > > on
> > > 2.4.
> > 
> > I've got all this working in the front end.  The next step is to alter
> > routine_id in the back end and in the translator to include and utilize
> > the new information.  That's not quite as easy...especially the 
> > translator.  But not impossible.
> > 
> > Matt
> 
> Are you going to make a feature-testing branch for this?
> 
> [off-topic]
> Heh. I started reading about svn a couple of days ago and I think I like it
> very much... the possibilities... I remember reading a Linux article a year
> or two ago about using svn to manage your home directory.
> 
> Now if I just had a firm grasp on this branch/tags/trunk think. I think I get
> it, but...
> 
> [on-topic]
> We'll have to think of some interesting regressions with which to test it. The
> first of which is alice-bob-chris-diane of course.
> 
> --
> "Any programming problem can be solved by adding a level of indirection."
> --anonymous
> "Any performance problem can be solved by removing a level of indirection."
> --M. Haertel
> "Premature optimization is the root of all evil in programming."
> --C.A.R. Hoare
> j.

But certainly not the last:
1/ will full duplex client-server relationship keep working between a main
library file and a file it (recursively) includes? It should, otherwise code
breaking may not be marginal.
2/ If a global symbol is privately used in a library, yet is global because it
is referred to in several files, will the main application keep seeing it? It
shouldn't; This is one of the problems to solve.
3/ If case 2/ is solved, and the application includes some other file after the
main library file considered in 1/, will that file see the symbol referred in 2/?
It shouldn't; this is one of the problems to solve.

CChris

31. Re: Packages

• Posted by Matt Lewis <matthewwalkerlewis at g?ail.c?m> Aug 08, 2007
• 523 views

CChris wrote:
> 
> But certainly not the last:
> 1/ will full duplex client-server relationship keep working between a main
> library
> file and a file it (recursively) includes? It should, otherwise code breaking
> may not be marginal.

No currently functional euphoria code should break.  The changes will just
make it easier to use 3rd party libraries without changing them.

> 2/ If a global symbol is privately used in a library, yet is global because
> it is referred to in several files, will the main application keep seeing it?
> It shouldn't; This is one of the problems to solve.

This is not a problem, it's one of your proposals for changing the 
language.

> 3/ If case 2/ is solved, and the application includes some other file after
> the main library file considered in 1/, will that file see the symbol referred
> in 2/? It shouldn't; this is one of the problems to solve.

N/A.

Matt

33. Re: Packages

• Posted by Jason Gade <jaygade at ya??o.com> Aug 08, 2007
• 538 views

CChris wrote:
> 
> Jason Gade wrote:
> > 
> > CChris wrote:
> > 
> > > But certainly not the last:
> > > 1/ will full duplex client-server relationship keep working between a main
> > > library
> > > file and a file it (recursively) includes? It should, otherwise code
> > > breaking
> > > may not be marginal.
> > 
> > Can you give an example? 
> > 
> 
> in main_file.e:
> global function server()
> -- 
> end function
> 
> include sub_file.e
> 
> x=client()
> ...
> --
> in sub_file.e:
> global function client()
> return server()
> end function
> 
> Of course this is too minimal an example to qualify as example code, but
> that's
> the meaning of full duplex client server relation.

I guess that it is too minimal, because this seems to work.

-- main_file.e

global function server()
    return "server "
end function

include sub_file.e

object x
x = client()
puts(1, x & "\n")

-- end main_file.e

-- sub_file.e

global function client()
    return (server() & " " & "client\n")
end function -- client()

-- end sub_file.e

-- main.ex

include main_file.e
include sub_file.e -- probably not needed...

puts(1, "Exiting main...\n")

-- end main.ex

--
"Any programming problem can be solved by adding a level of indirection."
--anonymous
"Any performance problem can be solved by removing a level of indirection."
--M. Haertel
"Premature optimization is the root of all evil in programming."
--C.A.R. Hoare
j.

35. Re: Packages

• Posted by CChris <christian.cuvier at a?riculture.g?uv.fr> Aug 08, 2007
• 512 views
• Last edited Aug 09, 2007

Jason Gade wrote:
> 
> CChris wrote:
> > 
> > Jason Gade wrote:
> > > 
> > > CChris wrote:
> > > 
> > > > But certainly not the last:
> > > > 1/ will full duplex client-server relationship keep working between a
> > > > main
> library</font></i>
> > > > file and a file it (recursively) includes? It should, otherwise code
> > > > breaking
> > > > may not be marginal.
> > > 
> > > Can you give an example? 
> > > 
> > 
> > in main_file.e:
> > global function server()
> > -- 
> > end function
> > 
> > include sub_file.e
> > 
> > x=client()
> > ...
> > --
> > in sub_file.e:
> > global function client()
> > return server()
> > end function
> > 
> > Of course this is too minimal an example to qualify as example code, but
> > that's
> > the meaning of full duplex client server relation.
> 
> I guess that it is too minimal, because this seems to work.
> 
> }}}
<eucode>
> -- main_file.e
> 
> global function server()
>     return "server "
> end function
> 
> include sub_file.e
> 
> object x
> x = client()
> puts(1, x & "\n")
> 
> -- end main_file.e
> 
> -- sub_file.e
> 
> global function client()
>     return (server() & " " & "client\n")
> end function -- client()
> 
> -- end sub_file.e
> 
> -- main.ex
> 
> include main_file.e
> include sub_file.e -- probably not needed...
> 
> puts(1, "Exiting main...\n")
> 
> -- end main.ex
> </eucode>
{{{

> 
> --
> "Any programming problem can be solved by adding a level of indirection."
> --anonymous
> "Any performance problem can be solved by removing a level of indirection."
> --M. Haertel
> "Premature optimization is the root of all evil in programming."
> --C.A.R. Hoare
> j.

It does work. I wanted to check that it still would even with the suggested mod.

CChris

37. Re: Packages

• Posted by Matt Lewis <matthewwalkerlewis at ?mail.c?m> Aug 08, 2007
• 555 views
• Last edited Aug 09, 2007

Jason Gade wrote:
> 
> Are you going to make a feature-testing branch for this?

No way.  Live fast, baby.

I figure that since Rob stated before that this would probably go into the 
code, and since no one has spoken ill of it (except to say that it doesn't
go far enough--and we can always extend it later), I'm putting it right
into the trunk.

...and it's there right now.  I managed to get it to work in the interpreter
and the translator--although it occurs to me that I didn't update the 
Euphoria-based backend...so I'll have to go back and do that (it's just
RTLookup, which is used by routine_id--the other resolution will work, 
because it's all front-end stuff).  I also even updated the docs (though
I suspect some may want to edit them).

And here is the updated test suite for the new feature.  I added frank.e
to show that included files could still access global symbols in the 
files that included them (as long as the symbol was declared before the
file was included, of course).

-- alice.e
global sequence name
name = "Alice"

-- bob.e
include alice.e
global integer x
x = 1
global procedure bob_name()
printf(1,"Bob says name = \"%s\"\n",{name})
end procedure


global procedure foo()
	puts(1, "bob.e foo()\n" )
end procedure

-- chris.e
global sequence name
? x -- ok, this is dumb, but shows that normal
-- Eu resolution is still in effect
name = "Chris"

global procedure foo()
	puts(1,"chris.e foo()\n" )
end procedure

-- diane.e
include chris.e
include frank.e


global procedure diane_name()
	printf(1,"Diane says name = \"%s\"\n",{name})
end procedure


-- erin.ex
global procedure erin()
	puts(1, "erin\n")
end procedure

include bob.e as b
include diane.e as d
include alice.e as a
include chris.e as c
bob_name()
diane_name()
printf(1,"a:name = \"%s\" c:name = \"%s\" d:name = \"%s\"\n",{a:name,c:name,
d:name})
printf(1, "d:x = %g\n", d:x )
printf(1, "erin routine_id(\"foo\")   = %d\n",   routine_id("foo") )
printf(1, "erin routine_id(\"b:foo\") = %d\n", routine_id("b:foo") )
printf(1, "erin routine_id(\"c:foo\") = %d\n", routine_id("c:foo") )
printf(1, "erin routine_id(\"d:foo\") = %d\n", routine_id("d:foo") )

puts(1, 
"\n\nAbove output should read:\n" &
"1\n" &
"erin\n" &
"Bob says name = \"Alice\"\n" &
"Diane says name = \"Chris\"\n" &
"a:name = \"Alice\" c:name = \"Chris\" d:name = \"Chris\"\n" &
"d:x = -1.23457\n" &
"erin routine_id(\"foo\")   = -1\n" &
"erin routine_id(\"b:foo\") = 0\n" &
"erin routine_id(\"c:foo\") = 1\n" &
"erin routine_id(\"d:foo\") = 1\n"  )

global atom x
x = -1.23456789
erin()

39. Re: Packages

• Posted by Bernie Ryan <xotron at ??uefrog.com> Aug 08, 2007
• 520 views
• Last edited Aug 09, 2007

c.k.lester wrote:
> 
> Matt Lewis wrote:
> > 
> > I figure that since Rob stated before that this would probably go into the 
> > code, and since no one has spoken ill of it (except to say that it doesn't
> > go far enough--and we can always extend it later), I'm putting it right
> > into the trunk.
> 
> What exactly is it that you're putting into the trunk?

Peanuts !

Bernie

My files in archive:
WMOTOR, XMOTOR, W32ENGIN, MIXEDLIB, EU_ENGIN, WIN32ERU, WIN32API 

Can be downloaded here:
http://www.rapideuphoria.com/cgi-bin/asearch.exu?dos=on&win=on&lnx=on&gen=on&keywords=bernie+ryan

41. Re: Packages

• Posted by Derek Parnell <ddparnell at bigp?nd?com> Aug 09, 2007
• 510 views

CChris wrote:
> > 
> > Because this is the most serious problem the namespace/global resolution
> > system
> > has.

 
Matt Lewis wrote:
> So far, you're the only one who thinks so.  Not that it isn't something 
> that may be ultimately desirable, but calling it "the most serious
> problem" seems like hyperbole to me.  If there's anyone else, please
> chime in, and help CChris explain why this is so serious.

I believe Chris has an exagerated sense of this issue too. It is not a problem
in the sense that people are tripping up on it all the time and are thus being
frustrated by the language design.

This strikes me as a concept that Chris would like to see implemented because
its a neat one. And I agree with that. The concept of "packages" for Euphoria is
a nice one, but the lack of it is not, as we speak, causing any general angst.

I think that if I were to advocate such a concept, I'd do only two things:
(1) Create a new keyword that defines the scope of an identifier as being
limited to the package. This would work like 'global' but things outside the
package could not access the identifier.
  eg.
 sequence mVersion = "1.0"   -- seen only in containing file
 package integer vNextID     -- seen only in containing package
 global constant Foo = "FOO" -- seen only in containing application

(2) Define the term package as meaning "all the files in the same directory".

Thus statements in C:\foo\abc.e can access 'package' items in c:\foo\def.e but
statements in C:\bar\qwerty.e cannot see those 'package' items.

-- 
Derek Parnell
Melbourne, Australia
Skype name: derek.j.parnell

43. Re: Packages

• Posted by Jason Gade <jaygade at yahoo.?om> Aug 09, 2007
• 525 views

Robert Craig wrote:
> 
> Matt Lewis wrote:
> > I figure that since Rob stated before that this would probably go into the 
> > code, and since no one has spoken ill of it (except to say that it doesn't
> > go far enough--and we can always extend it later), I'm putting it right
> > into the trunk.
> 
> I believe that was the change that I was planning to go ahead with,
> but lost interest in after a while. I have no objections now, as long
> as you are prepared to do all the work on it. 
> I believe there was a strong consensus in favor of it 
> a couple of years ago, and there probably still is.
> 
> Regards,
>    Rob Craig
>    Rapid Deployment Software
>    <a href="http://www.RapidEuphoria.com">http://www.RapidEuphoria.com</a>

I remember that you had a reason for not including it (no pun intended) but I 
couldn't find it with a quick search of the list...

--
"Any programming problem can be solved by adding a level of indirection."
--anonymous
"Any performance problem can be solved by removing a level of indirection."
--M. Haertel
"Premature optimization is the root of all evil in programming."
--C.A.R. Hoare
j.

45. Re: Packages

• Posted by CChris <christian.cuvier at agri?ultu?e.gouv.fr> Aug 09, 2007
• 516 views

Derek Parnell wrote:
> 
> CChris wrote:
> > > 
> > > Because this is the most serious problem the namespace/global resolution
> > > system
> > > has.
> 
>  
> Matt Lewis wrote:
> > So far, you're the only one who thinks so.  Not that it isn't something 
> > that may be ultimately desirable, but calling it "the most serious
> > problem" seems like hyperbole to me.  If there's anyone else, please
> > chime in, and help CChris explain why this is so serious.
> 
> I believe Chris has an exagerated sense of this issue too. It is not a problem
> in the sense that people are tripping up on it all the time and are thus being
> frustrated by the language design. 
> 
> This strikes me as a concept that Chris would like to see implemented because
> its a neat one. And I agree with that. The concept of "packages" for Euphoria
> is a nice one, but the lack of it is not, as we speak, causing any general
> angst.
> 
> I think that if I were to advocate such a concept, I'd do only two things:
> (1) Create a new keyword that defines the scope of an identifier as being
> limited
> to the package. This would work like 'global' but things outside the package
> could not access the identifier.
>   eg.
>  sequence mVersion = "1.0"   -- seen only in containing file
>  package integer vNextID     -- seen only in containing package
>  global constant Foo = "FOO" -- seen only in containing application
> 
> (2) Define the term package as meaning "all the files in the same directory".
> 
> Thus statements in C:\foo\abc.e can access 'package' items in c:\foo\def.e but
> statements in C:\bar\qwerty.e cannot see those 'package' items.
> 
> -- 
> Derek Parnell
> Melbourne, Australia
> Skype name: derek.j.parnell

Linking packages to physical directories may seem simpler, since we'd rely on
the OS and not on something new in the language to define what is the containing
package for a symbol. Well, let's see...

Unix-like file systems, as well as NTFS, allow a file to appear in several
different places using links of various flavors - possibly under different names.
Wouldn't links break havoc on the system you propose? At the same time, it would
enable a symbol to belong to various packages. I don't know if this is a good
idea, and will raise interesting issues. Speciallly if you can link files on
different computers.

If Euphoria was being designed from scratch, or if all actively used libraries
were rewritten fast to take advantage of the new facility, I would agree that
this is enough, the link issue notwithstanding. But this ia a very wrong
assertion. Note that, by "actively used", I mean: used with Euphoria interpreters
which might be upgraded. A lot of libs are used with older interpreters: they are
widely used, but not actively used in the previous sentences.

If you want to manage the long transition between uncontained and contained
globals, you'll need some external control system _as well_. Rewriting third
party code is calling for inconsistencies and other disasters. I think I
explained that in the "Why the dual mechanism" section of the paper I posted at
http://oedoc.free.fr/Packages.htm .

CChris

47. Re: Packages

• Posted by CChris <christian.cuvier at ?gricultur?.gouv.fr> Aug 09, 2007
• 532 views

Matt Lewis wrote:
> 
> Derek Parnell wrote:
> > 
> > I believe Chris has an exagerated sense of this issue too. It is not a
> > problem
> > in the sense that people are tripping up on it all the time and are thus
> > being
> > frustrated by the language design. 
> > 
> > This strikes me as a concept that Chris would like to see implemented
> > because
> > its a neat one. And I agree with that. The concept of "packages" for
> > Euphoria
> > is a nice one, but the lack of it is not, as we speak, causing any general
> > angst.
> > 
> > I think that if I were to advocate such a concept, I'd do only two things:
> > (1) Create a new keyword that defines the scope of an identifier as being
> > limited
> > to the package. This would work like 'global' but things outside the package
> > could not access the identifier.
> >   eg.
> >  sequence mVersion = "1.0"   -- seen only in containing file
> >  package integer vNextID     -- seen only in containing package
> >  global constant Foo = "FOO" -- seen only in containing application
> 
> This seems sane to me.
> 
> > (2) Define the term package as meaning "all the files in the same
> > directory".
> > 
> > Thus statements in C:\foo\abc.e can access 'package' items in c:\foo\def.e
> > but
> > statements in C:\bar\qwerty.e cannot see those 'package' items.
> 
> I like this less.  Along with CChris' reservations, I think that this
> would complicate the distribution process.  This system reminds me of
> java (not the same, obviously, but some real parallels).  Libraries
> that used the package functionality would have to be kept in their
> directory structure when distributed [as source] with applications.
> 
> It's not necessarily a bad thing, but I believe that it's different than
> how most people currently use libraries.  Easier might be to use some kind
> of a "with package xyz" sort of statement.  Package names would have to
> be treated like normal symbol names for resolution purposes, or we end
> up in the same mess all over again.
> 

All of the questions below are answered to in the paper. Details follow.

> To demonstrate this, suppose that both two math libraries used the same
> package:
> }}}
<eucode>
> with package math
> </eucode>
{{{

> Clearly, neither wants to see the other's packaged symbols.  Also, the app
> using these libs might want to see the packaged symbols of one, but not
> the other:
> }}}
<eucode>
> -- derek_math.e
> with package math
> 
> -- matt_math.e
> with package math
> 
> -- my_app.ex
> include derek_math.e as dmath
> include matt_math.e as mmath
> with package dmath:math
> </eucode>
{{{

> The "with package" may not be correct, since it's probably important to 
> declare whether you're extending the package or just using it.  Or maybe
> not.  
> 

package identifiers are exported symbols, hence cannot be redefined as packages
- they are not considered as variable/routine identifiers in my implementation -.
So, the second "with package math" will cause an error. I didn't consider a
mechanism to resolve such conflicts, but at least they are detected.
In my understanding, packaging is a very specific binding to a library, so that
package names are recommended to be very likely unique.
This point is probably not explicit enough in the paper, I'll clarify it tonight
(3pm here - cannot upload from office).

> Also, what if you use multiple "with package" statements? 

The various directives are pushed onto a stack, and the last issued one is the
active one. On reaching the end of a source file, all directives issued in the
file are popped off the stack. Popping the previous directive is allowed, using
"with previous_package". Popping a directive not defined in the current file
causes an error.

The paper explicitly states why a file may need two or more directives, a "with
package xxx" one and a "without package" one when the exported symbols start
being defined. This is a constraint resulting from define_before_use stringent
Euphoria policy. Properly flagging the identifiers as "internal" or "exported",
instead of the fuzzy "global", removes this need for sectioning.

> Can
> code be part of multiple packages?
> 

No. Every symbol belongs to a single package, the one specified by the directive
that was in force when it was defined. If no directive was ever issued in the
base file of the symbol, then:
* If the definition file is included, and if a directive was in effect at the
time of inclusion, then this directive is active - it is on top of the stack;
* Otherwise, the symbol belongs to the unnamed package. If the active directive
was "without package", the effect would be the same.

This is why it is recommended that files which don't include any other file, or
don't use intrapackage globals, be added a "without package" directive at the
beginning, so that their behaviour is not affected by the including file, and
remains as intended.

One can achieve the same result using a wrapper include file, as demonstrated in
the paper. This eliminates any code modification altogether.

A symbol may be moved from a context to another using the restrict directive.
This is a supposedly uncommon thing to do. I say "context", because it may be a
file name, namespece or package identifier indifferently.

> What if a symbol is declared as package, but there's no "with package"
> directive?
> 

Since there is no package directive, the symbol belongs to the unnamed package.
Since the unnamed package has no concept of packaged symbol, the "package"
modifier (which I call "internal") has no effect.

> Are only things surrounded by with/without package included in the package?

As mentioned before, everything belongs to a package. Either an explicit
directive is in force, and the symbol belongs to the corresponding named package
or to the unnamed package, or not, and the symbol belongs to the unnamed package.

By the way, if it is preferred to write "with package _none" rather than
"without package", this is not an issue at all. Thought the latter was more
intuitive.
> 
> No doubt there are more questions that need to be answered. 
> 
> Matt

Perhaps yet another answer: since the system is meant to make binding an
identifier to a variable a less error-prone process, and since it only deals with
this issue, only the front end needs to be modified. Translated/bound code is
unaffected, as well as any generated IL - I should add this IL bit to the paper
as well.

HTH
CChris

49. Re: Packages

• Posted by Matt Lewis <matthewwalkerlewis at gmail.??m> Aug 09, 2007
• 506 views

Derek Parnell wrote:
> 
> Maybe I just don't understand the problem or situation.
> 
> I thought that the idea was to have certain identifiers accessible from some
> files but not other files. To me, this seems to imply that we need to be able
> to indicate which identifiers can be shared this way and which files will do
> the sharing.

Yes, I agree.

> Now if you don't like the file system of grouping files into a package, we
> could
> devise a file-system agnostic method. For example, creating a special file
> that
> lists the files in a package. For example we could have a file called
> win32lib.e
> that contains ...
> 
>  package win32lib
>     include win32/memory.e
>     include win32/core.e
>     include foo/bar.e
>  end package
> 
> Which would mean that statements inside these three files could access the
> 'package'
> designated identifiers in those files and no other file could.
> 
> Then to use this package in an application we just use the standard mechanism
> ...
> 
>   -- myap.exw
>   include win32lib.e

Ok, so we (that is, the parser, etc) would just have to know not to allow
myap.exw to see any of the symbols declared as 'package' within any of those
files, unless it was included elsewhere outside of a package definition.

So [thinking out loud here] let's assume that machine.e had some stuff
declared as package, and it's included somewhere within win32lib.  myap.exw
could
still see the machine.e packaged symbols if it explicitly included 
machine.e somewhere.

Should we use a different syntax for either declaring symbols or the 
package...end package to differentiate which one we're doing?) Maybe
with/without package:

-- win32lib.e
 with package
    include win32/memory.e
    include win32/core.e
    include foo/bar.e
 without package

 -- myap.ex
 include win32lib.e

Do we even need to name a package?  How would the name be used?

And presumably, all symbols declared (in an include or in the file itself)
would be part of the package, and subject to the new visibility rules:

-- app.ex
with package

package atom myAtom
include someFile.e  -- can use myAtom

without package

include otherFile.e -- can't use myAtom

What about 'nested' packages?  For example, somewhere within the bowels 
of win32lib, there's a file or other library that declares its own package.
Should files within the second-level package be able to see symbols within
its including package?

-- app.ex
with package
package atom app
include a.e
without package
...

-- a.e
...
with package
include b.e
without package
...

-- b.e
? app

It doesn't seem to me like a very sane thing to do, but you know that 
someone will try it.  I ask in the interest of trying to flesh out some
corner cases.

> I'm absolutely convinced that Chris' implementation is far to complex to be
> useful or used. I cannot see the need for "with package" etc ... 

Yeah, I'm concerned about its complexity, too.  Now that we have a better
namespace and symbol resolution system, this is more about encapsulation
for its own sake (generally a good thing) than about fixing actual problems
that people have.

> Matt wondered how to use a maths package from two different 'vendors'. Simple!
> 
>    include derek/math.e as dp
>    include matt/math.e as ml
> 
>    ...
>    a = dp:abs(x)
>    b = ml:abs(x)

My questions were really assuming that we were somehow 'inserting' symbols
into specific packages, and that it would be meaningful to give them 
names.  Part of my thinking (which I admittedly hadn't fully explored)
was about how the parser would handle this sort of thing.

> And I can't see how keeping all the files of a package supplied by a single
> vendor in the same directory is wierd or unusual.

My concern was more that once we go down this road, we *have to* keep 
these things in separate directories or risk breakage.  I'd prefer to
stay away from such a solution if possible.

Matt

51. Re: Packages

• Posted by CChris <christian.cuvier at ag?i?ulture.gouv.fr> Aug 09, 2007
• 523 views

Matt Lewis wrote:
> 
> CChris wrote:
> > 
> > Perhaps yet another answer: since the system is meant to make binding an
> > identifier
> > to a variable a less error-prone process, and since it only deals with this
> > issue, only the front end needs to be modified. Translated/bound code is
> > unaffected,
> > as well as any generated IL - I should add this IL bit to the paper as well.
> 
> Not true.  There's also the issue of taking routine_ids.  That requires
> some communication between the front and back ends to let RTLookup and
> CRTLookup to understand what's in and out of the packages.  I ran into
> this implementing the symbol resolution.
> 
> It required changing the interface between front and backend (take a look
> at the end of backend() in backend.e) and the initialization of 
> translated code.  Take a look at the code generated before the call to 
> eu_startup() in main-.c (and the changes to eu_startup() in be_runtime.e).
> 
> We'll need to run a similar drill for packages, regardless of how it 
> finally gets done.
> 
> Matt

Hmm. I'll have to take a deeper look at this. Using my modified interpreter, I
couldn't get it to barf. I didn't check how visibility of routine_ids is handled
in translated code, so you are probably right there.  The IL generation remains
unaffected though.

Unless... 

Did you notice how annoying it is to define a routine as global only for the
sake of passing a valid id for it? "Long distance" calling a local routine is
easy if the latter provides its callback address somewhere, which is a hideous
way to hide it from the global resolution mechanism when it is an annoyance, and
yet use it as global, since machine addresses are global.

Wouldn't it be time to simplify the use of the language a little and allow
calling routines by their previously obtained ids, even if they are not in scope?
The fact that one must use call_func/call_proc() to do this will make it clear to
the reader of the code that what is done is being done outside the basic, regular
calling system. You wouldn't be able to get the routine_id of an out of scope
symbol directly anyway - the identifier resolution system will prevent it. And it
would make some (six letter word) constructs much cleaner to implement.

Perhaps are you wondering why to pass ids of local outines anywhere in the first
place? Just go take a look at http://oedoc.free.fr/Fichiers/ESL/routines.zip .
Among other things, this file defines functions on routines, which need ids as
arguments. So, currently, if a file wishes to build a third routine from two
local ones inside it, it must make them global so that their routine ids are
valid in the external library - even though the functions themselves are never
called outside of their intended scope! Test it; you'll see how pointlessly
unpractical this routine id validity system can be. Likewise, event handlers need
to be global only because they are called by an outside GUI library instead of
the OS itself. Not a very convincing justification.

The issues of global resolution and routine ids were a priori pretty much
independent. Perhaps this is a good opportunity to improve two very indirectly
related areas of the language in one sweep.

Just some thoughts. 
CChris

53. Re: Packages

• Posted by Matt Lewis <matthewwalkerlewis at gmail.c??> Aug 09, 2007
• 532 views

CChris wrote:
> 
> 
> Hmm. I'll have to take a deeper look at this. Using my modified interpreter,
> I couldn't get it to barf. I didn't check how visibility of routine_ids is
> handled
> in translated code, so you are probably right there.  The IL generation
> remains
> unaffected though.

Yes, IL will remain the same.  It's just the implementation of RTLookup
(or CRTLookup for translated code) that has to change.  Those routines do
symbol resolution at runtime, so your compile-time changes won't affect 
them.

<snip>

> The issues of global resolution and routine ids were a priori pretty much
> independent.
> Perhaps this is a good opportunity to improve two very indirectly related
> areas
> of the language in one sweep.

I think you're confusing this with the issue that came up when Rob forgot to 
check to see if the target of the routine_id call were declared after the 
actual routine_id call itself.  As Rob, posted, call_func/proc completely
ignore scope or order of declaration issues.  They're basically unrestricted
function pointers.  You're just restricted in how you create them.

Matt

55. Re: Packages

• Posted by Pete Lomax <petelomax at blue?o?der.co.uk> Aug 10, 2007
• 529 views

Derek Parnell wrote:
> 
> Matt Lewis wrote:
> > 
> > Derek Parnell wrote:
> > > 
> > > I thought that the idea was to have certain identifiers accessible from
> > > some
> > > files but not other files. To me, this seems to imply that we need to be
> > > able
> > > to indicate which identifiers can be shared this way and which files will
> > > do
> > > the sharing.
> > 
> > Yes, I agree.
Ditto
> > 
> > > Now if you don't like the file system of grouping files into a package, we
> > > could
> > > devise a file-system agnostic method. For example, creating a special file
> > > that
> > > lists the files in a package. For example we could have a file called
> > > win32lib.e
> > > that contains ...
> > > 
> > >  package win32lib
> > >     include win32/memory.e
> > >     include win32/core.e
> > >     include foo/bar.e
> > >  end package
> > > 
I was thinking along the same lines, but your last comment made me realise this
is not needed, see below.

> > So [thinking out loud here] let's assume that machine.e had some stuff
> > declared as package, and it's included somewhere within win32lib.  

<aside>
A point that should be made here, or rather just stating the obvious:
As things stand, machine.e does not have any package definitions, so if it
happens to be included as part of win32lib, in no way does it mean that anything
global in it will suddenly become invisible to any program that includes win32lib
with some new whizz-bang package system.
</aside>
> > myap.exw could
> > still see the machine.e packaged symbols if it explicitly included 
> > machine.e somewhere.
<aside continued>
And myap.exw could still see machine.e globals even if it does not.
</aside>
> 
> To me, a package is a related set of files designed to operate as such by the
> author of those files.
> > What about 'nested' packages?
> > My concern was more that once we go down this road, we *have to* keep 
> > these things in separate directories or risk breakage.  I'd prefer to
> > stay away from such a solution if possible.
> 
> But, the author of "packaged" symbols designed it to be used in a certain
> manner,
> so the author expects the package to stay together.

From all this I read it need not be a "directory" at all but could and probably
 should be explicitly derived from the include statements.

If it were up to me, I'd probably go with "package <name>" as the scope opener
to allow explictly naming the scope that is being shared/not open a new one if
that name is already active, and "private integer/function".

As an example for nesting, suppose dll.e wants to privately share a coupla
things with machine.e (as unlikely as that may actually be):

-- win32lib.ew
package win32lib
private r_DoStuff
include dll.e
   -- dll.e
   package dllmach
   private x,y
   include machine.e
       -- machine.e
       package dllmach
       private p,q
       <eof>
   <eof>
include w32file.e
<eof>

The proposal I make is that "package <name>" opens a new "scope" which covers
sub-includes and automatically closes at eof.

Hence in the above, dll.e and machine.e can see x,y,p,q but win32lib cannot,
and win32lib and w32file can see r_DoStuff but anything that includes win32lib
cannot. Further w32file need not have any package statement unless it wants to be
explicit about it.

A couple of undecided points:
 Whether dll.e and machine.e can also see r_DoStuff, or the new scope should 
  automatically suspend handling of an including package scope.
 Whether the scope name remains known/global after the <eof> of dll.e, but I 
  doubt it needs to.

I agree with Derek this package stuff means that the author intended them to be
private (as above a keyword I favour) and any notion of accessing these things
from outside is a plain no-no.

Regards,
Pete
PS One argument I had against the directory idea was that it could hamper the
structure of a lib component itself into subdirectories. Also, if two independent
components want to include series.e it could mean that it would have to be
installed twice. But I think the above covers it anyway.

57. Re: Packages

• Posted by Matt Lewis <matthewwalkerlewis at g?a?l.com> Aug 10, 2007
• 521 views

Pete Lomax wrote:
> 
> <aside>
> A point that should be made here, or rather just stating the obvious:
> As things stand, machine.e does not have any package definitions, so if it
> happens
> to be included as part of win32lib, in no way does it mean that anything
> global
> in it will suddenly become invisible to any program that includes win32lib
> with
> some new whizz-bang package system.
> </aside>

> <aside continued>
> And myap.exw could still see machine.e globals even if it does not.
> </aside>

Yes, it was an incredibly poor example.

> If it were up to me, I'd probably go with "package <name>" as the scope
> opener to allow explictly naming the scope that is being shared/not open a new
> one if that name is already active, and "private integer/function".
> 
> As an example for nesting, suppose dll.e wants to privately share a coupla
> things
> with machine.e (as unlikely as that may actually be):
> 
> }}}
<eucode>
> -- win32lib.ew
> package win32lib
> private r_DoStuff
> include dll.e
>    -- dll.e
>    package dllmach
>    private x,y
>    include machine.e
>        -- machine.e
>        package dllmach
>        private p,q
>        <eof>
>    <eof>
> include w32file.e
> <eof>
> </eucode>
{{{

> 
> The proposal I make is that "package <name>" opens a new "scope" which covers
> sub-includes and automatically closes at eof.
> 
> Hence in the above, dll.e and machine.e can see x,y,p,q but win32lib cannot,
> and win32lib and w32file can see r_DoStuff but anything that includes win32lib
> cannot. Further w32file need not have any package statement unless it wants
> to be explicit about it.

My concern with this is that everyone has to choose unique package names,
or libraries with conflicting package names risk leaking into each other.

I like Derek's implementation, but still question whether we need to
name packages.  The only reason I can see would be for diagnostic reporting.
We could use the file name, but what if there are multiple packages in a
single file?

-- gui.e
package memory
    include alloc.e
    include free.e
end package

package gui
    include window.e
    ...
end package

If we decide names aren't required (or maybe even if we do), then the 
following would work:

-- gui.e
with package memory
    include alloc.e
    include free.e
without package

with package gui
    include window.e
    ...
without package

Another advantage of with/without is that we don't introduce another keyword
into the language.

Matt

59. Re: Packages

• Posted by Pete Lomax <petelomax at blueyond?r?co.uk> Aug 10, 2007
• 510 views

Matt Lewis wrote:
> 
> So if the gui.e guy wanted to be able to see stuff in the memory package
I'm struggling to see why gui.e would want priviledged access to the private
bits of memory.e in a way other components were not allowed. Surely if gui.e
wants to use memory.e it should just stick with the publically defined external
api??

> My concern with this is that everyone has to choose unique package names,
> or libraries with conflicting package names risk leaking into each other.
Hmm. It may be possible to use the one guaranteed unique thing we have to hand:

-- win32lib.ew
package ./win32lib.ew
include file/w32file.e
<eof>
-- w32file.e
package ../win32lib.ew
<eof>

IE the package name is a full expanded path and filename. There may be other
benefits to this, namely not using F:\test\win32lib.ew with
C:\Euphoria\include\w32file.e, which may have version incompatibilities.

Regards,
Pete

61. Re: Packages

• Posted by Matt Lewis <matthewwalkerlewis at ?mail.?om> Aug 10, 2007
• 534 views

Pete Lomax wrote:
> 
> Matt Lewis wrote:
> > 
> > So if the gui.e guy wanted to be able to see stuff in the memory package
> I'm struggling to see why gui.e would want priviledged access to the private
> bits of memory.e in a way other components were not allowed. Surely if gui.e
> wants to use memory.e it should just stick with the publically defined
> external
> api??

Hey, I'm not saying it's a good idea.  Although there are times when you 
find that a library does what you need, but some of the details are kept
private, so I think that there are legitimate reasons to do this.  This
would be a better solution than just going in and making stuff global.

The two authors could be one and the same, and decides to allow two 
libraries to work together more closely.  There are probably lots of 
good and bad reasons to do this.

Also, good idea or not, it will be tried, so we should define the expected
behavior up front.  I'm just trying to think through the consequences of
the design.

> > My concern with this is that everyone has to choose unique package names,
> > or libraries with conflicting package names risk leaking into each other.
>
> Hmm. It may be possible to use the one guaranteed unique thing we have 
> to hand:

-- win32lib.ew
package ./win32lib.ew
include file/w32file.e
<eof>
-- w32file.e
package ../win32lib.ew
<eof>

> IE the package name is a full expanded path and filename. There may be other
> benefits to this, namely not using F:\test\win32lib.ew with
> C:\Euphoria\include\w32file.e,
> which may have version incompatibilities.

This just seems to be a portability nightmare to me.  Also, I still don't 
see the need for having an actual package name as a global symbol.  It 
just means that you have to wrap your packages with another file, rather
than sprinkling the package declarations throughout them.  This has the
bonus that it's easy to see what's in there, rather than searching through
the files.

Matt

63. Re: Packages

• Posted by Matt Lewis <matthewwalkerlewis at gm?il.co?> Aug 10, 2007
• 514 views

Pete Lomax wrote:
> 
> Matt Lewis wrote:
> >
> > > the package name is a full expanded path and filename.
> > 
> > This just seems to be a portability nightmare to me.
>
> Huh? Is there some new OS out there which does not have paths & filenames?
> I honestly cannot think of any reason why that raised a portability issue.

Well, there's the slash issue, but of course, that can be dealt with.  
However, some file systems are case sensitive, and others are not.  I
believe that most people stick with all lowercase, but this is certainly
not a requirement.  Again, not unsurmountable (after all, the same issue
is present when including a file).  Nevertheless, as far as I'm 
concerned, it's creating ways to fail when (I believe that) there are 
other ways to accomplish the same goals without these drawbacks.

I probably used the wrong word.  But it just seems too fragile to me, 
since if any of those things change in relation to EUINC or each other, 
the packaging is easily destroyed.  Not so much portability across
OSes, but between people.  The author of an application that uses a
library might prefer to distribute the app with all of the source files
in a single directory.  Or organize his includes differently.

You might argue (and on a different, but similar issue, Derek already 
has) that as end users, we should follow the end user's intent to keep
these things packaged, and in your case, that would mean following his
path structure.  It also seems more difficult to maintain than it needs 
to be.

I suppose that part of my goal here is to enrich the language without 
adding new constraints about the way it's used.  And I think that in
this case we can accomplish this, but not the way you've proposed.

> > Also, I still don't 
> > see the need for having an actual package name as a global symbol.
>
> I only percieve package names as global in the sense that in parser.e the new
> routine doPackage() has a permanent list of ones it has seen before.

Yes, that's what I meant.  But to me this just begs for package naming 
conflicts.
 
> > It 
> > just means that you have to wrap your packages with another file, rather
> > than sprinkling the package declarations throughout them.  This has the
> > bonus that it's easy to see what's in there, rather than searching through
> > the files.
>
> Now I'm confused: what "It" means that, and who said it?
> AFAIK, I'm the one arguing against "directory" and "file" as the scope, while
> leaving the latter as the default by auto-closing at eof.

Sorry.  'It' refers to defining packages in the way (or equivalent) Derek
proposed.  
 
> Here is another ficticious scenario:
> 1) file.e and machine.e want to share some private data for a new RTFatal
> style
> error incorporating machine info and last file access details.
> 2) file.e and database.e want to share some buffer pointers etc for a new
> flush(-1)
> or flush_all() statement.
> Obviously as with all trivial examples it would not really matter if you had
> to throw all this in the same pot, but it ought to be more scaleable.
> It seems harder to me to divide the data up like that with a "file list"
> mechanism
> than a "private scope xxx" one. With the former you might have to duplicate
> the entire list in multiple files whereas with the latter you would only have
> to repeat the "xxx" name.
> 
> I can see that a "package filemach,filedb" could be useful to declare a
> private
> item in file.e which is visible to both machine.e and database.e although
> other
> stuff in "filemach" is not visible to database.e and other stuff in "filedb"
> is not visible to machine.e. I can also see that an explicit "end package" and
> multiple package directives in the same file could be useful, I never actually
> ruled that out but then again I never actually said as much either.
> 
> This all seems quite trivial and maybe even elegant in my head, although I can
> tell it is not getting across like that 

It never does, does it. :)  

On the one hand, I can see the benefit of having the package declaration 
right in the file where you're using it, cluing you in to look for 
private symbols in other files.  But that's not really any different 
than global variables, and from within a file, there's no way to know if 
an external symbol is global or private, so I don't think that argument
holds a lot of water.

On the other hand, using something similar to Derek's method defines all
of the files in the packages in one place, so there's no need to search
through lots of files (and still maybe not be sure that you found them
all).  And by using this method, we've removed the requirement (I think)
for adding a new category of things with names to clutter up the namespace.

Matt

65. Re: Packages

• Posted by Pete Lomax <petelomax at bl??yonder.co.uk> Aug 11, 2007
• 545 views

Matt Lewis wrote:
> 
> other ways to accomplish the same goals without these drawbacks.
>
I guess what is needed here is a set of practical challenges that each proposer
has to solve, a mini-exam if you will. Then we can compare the answers each
method gives. Any starters?

Regards,
Pete

67. Re: Packages

• Posted by Pete Lomax <petelomax at blue?onder?co.uk> Aug 11, 2007
• 527 views

CChris wrote:
> 
> Perhaps your Design Central site would be the right place to host this:
<snip>
> Sounds good?
Sure, I can do that.

Regards,
Pete