OpenEuphoria

Euphoria Ticket #412: Distribute bundled tools as binaries

• Reported by mattlewis Nov 19, 2010
• 43 comments

Details

Type:	Task	Severity:	Normal	Category:	Distribution
Assigned To:	mattlewis	Status:	Fixed	Reported Release:
Fixed in SVN #:	4222	View VCS:	4222	Milestone:	4.0.0RC2

1. Comment by jimcbrown Nov 22, 2010

Is this right?

+       echo eubind $$\@ >> $(DESTDIR)$(PREFIX)/bin/eushroud 

I don't see eushroud passing the -shroud_only option to eubind. I also don't see how bind.ex would shroud unless that option was present.

Also, I was under the impression that only Linux/GNU allowed this type of script. E.g. on Solaris or NetBSD, the shebang is required or the shell script won't execute.

2. Comment by jeremy Nov 22, 2010

I think we decided to not distrubute eushroud as a binary, right? It's just eubind -shroud Thus, eushroud should just disappear?

3. Comment by jimcbrown Nov 22, 2010

We could make a binary out of this:

include std/cmdline.e 
sequence s = command_line() 
s = {"eubind", "-shroud_only"} & s[3..$] 
system(build_commandline(s), 2) 

4. Comment by mattlewis Nov 22, 2010

It's currently implemented like this:

#!/bin/sh 
eubind $@ 

So...it's missing the -shroud_only flag, but it's not really a binary itself.

5. Comment by jeremy Nov 22, 2010

the problem with that is on Windows. We cannot fully support that on Windows thus platform usage will be different.

6. Comment by jimcbrown Nov 22, 2010

We could make eushroud.bat

@ECHO OFF 
eubind -shroud_only %1 %2 %3 %4 %5 %6 %7 %8 %9 

7. Comment by jimcbrown Nov 22, 2010

We should probably make it a true binary. For DOS batch files, it's pretty hard to get it to accept filenames with spaces or quotes in them.

Right now the shell script version won't take those either, but we could fix it by changing

$@ 

to

"$@" 

8. Comment by jeremy Nov 22, 2010

or we could just tell people to shroud, eubind -shroud

9. Comment by jeremy Nov 22, 2010

Looking at the list in the description, shouldn't we also include:

• eutest
• eudoc
• creolehtml

? That would then be the list I included w/the Windows install. I think dis.ex should be eudis.ex as well, no? Better citizen on *nix systems.

10. Comment by jimcbrown Nov 22, 2010

I don't think so. eudis and eubind are a third and fourth mode of the interpreter/translator (including main.e and calling BackEnd() ) so it makes sense for them to stay part of this project.

eudist and more weakly eucoverage.ex could possibly be moved to their own projects. But the difference between these 4 and those 3 is that those 3 are their own separate projects (well, at least that is the plan for eutest).

11. Comment by jeremy Nov 22, 2010

In regards to the eushroud.bat file with %1-%9, consider that you could very easily bust right past the %9 and then things would just not work as you thought. Consider:

C:\MyProgram> eushroud.bat -full_debug -out abc.il -i ../include -i ../win32lib -c eu.cfg -i ../personal_incs 

In this case the command would fail but the user supplied no commands in err. The final -i wouldn't get the ../personal_incs sent to eubind.

12. Comment by jeremy Nov 22, 2010

I don't understand what you mean here. Also, if it includes main.e, then that's more reason to make it a binary dist because going forward into a debian package (and most other Linux package systems) there will be a binary install and source install. Many will not install the source and it's not really proper to make the binary package depend on the source package.

As for eutest, that will become a separate project however the current version of that project should be bundled with Euphoria. Except in the case possibly for Linux package managers that have both eutest and euphoria already accepted into their package systems.

13. Comment by jimcbrown Nov 22, 2010

I don't know of a way around that. It's a limitation of using DOS Batch files.

14. Comment by jimcbrown Nov 22, 2010

What I was saying was that it makes sense to draw a line between those tools that are part of the project (and thus should be packaged with it) and those that are not (and thus, should not be).

I don't have either eudoc or creolehtml installed, and haven't needed them.

15. Comment by jeremy Nov 22, 2010

Right, that's why I think we should simply do away with eushroud. Supporting it on one platform and not another (or partially/error prone) is a bad idea I think. It would be better to state to shroud something: eubind -shroud_only I think.

16. Comment by jeremy Nov 22, 2010

With eudoc and creolehtml... I guess I was thinking along the lines of promotion of the method for documenting your Euphoria source. There is power in a standard, javadoc for example. Actually in the manual there are docs about how to use eudoc, under the miniguides section.

17. Comment by jimcbrown Nov 22, 2010

Well, a binary for eushroud is only 4 lines of code... but eubind -shroud_only might be the right way to go.

Anyways, including eudoc and creolehtml make sense from that perspective. I can see how this is going to be technically hard to implement (as the person might not have eudoc and creolehtml source checked out and even if it is, there's no standard place to look for it), but that doesn't mean it's a bad idea.

18. Comment by jeremy Nov 22, 2010

I guess the packagers at this point in time is a pretty tight nit group, we did it for RC2. At least on Windows, Linux (generic, non-debian), FreeBSD and OS X.

19. Comment by mattlewis Nov 22, 2010

I don't think we should be including eudoc and creolehtml in the main euphoria distribution.

The availability of source vs a source package is a red herring. Source packages (at least debian source packages) are really just tarballs of code. We install other euphoria source code along with the binary, namely the standard library. In theory, that could be a separate developer package, but this is a lot more convenient.

20. Comment by jimcbrown Nov 22, 2010

And what about euindent?

21. Comment by jeremy Nov 22, 2010

Then we should remove the sections from the manual. Further, I would wonder then what is the justification of putting eutest in the distribution? Others are free to come up with their own testing tool.

22. Comment by jeremy Nov 22, 2010

I see that as an add-on utility, not necessary for any standard or any part of making Euphoria function. Thus, I wouldn't include it.

23. Comment by mattlewis Nov 22, 2010

The debian package, at least, includes all of the demos and the stuff that's in the bin directory. So, while we're not distributing binaries, the stuff is still there. I think this is a reasonable strategy for including stuff that's interesting, but not necessarily critical for using euphoria.

That stuff ends up in /usr/share/euphoria btw.

24. Comment by jeremy Nov 22, 2010

We need to think about the larger scope, I think. Debian is just one Linux OS. A source package is not a red herring but a must, thus we should be planning for it right now even if we do not have it right now. The planning is very simply do not make bin programs dependent on source. If it were a complex case that was not easily solved, I wouldn't worry about it now but it's very simple to make all of our bin programs function w/o the presence of source.

Further, a source package already exists on Windows. During the installation, the users have the ability to choose many packages they wish to install or don't wish to install. One of those is source. Thus, the Windows distribution already deals with this.

25. Comment by mattlewis Nov 22, 2010

We do create a source package, at least for debian packaging. Please read what I wrote. You're confusing a source package with including all or part of the euphoria source code in another package. Our package includes some euphoria utilities, some of which may run interpreted.

26. Comment by mattlewis Nov 22, 2010

As to eutest vs eudoc / creolehtml: I think eutest is sufficiently documented, whereas eudoc / creolehtml are far from it. To be fair, working with eutest is relatively simple in comparison. So, it may be a good idea to include those in the future, but not now. I think also that eutest is generally more mature than eudoc / creolehtml. If anything, I agree that eutest could become its own thing eventually.

But at this stage, when there is very little unit testing in the euphoria community, I think eutest also serves an important purpose in the evangelization of testing our code.

27. Comment by jeremy Nov 22, 2010

So it's possible to install a debian binary package and not include anything, anywhere from source/ ? And it's also possible to install a debian source package that does not include any bins at all, just the source?

28. Comment by mattlewis Nov 22, 2010

We can build whatever sort of package we want, as far as the .deb type packages go. In the .deb world, you get the .deb files and .tar.gz files. The .tar.gz files are the source packages. They get some special stuff for ID purposes, signatures, etc, but are basically just a source tree that you can build, install, etc on your own.

The .deb files are used by dpkg, and install, configure, define dependencies, etc. Our package might be a little too inclusive for strict debian standards, as they seem to generally have 3 different types:

• regular (usually, whatever you need to run the program)
• docs
• dev (usually header files and/or debug versions of binaries0

It's also common to have meta packages, which just pull together different packages. Longer term, it's probably worth taking a closer look at how other interpreted languages break components into packages. It's not always clear to me what would go into the "regular" vs "dev" packages. What about the standard library? Demos? At this point, we have an all-inclusive package.

29. Comment by jeremy Nov 22, 2010

So we do not have source packages as I am speaking of them. My whole point is simply that anything meant to be executed on the command line, eui, eubind, eucoverage, etc... should not rely on the source being present. One should be able to

# rm -rf /usr/share/euphoria/source 

and continue to use all the Euphoria commands w/o issue.

30. Comment by mattlewis Nov 22, 2010

I think that's currently the case, at least for the stuff that actually relies on the euphoria source code (I'm not sure about some demos or other things that might have snuck up on me like euindent). But this seems like a pretty arbitrary standard.

In fact, when we actually start being able to embed euphoria as a scripting language inside a euphoria program, the source will likely be necessarily distributed.

31. Comment by jeremy Nov 22, 2010

Just a FYI, euindent is not an official tool, it's not even in the eu svn/hg repos. It's just a tool I was working on as a side thing.

32. Comment by jeremy Nov 22, 2010

I am no debian expert but when getting efte into debian I learned quite a bit about how difficult it is. It was just my experience, every one I worked with expressed how difficult it was including the debian devs. Many stated that it was what keeps debian clean and usable.

One of the issues was we had the default configuration in the source directory (src) because it was compiled into some versions as a binary resource. Our package was rejected because the src directory was required for the binary package. Thus, it's not arbitrary, but I'm no debian guru, just the experience from one little package.

How other scripting languages break things up, from my non-technical analysis of peeking around, I would say:

1. /usr/bin/{eui, eudist, euc, eutest, ...}
2. /usr/share/doc/euphoria/{demos, tutorial, html, *.pdf}
3. /usr/lib/euphoria/{*.e, std/*, euphoria/*, ...}

33. Comment by mattlewis Nov 22, 2010

Yes, I'm by no means arguing that we're ready, packaging wise, for something on the level of debian. However, we're pretty close to the line of maxing out available resources while having something useful.

34. Comment by jeremy Nov 22, 2010

I believe there is actually more documentation on eudoc than eutest. There is documentation in the included tools section of the manual as well as a pretty good wiki page that is linked from the manual. I use eudoc in a few projects and have not run across any problems. The only way it's going to mature further is if others begin using it as well. A standard doc format is important, especially in a smaller community. I firmly believe that Testing and Documenting facilities should be provided by Euphoria and highly encouraged for all projects. Giving people a standard tool to do such a thing is a great start. Next comes encouragement.

35. Comment by mattlewis Nov 22, 2010

This may very well be true. However, there is very little documentation that I am aware of for creolehtml. Eudoc itself is fairly simple and straightforward. I think I've written about as much documentation on creolehtml as anyone, and I think a lot of it is now out of date, and a lot of it is incomplete.

In particular, how do the plug ins work? What are the available settings, and what do they do?

36. Comment by jeremy Nov 22, 2010

Yes, that's true. However, I do not think that should stop it's use. There is basic information on how to use it in conjunction with eudoc. Most people are not going to get into writing plugins and other extensive runtime modifications. Most are going to do:

$ eudoc -o myhelp.txt mylib/abc.e mylib/def.e mylib/xyz.e 
$ creolehtml myhelp.txt 
$ firefox myhelp.html 

Further, as you said previously we are pushing our limits on available resources. We have a wiki that anyone can edit. As questions are asked, we should get into the habit of answering the question by updating the wiki and pointing people there. We write once, answer once and have a lasting resource. As the wiki docs build, they are in creole! Just migrate to creolehtml's own help page.

37. Comment by DerekParnell Nov 22, 2010

It's my responsibility to document creolehtml and creole.

These should have already been written. Sorry.

38. Comment by mattlewis Nov 22, 2010

The problem isn't writing plugins so much as using the existing ones. It's probably fairly easy to get basic docs built from scratch. But the last time I looked, there was a lot of stuff in the best existing example (i.e., euphoria) that aren't documented.

39. Comment by jeremy Nov 22, 2010

Ok. So if before 4.0RC2 is released a simple bullet list is added to a wiki page, creolehtml, that explains the things used in euphoria's docs, then that would calm your nerves some about releasing creolehtml w/Euphoria? Mainly:

TOC, @@ macros, and a the %% variable options?

40. Comment by mattlewis Nov 22, 2010

The documentation would go a long ways towards making it release quality. Though I'm not sure how I'll integrate it into the deb package.

41. Comment by jeremy Nov 22, 2010

That is a problem, yes. It's not a tool that fits our description of what should be included in the bin directory. Ideally, creolehtml would be a dpkg of it's own, but I'm sure that's a lot of work. Maybe name it differently? eudoc2html? That would minimize it's purpose greatly but make it easier to integrate right now into our existing dpkg.

42. Comment by jeremy Nov 26, 2010

We are done w/this, right? The windows, FreeBSD and generic Linux builds do this already, so it was just the debian package that needed updated.

43. Comment by mattlewis Nov 26, 2010

Yes, these things are binaries now. There are some lingering issues about whether to include eushroud, but I'm closing this since these tools are all binaries now.

Include:

     News
     Tickets
     Forum
     Wiki
     Manual