OpenEuphoria

2. Re: ooeu [was Re: Proposal for math.e (2007-08-23)]

• Posted by Kat <KAT12 at coosah?.ne?> Aug 30, 2007
• 574 views

Matt Lewis wrote:
> 
> Kat wrote:
> > 
> > CChris wrote:
> > > 
> > > As for string execution, did you look at Matt's OOEu variant?
> > 
> > Yes, i did, and i like OOEU a lot. OOEU's oop-ish if you like, 
> > but it isn't if you don't want. I like that implementation of classes a lot.
> > 
> > But.. it can be *extremely* slow at some things. I suspect converting 
> > it to the newer Eu versions would speed it up, since most of the lag 
> > seems to be in memory allocation when dealing with dozens of megabytes 
> > of data coming in.
> 
> The issue is that I haven't ported the eval() functionality to the C-based
> backend yet.  I think it's possible (at least for interpreted code), but
> there are some serious changes that need to be made.  

But even loading a disk file into an array already made with repeat(), 
the bigger the file, the slower it goes, drastically. I am talking about 
a 34megabyte file in this case, but i have files 100x that big.

<snip>

> > I hope all of OOEU appears in some form in openeu, with appropriate 
> > credits to Matt.
> 
> Thanks.  I think certain things should be left out, like some of the 
> dynamic goto stuff.  I think the static gotos are a good thing to add,
> or at the minimum, a more structured form of them for things like exiting 
> loops with more flexibility and power than we currently have.  There are
> also some changes that I would make to the way OOP works to make it 
> better.

As usual, i'd prefer all of it be included. While the dynamic goto 
is used rarely, that's not because it isn't needed in the cases it's used.
I needed it way back in C64 days, and wrote enough peeks and pokes to 
modify the oem rom code to make it happen. As for Euphoria, it makes a 
handy case() statement as well. 

I suppose there's no need to restart a flame war about goto, 
so i won't argue it anymore, i'll just say i want it.

It was me, i confess,
and you prolly figured out, i emailed CK to add goto to the bounty site.
It doesn't make sense to throw away good code you already wrote 
just because a few people won't use it.

Kat

4. Re: ooeu [was Re: Proposal for math.e (2007-08-23)]

• Posted by c.k.lester <euphoric at ?k?ester.com> Aug 30, 2007
• 563 views

Kat wrote:
> 
> It was me, i confess,
> and you prolly figured out, i emailed CK to add goto to the bounty site.

:D

The bounties on the Bounty Board are fake, for example purposes only. Although
I might pay Matt $50,000 to get that wxIDE fixed up. ;)

Anyway, like I said, the Euphoria Bounty Board (EBB) might be operational in
a few weeks. If you have ideas or bounties you want to start, now would be a
good time to email me.

6. Re: ooeu [was Re: Proposal for math.e (2007-08-23)]

• Posted by Matt Lewis <matthewwalkerlewis at ?mai?.com> Aug 30, 2007
• 573 views
• Last edited Aug 31, 2007

Kat wrote:
> 
> But even loading a disk file into an array already made with repeat(), 
> the bigger the file, the slower it goes, drastically. I am talking about 
> a 34megabyte file in this case, but i have files 100x that big.

That seems odd.  I guess I haven't really worked with a data set that 
large to notice the progressive slow down.  Is it possibly an issue of 
swapping memory out?  I'll try to run some tests to see what happens.

> <snip>
> 
> > > I hope all of OOEU appears in some form in openeu, with appropriate 
> > > credits to Matt.
> > 
> > Thanks.  I think certain things should be left out, like some of the 
> > dynamic goto stuff.  I think the static gotos are a good thing to add,
> > or at the minimum, a more structured form of them for things like exiting 
> > loops with more flexibility and power than we currently have.  There are
> > also some changes that I would make to the way OOP works to make it 
> > better.
> 
> As usual, i'd prefer all of it be included. While the dynamic goto 
> is used rarely, that's not because it isn't needed in the cases it's used.
> I needed it way back in C64 days, and wrote enough peeks and pokes to 
> modify the oem rom code to make it happen. As for Euphoria, it makes a 
> handy case() statement as well. 

I should have been more specific.  I was really referring to the dynamic
labels.  I agree that being able to use a variable with the goto statement
would be good.  I don't regret writing the dynamic labels, cause it was
kinda neat, but it makes things more complicated, and is really just 
asking for spaghetti code.

> I suppose there's no need to restart a flame war about goto, 
> so i won't argue it anymore, i'll just say i want it.
> 
> It was me, i confess,
> and you prolly figured out, i emailed CK to add goto to the bounty site.
> It doesn't make sense to throw away good code you already wrote 
> just because a few people won't use it.


No argument here. :)

Matt

8. Re: ooeu [was Re: Proposal for math.e (2007-08-23)]

• Posted by Matt Lewis <matthewwalkerlewis at ?mail?com> Aug 30, 2007
• 572 views
• Last edited Aug 31, 2007

Matt Lewis wrote:
> 
> Kat wrote:
> > 
> > But even loading a disk file into an array already made with repeat(), 
> > the bigger the file, the slower it goes, drastically. I am talking about 
> > a 34megabyte file in this case, but i have files 100x that big.
> 
> That seems odd.  I guess I haven't really worked with a data set that 
> large to notice the progressive slow down.  Is it possibly an issue of 
> swapping memory out?  I'll try to run some tests to see what happens.

OK, I ran a test using regular Euphoria, and saw a progressive decrease in
performance.  I created a text file with 5,000,000 lines that was about
184MB in size, and read it in.  The program ended up with about a gig
of memory by the end, most of it swapped out.

Based on this, it seems like you may need to rethink the way you're doing
this, because using euphoria like this may not be the way to go.  The 
first thing that comes to mind is to look at what Pete's done with Edita.
I believe that he compresses three characters into a single sequence element
(only three so that it never breaks the 30-bit integer barrier and turns
into a double, sucking up 12 additional bytes of memory).

Here were the results, and the code I used:
         100000 0.109
         200000 0.109
         300000 0.11
         400000 0.094
         500000 0.093
         600000 0.109
         700000 0.11
         800000 0.094
         900000 0.094
        1000000 0.11
        1100000 0.094
        1200000 0.093
        1300000 0.11
        1400000 0.109
        1500000 0.093
        1600000 0.094
        1700000 0.11
        1800000 0.094
        1900000 0.109
        2000000 0.093
        2100000 0.094
        2200000 0.11
        2300000 0.094
        2400000 0.094
        2500000 0.313
        2600000 0.407
        2700000 1.672
        2800000 1.719
        2900000 0.422
        3000000 0.375
        3100000 0.609
        3200000 0.422
        3300000 0.438
        3400000 0.313
        3500000 0.25
        3600000 0.938
        3700000 0.5
        3800000 0.329
        3900000 0.344
        4000000 2.421
        4100000 2.625
        4200000 1.922
        4300000 2.672
        4400000 6.797
        4500000 5.157
        4600000 1.985
        4700000 2.781
        4800000 4.109
        4900000 2.75
        5000000 8.281

constant lines = 5000000
integer fout

fout = open( "kat.txt", "r" )
if fout = -1 then
	puts(1, "Writing file\n" )
	fout = open("kat.txt", "w" )
	for i = 1 to lines do
		printf(fout, "This is kat.txt line number %d.\n", i )
		if remainder( i, 100000 ) = 0 then
			puts(1, '.' )
		end if
	end for
end if
close( fout )

puts(1, "\nReading file\n" )
integer fin
fin = open( "kat.txt", "r" )
object in
integer count
count = 1
atom t
sequence file
file = repeat( "", lines )

t = time()
in = gets(fin)

while sequence(in) do
	file[count] = in
	count += 1
	if remainder(count, 100000) = 0 then
		printf(1, "%15d %g\n", {count, time() - t})
		t = time()
	end if
	in = gets(fin)
end while
close(fin)
puts(1, "Done.\n")

10. Re: ooeu [was Re: Proposal for math.e (2007-08-23)]

• Posted by Kat <KAT12 at ?oosahs.net> Aug 30, 2007
• 585 views
• Last edited Aug 31, 2007

Matt Lewis wrote:
> 
> Matt Lewis wrote:
> > 
> > Kat wrote:
> > > 
> > > But even loading a disk file into an array already made with repeat(), 
> > > the bigger the file, the slower it goes, drastically. I am talking about 
> > > a 34megabyte file in this case, but i have files 100x that big.
> > 
> > That seems odd.  I guess I haven't really worked with a data set that 
> > large to notice the progressive slow down.  Is it possibly an issue of 
> > swapping memory out?  I'll try to run some tests to see what happens.
> 
> OK, I ran a test using regular Euphoria, and saw a progressive decrease in
> performance.  I created a text file with 5,000,000 lines that was about
> 184MB in size, and read it in.  The program ended up with about a gig
> of memory by the end, most of it swapped out.
> 
> Based on this, it seems like you may need to rethink the way you're doing
> this, because using euphoria like this may not be the way to go.

Possibly, i agree, and since i upgraded to Eu 3.1 from 2.3, 
and got the latest OOEU v1.9.1,
i ran the tests again, on the data file in question.

>  The 
> first thing that comes to mind is to look at what Pete's done with Edita.
> I believe that he compresses three characters into a single sequence element
> (only three so that it never breaks the 30-bit integer barrier and turns
> into a double, sucking up 12 additional bytes of memory).
> 
> Here were the results, and the code I used:
>          100000 0.109
>          200000 0.109
>          300000 0.11
<snip>
>         4800000 4.109
>         4900000 2.75
>         5000000 8.281

I got similar results.

I modified your code to have a wait_key() when ending, and to load the 
35megabyte file i have. Since your code uses gets(), where i was using 
getc() and then parsing on {10,13} later, i also inserted code to filter 
those out. This gives:

include get.e -- for wait_key()
  object junk -- for wait_key()
 
  constant lines = 5000000
  integer fout
if 1=2 then -- good place for a goto, but then i could not have run it on
exw.exe
  fout = open( "kat.txt", "r" )
  if fout = -1 then
  	puts(1, "Writing file\n" )
  	fout = open("kat.txt", "w" )
  	for i = 1 to lines do
  		printf(fout, "This is kat.txt line number %d.\n", i )
  		if remainder( i, 100000 ) = 0 then
  			puts(1, '.' )
  		end if
  	end for
  end if
  close( fout )
end if

  puts(1, "\nReading file\n" )
  integer fin
  fin = open( "C:\\mine_bbcnews\\newgetter-wget\\newsurls.txt", "r" )
  object in
  integer count
  count = 1
  atom t
  sequence file
  file = repeat( "", lines )
  
  t = time()
  in = gets(fin)
  
  while sequence(in) do
        while match({in[length(in)]},{10,13}) do
          in = in[1..length(in)-1]
        end while
  	file[count] = in
  	count += 1
  	if remainder(count, 100000) = 0 then
  		printf(1, "%15d %g\n", {count, time() - t})
  		t = time()
  	end if
  	in = gets(fin)
  end while
  close(fin)
  puts(1, "Done.\n")
junk = wait_key()

exw.exe v3.1 gives:

Reading file
         100000 1.47
         200000 4.16
         300000 7.98
         400000 12.92
         500000 29.58
Done.

ooeu v1.9.0 gives:

Reading file
         100000 0.625
         200000 0.609
         300000 0.641
         400000 0.578
         500000 0.547
Done.

So OOEU beat the pants off RDS's Eu.

Then i ran the code using getc() in 

include file.e -- for fget()
  include get.e -- for wait_key()
  object junk -- for wait_key()

sequence urllist
atom starttime, endtime

for loop = 1 to 5 do
  starttime = time()
  urllist = getf("c:\\mine_bbcnews\\newgetter-wget\\newsurls.txt")
  endtime = time()

  puts(1,sprintf("%d",endtime - starttime)&"\n")
end for

junk = wait_key()

where fget() is:

global function getf(sequence filename)
object junk, file, filecontents, readbyte

 file = open(filename,"rb")
 if not equal(file,-1) then
   junk = dir(filename) -- get some info about it
   junk = junk[1][3] -- the filesize
   filecontents = repeat(0,junk)
   junk = 0
   readbyte = getc(file)
   while not equal(readbyte,-1) do
     junk += 1
     filecontents[junk] = readbyte
     readbyte = getc(file)
   end while
   close(file)
   return filecontents
 end if
 return -1
end function

and RDS Eu gave me:
4
3
4
4
3

and OOEU gave me:
23
17
15
13
16

and in that case, RDS Eu beat the pants off of OOEU !

So while i was testing, i ran the following:

include get.e -- for wait_key()
  object junk -- for wait_key()
 
  include Strtok-v2-1.e -- for parse()
  sequence parsed
 
  constant lines = 5000000
  integer fout
  atom starttime, endtime

  puts(1, "\nReading file\n" )
  integer fin
  starttime = time()
  fin = open( "C:\\mine_bbcnews\\newgetter-wget\\newsurls.txt", "r" )
  object in
  integer count
  count = 1
  atom t
  sequence file
  file = repeat( "", lines )
  
  t = time()
  in = gets(fin)
  
  while sequence(in) do
  	file[count] = in
  	count += 1
  	in = gets(fin)
  end while
  close(fin)
  endtime = time()
    
  puts(1,sprintf("%d",endtime - starttime)&"\n")
  puts(1, "Done.\n")
  
  for loop = 1 to 5 do
    starttime = time()
    parsed = parse(file,{10,13})
    endtime = time()
    puts(1,sprintf("%d",endtime - starttime)&"\n")
  end for
   
  
junk = wait_key()

RDS Eu gave me:

Reading file
84
Done.
0
1
1
1
0

OOEU gave me:

Reading file
1
Done.
21
20
20
21
21

Well,, in any case, the OOEU times have fallen from what i recall. 
There's the test results. I don't know what's going on....

Kat

12. Re: ooeu [was Re: Proposal for math.e (2007-08-23)]

• Posted by don cole <doncole at pacbell.?e?> Aug 31, 2007
• 621 views

Kat wrote:
> 
<SNIP>
>   
>   while sequence(in) do
>         while match({in[length(in)]},{10,13}) do
>           in = in[1..length(in)-1]
>         end while
>   	file[count] = in
>   	count += 1
>   	if remainder(count, 100000) = 0 then
>   		printf(1, "%15d %g\n", {count, time() - t})
>   		t = time()
>   	end if
>   	in = gets(fin)
>   end while
>   close(fin)
<SNIP>

Hello Kat,

Nice to see you back.

  Not nitpick, (now I proceed to nitpick).

 Since version 2.4 You could change this to:

while sequence(in) do
        while match({in[$]},{10,13}) do
           in = in[$-1]
         end while
   	file[count] = in
   	count += 1
   	if remainder(count, 100000) = 0 then
   		printf(1, "%15d %g\n", {count, time() - t})
   		t = time()
   	end if
   	in = gets(fin)
   end while
   close(fin)

Don Cole

14. Re: ooeu [was Re: Proposal for math.e (2007-08-23)]

• Posted by CChris <christian.cuvier at agriculture.?ouv?fr> Aug 31, 2007
• 625 views

Kat wrote:
> 
> CChris wrote:
> > 
> > Kat wrote:
> > > 
> > > (i hate this idiotic web interface, scrolling back and forth... , 
> > > inserting <cr> everywhere... i have a 17inch monitor, and am typing in a 
> > > little 3x8 box with less text displayed than a 10inch 40column screen... 
> > > copy/pasting to a text editor now.... shirt, i had 80-120col on 
> > > a C64, grrr...)
> > 
> > What is that box? Wonder if I was born when it was built lol (probably)
> 
> I tried replying, Chris, but the reply was caught in moderation 
> here and your private emailbox is over quota.
> 
> Kat

The address above has died two year ago, or is supposed to gave died.
Use oedoc hat free doht fr instead.

CChris