1. memory
- Posted by tone.skoda at gmx.net
Jan 20, 2003
i want to write a utility which will sit in my taskbar all the time.
euphoria program takes a lot of memory, more than c, right? but if i
translated this program with eu2c then it would take less memory, same as
program written in C?
2. Re: memory
You'd still have the Euphoria runtime system, which requires some memory. I
you minimize the use of sequences i guess you could reduce the amount of
memory needed by the program, but if memory usage is a major issue (eg. you
want it to run inside a few kB) then I'd suggest you write the program in C
or assembly.
>but if i translated this program with eu2c then it would take less memory,
>same as program written in C?
3. memory
Hey all,,
I have a short program that opens a 6.8megabyte file, containing text
separated by lots of {0}. The object is to eliminate the {zero}s, and reformat
the results to a more text-looking file. What i can't figure is that Taskinfo
says Eu is using 86Megabytes in memory to do it!
Are these the lines doing it? Is a new instance of data created every time it
is mentioned in the line?:
puts(1,"removing 10 nulls\n")
place = match({0,0,0,0,0,0,0,0,0,0},data)
while place do
data = data[1..place] & data[place+10..length(data)]
place = match({0,0,0,0,0,0,0,0,0,0},data)
end while
Kat
4. Re: memory
> I have a short program that opens a 6.8megabyte file, containing text
> separated by lots of {0}. The object is to eliminate the {zero}s, and
reformat
> the results to a more text-looking file. What i can't figure is that
Taskinfo
> says Eu is using 86Megabytes in memory to do it!
>
> Are these the lines doing it? Is a new instance of data created every time
it
> is mentioned in the line?:
>
> puts(1,"removing 10 nulls\n")
> place = match({0,0,0,0,0,0,0,0,0,0},data)
> while place do
> data = data[1..place] & data[place+10..length(data)]
> place = match({0,0,0,0,0,0,0,0,0,0},data)
> end while
Hi Kat,
I suspect something like that is happening. As a rule, I try never to
re-create an existing sequence because of all the copies that Eu might make
of it.
Try this instead...
puts(1,"removing 10 nulls\n")
place = match({0,0,0,0,0,0,0,0,0,0},data)
dataend = length(data)
while place > 0 and place < dataend do
data[place .. dataend - 10] = data[place+10 .. dataend]
dataend -= 10
place = match({0,0,0,0,0,0,0,0,0,0},data)
end while
data = data[1 .. dataend]
the difference is that this overwrites existing elements in 'data' during
the removal phase and then truncates the extra stuff at the end. It should
also be a lot faster.
----- Original Message -----
From: "Kat" <gertie at PELL.NET>
To: "EUforum" <EUforum at topica.com>
Sent: Thursday, July 12, 2001 5:13 PM
Subject: memory
5. Re: memory
Dear Eu users:
The very interesting question on this subject!
Today, I have updated my site and try to try it.
As you, maybe, know, I prefer the work under the
extremal conditions.
This is 386-25 Mhz 8M RAM machine and Win-95 OS.
Today parametres of my hardware were :
8 Mb only for IE 3.1, + 9.5 Mb system swap-file,
+ 5.2 Mb free space on disk D: (WINDOWS disk on
my machine), so, more than 20Mb of virtual memory.
I can not open my index.html on the server of my
Internet-provaider because of too little available
memory for IE 3.1.
My index.html is just 11332 bytes file.
20971520 bytes are free and 11332 can not be
placed, so, 1 byte of the useful for me info requires
more than 1850 bytes for MS Windows.
1 : 2000 for the round count.
But virtual memory on my machine is *my* property,
not MS's. But MS Windows is property of MS. 
Symbolic 2000, no ?
Regards,
Igor Kachan
kinz at peterlink.ru
6. Re: memory
Howdy y'all!
Igor Kachan wrote:
Igor, I went to your site. I had no problems opening it, and it has
been updated, because it is date July 12.
> This is 386-25 Mhz 8M RAM machine and Win-95 OS.
>
> Today parametres of my hardware were :
> 8 Mb only for IE 3.1, + 9.5 Mb system swap-file,
> + 5.2 Mb free space on disk D: (WINDOWS disk on
> my machine), so, more than 20Mb of virtual memory.
Hum. Well, you know Igor that when you have a machine pushed to its
limit, strange things happen. I think your machine is definitely pushed
to its limit.
> I can not open my index.html on the server of my
> Internet-provaider because of too little available
> memory for IE 3.1.
> My index.html is just 11332 bytes file.
>
> 20971520 bytes are free and 11332 can not be
> placed, so, 1 byte of the useful for me info requires
> more than 1850 bytes for MS Windows.
>
> 1 : 2000 for the round count.
That happened to me also, in Windows 95. I am not sure why.
Considering it is Windows, I am sure that the OS is trying to make that
byte all pretty so you don't send it away. Seriously, there may be
other processes in Windows which are looking at or acting on that byte,
and that could be causing the memory situation. That is just a thought.
>
> But virtual memory on my machine is *my* property,
> not MS's. But MS Windows is property of MS.
>
Virtual memory is *your* property? Don't let Microsoft hear you say
that!
Have a good day, Igor!
Travis Beaty
Claude, Texas.
8. Re: memory
Howdy Kat!
Perhaps it would be better, although of course slower, if you left the
data on your drive till you needed it.
-- untested code
integer iByteIn, iFileHandle
iFileHandle = open("thedata.dat", "r")
if iFileHandle = -1 then
puts(1, "well hell...\n")
return
end if
iByteIn = 0
data = {}
while iByteIn != -1 do
iByteIn = getc(iFileHandle)
if (iByteIn != -1) and (iByteIn != 0) then
data &= iByteIn
end if
end while
close(iFileHandle)
-- end untested code
That would be slower, but it would keep all that data out of memory. Or
maybe ship it in using bite-sized pieces?
Travis Beaty
Claude, Texas.
9. Re: memory
Kat writes :
> Hey all,,
>
> I have a short program that opens a 6.8megabyte file, containing text
> separated by lots of {0}. The object is to eliminate the {zero}s, and
reformat
> the results to a more text-looking file. What i can't figure is that
Taskinfo
> says Eu is using 86Megabytes in memory to do it!
>
> Are these the lines doing it? Is a new instance of data created every time
it
> is mentioned in the line?:
>
> puts(1,"removing 10 nulls\n")
> place = match({0,0,0,0,0,0,0,0,0,0},data)
> while place do
> data = data[1..place] & data[place+10..length(data)]
> place = match({0,0,0,0,0,0,0,0,0,0},data)
> end while
>
>
> Kat
>
Hello Kat,
Wouldn't be a better way to do the job this code?:
include file.e
include get.e
integer fn
fn = open("file.txt","rb")
integer char
sequence result
result = {}
while 1 do
char = getc(fn)
if char = -1 then exit -- EOF
eslif char = 0 then -- ignore
else result &= char -- append to the result
end if
end while
close(fn)
Martin
10. Re: memory
On 12 Jul 2001, at 19:04, martin.stachon at worldonline.cz wrote:
>
> Kat writes :
> > Hey all,,
> >
> > I have a short program that opens a 6.8megabyte file, containing text
> > separated by lots of {0}. The object is to eliminate the {zero}s, and
> reformat
> > the results to a more text-looking file. What i can't figure is that
> Taskinfo
> > says Eu is using 86Megabytes in memory to do it!
> >
> > Are these the lines doing it? Is a new instance of data created every time
> it
> > is mentioned in the line?:
> >
> > puts(1,"removing 10 nulls\n")
> > place = match({0,0,0,0,0,0,0,0,0,0},data)
> > while place do
> > data = data[1..place] & data[place+10..length(data)]
> > place = match({0,0,0,0,0,0,0,0,0,0},data)
> > end while
> >
> >
> > Kat
> >
>
> Hello Kat,
>
> Wouldn't be a better way to do the job this code?:
>
> include file.e
> include get.e
> integer fn
> fn = open("file.txt","rb")
> integer char
> sequence result
> result = {}
> while 1 do
> char = getc(fn)
> if char = -1 then exit -- EOF
> eslif char = 0 then -- ignore
> else result &= char -- append to the result
> end if
> end while
> close(fn)
That way makes 6.8 million file accesses. Eventually, i went to pruning a
little off the front of data in a loop, and processing 50 bytes at a time. The
program eventually was using 120megs of ram and was thrashing the
harddrive for 6 hours, so i killed it off.
I'd still like to know why it wanted so much memory for a little 6.8meg text
file.
Kat
11. Re: memory
- Posted by euman at bellsouth.net
Jul 12, 2001
I had the very same problem only with a 4.8 meg file
and come to find out its because of concat sequences.
and the inability of Euphoria to have line chars past 197
characters.
I think this is the same problem you ran into.
Euman
euman at bellsouth.net
----- Original Message -----
From: "Kat" <gertie at PELL.NET>
To: "EUforum" <EUforum at topica.com>
Sent: Thursday, July 12, 2001 19:30
Subject: Re: memory
>
>
> On 12 Jul 2001, at 19:04, martin.stachon at worldonline.cz wrote:
>
>
> >
> > Kat writes :
> > > Hey all,,
> > >
> > > I have a short program that opens a 6.8megabyte file, containing text
> > > separated by lots of {0}. The object is to eliminate the {zero}s, and
> > reformat
> > > the results to a more text-looking file. What i can't figure is that
> > Taskinfo
> > > says Eu is using 86Megabytes in memory to do it!
> > >
> > > Are these the lines doing it? Is a new instance of data created every time
> > it
> > > is mentioned in the line?:
> > >
> > > puts(1,"removing 10 nulls\n")
> > > place = match({0,0,0,0,0,0,0,0,0,0},data)
> > > while place do
> > > data = data[1..place] & data[place+10..length(data)]
> > > place = match({0,0,0,0,0,0,0,0,0,0},data)
> > > end while
> > >
> > >
> > > Kat
> > >
> >
> > Hello Kat,
> >
> > Wouldn't be a better way to do the job this code?:
> >
> > include file.e
> > include get.e
> > integer fn
> > fn = open("file.txt","rb")
> > integer char
> > sequence result
> > result = {}
> > while 1 do
> > char = getc(fn)
> > if char = -1 then exit -- EOF
> > eslif char = 0 then -- ignore
> > else result &= char -- append to the result
> > end if
> > end while
> > close(fn)
>
> That way makes 6.8 million file accesses. Eventually, i went to pruning a
> little off the front of data in a loop, and processing 50 bytes at a time. The
> program eventually was using 120megs of ram and was thrashing the
> harddrive for 6 hours, so i killed it off.
>
> I'd still like to know why it wanted so much memory for a little 6.8meg text
> file.
>
> Kat
>
>
>
>
12. Re: memory
Kat wrote:
> On 12 Jul 2001, at 19:04, martin.stachon at worldonline.cz wrote:
> >
> > Kat writes :
> > > Hey all,,
> > >
> > > I have a short program that opens a 6.8megabyte file, containing text
> > > separated by lots of {0}. The object is to eliminate the {zero}s, and
> > > reformat the results to a more text-looking file. What i can't figure
> > > is that Taskinfo says Eu is using 86Megabytes in memory to do it!
> > >
> > > Are these the lines doing it?
[snip Kat's code that processes a block at a time]
> > Hello Kat,
> >
> > Wouldn't be a better way to do the job this code?:
[snip Martin's code that processes a byte at a time]
> That way makes 6.8 million file accesses.
Actually it's more like 850 file accesses. Euphoria has a built in 8k
buffer. A couple of years ago I made a suggestion (on the old mailing list)
for Rob C. to make the buffer size variable. He provided a few good reasons
why he [w|c|sh]ouldn't do it, none of which I can remember right now... ;-|
> I'd still like to know why it wanted so much memory for a little 6.8meg
text
> file.
Euphoria stores bytes from a file as integers. Integers take up 4 bytes of
memory apiece. So, excluding sequence overheads, a 6.8Mb file takes up
27.2Mb of memory. To make matters worse, operating on that sequence without
considering the inner workings of Euphoria could double, triple or even
quadruple the amount of memory needed - entirely consistent with the 86Mb
you mentioned earlier in the thread.
HTH,
Carl
--
Carl R White - aka - Cyrek
eMail: carlw at legend.co.uk
cyrek at bigfoot.com
URL: nope none nada zip
13. Re: memory
- Posted by gertie at ad-tek.net
Jul 13, 2001
On 13 Jul 2001, at 10:53, euphoria at carlw.legend.uk.com wrote:
>
> > That way makes 6.8 million file accesses.
>
> Actually it's more like 850 file accesses. Euphoria has a built in 8k
> buffer. A couple of years ago I made a suggestion (on the old mailing list)
> for Rob C. to make the buffer size variable. He provided a few good reasons
> why he [w|c|sh]ouldn't do it, none of which I can remember right now... ;-|
Eu manages it's own disk buffers? Cool,, but if it doesn't,, then it's the OS
doing the
buffering, and that means Eu is still doing 6.8 million file accesses, even if
the OS
buffers the access to the hardware.
Kat
14. Re: memory
Kat wrote:
> On 13 Jul 2001, at 10:53, euphoria at carlw.legend.uk.com wrote:
>
> >
> > > That way makes 6.8 million file accesses.
> >
> > Actually it's more like 850 file accesses. Euphoria has a built in 8k
> > buffer. A couple of years ago I made a suggestion (on the old mailing
list)
> > for Rob C. to make the buffer size variable. He provided a few good
reasons
> > why he [w|c|sh]ouldn't do it, none of which I can remember right now...
;-|
>
> Eu manages it's own disk buffers? Cool,, but if it doesn't,, then it's the
OS doing the
> buffering, and that means Eu is still doing 6.8 million file accesses,
even if the OS
> buffers the access to the hardware.
Whether it's the OS or Euphoria the number of file accesses doesn't really
matter. For 8191 out of every 8192 (or whatever) pseudo-file accesses, the
access is as fast as a memory access rather than a disk access. It should be
just as fast as using a sequence. Admittedly there is the 1 other access to
the file that takes a little longer. :)
Playing with the numbers revealed one of Rob's reasons for not increasing
the buffer size... The slowdown is always 12% (comparing seq->seq and
file->seq) no matter how big the OS/Eu buffer is. Doing your own buffering
may offer a speed increase only because you have _access_ to the buffer with
Euphoria it seems.
Carl
--
Carl R White - aka - Cyrek
eMail: carlw at legend.co.uk
cyrek at bigfoot.com
URL: nope none nada zip
15. Re: memory
Kat writes:
> > Kat writes :
> > > Hey all,,
> > >
> > > I have a short program that opens a 6.8megabyte file, containing text
> > > separated by lots of {0}. The object is to eliminate the {zero}s, and
> > reformat
> > > the results to a more text-looking file. What i can't figure is that
> > Taskinfo
> > > says Eu is using 86Megabytes in memory to do it!
> > >
> > > Are these the lines doing it? Is a new instance of data created every
time
> > it
> > > is mentioned in the line?:
> > >
> > > puts(1,"removing 10 nulls\n")
> > > place = match({0,0,0,0,0,0,0,0,0,0},data)
> > > while place do
> > > data = data[1..place] & data[place+10..length(data)]
> > > place = match({0,0,0,0,0,0,0,0,0,0},data)
> > > end while
> > >
> > >
> > > Kat
> > >
> >
> > Hello Kat,
> >
> > Wouldn't be a better way to do the job this code?:
> >
> > include file.e
> > include get.e
> > integer fn
> > fn = open("file.txt","rb")
> > integer char
> > sequence result
> > result = {}
> > while 1 do
> > char = getc(fn)
> > if char = -1 then exit -- EOF
> > eslif char = 0 then -- ignore
> > else result &= char -- append to the result
> > end if
> > end while
> > close(fn)
>
> That way makes 6.8 million file accesses. Eventually, i went to pruning a
> little off the front of data in a loop, and processing 50 bytes at a time.
The
> program eventually was using 120megs of ram and was thrashing the
> harddrive for 6 hours, so i killed it off.
>
> I'd still like to know why it wanted so much memory for a little 6.8meg
text
> file.
>
> Kat
No, Euphoria doesn't read by 1 byte with getc(). It uses buffer.
(See docs)
Martin
