1. out of memory, even tho remembered nothing!
- Posted by useless_ Oct 17, 2012
- 1676 views
See :
http://openeuphoria.org/pastey/159.wc for the source code
http://openeuphoria.org/pastey/160.wc for the err file.
Basically, open a huge file, gets() on it till done or crash. It crashed.
Windoze's Task Manager said eui.exe is using only 1,456 KB of memory. Nothing is stored, the length of the line gets()'d is added to the running total, and everything else is not remembered. Running out of memory seems a lil odd then.
So how does this work?
For what it's worth, i used seek() on a file nearly as big, and it did not crash.
useless
2. Re: out of memory, even tho remembered nothing!
- Posted by DerekParnell (admin) Oct 18, 2012
- 1582 views
What happens when you change the program to either of these two alternatives ...
Alternative #1
include std/filesys.e -- for move, del, create, dir include std/console.e -- any_key() atom size = 0 with trace sequence readfilename = "E:\\YAGO\\yago2core_20120109\\using.tsv" puts(1,"filesize actually read in with gets()\n\n") object junk, junk1 , junk2 , junk3, readfile -- odd something else: winders says : 7.28 GB 7,823,038,472 bytes) -- when the app crashed, it printed out : size = 9009698613 -- how would it read 1.5 Gbytes past the end of the file? readfile = open(readfilename,"rb") while 1 do junk = gets(readfile) size += length(junk) -- out of memory here, size = 9009698613 junk = 0 puts(1,sprintf("%d",size)&"\r") end while close(readfile) puts(1,"\nwell, if i got here, there was no crash!\n\n") any_key() abort(0)
Alternative #2
include std/filesys.e -- for move, del, create, dir include std/console.e -- any_key() atom size = 0 with trace sequence readfilename = "E:\\YAGO\\yago2core_20120109\\using.tsv" puts(1,"filesize actually read in with gets()\n\n") procedure do_it() object junk, junk1 , junk2 , junk3, readfile -- odd something else: winders says : 7.28 GB 7,823,038,472 bytes) -- when the app crashed, it printed out : size = 9009698613 -- how would it read 1.5 Gbytes past the end of the file? readfile = open(readfilename,"rb") while 1 do junk = gets(readfile) size += length(junk) -- out of memory here, size = 9009698613 puts(1,sprintf("%d",size)&"\r") end while close(readfile) end procedure do_it() puts(1,"\nwell, if i got here, there was no crash!\n\n") any_key() abort(0)
3. Re: out of memory, even tho remembered nothing!
- Posted by CoJaBo2 Oct 18, 2012
- 1599 views
while 1 do junk = gets(readfile) size += length(junk) -- out of memory here, size = 9009698613 puts(1,sprintf("%d",size)&"\r") end while
gets() returns an atom on EOF; since the check for this is missing, it keeps trying to read indefinately after the end of the file.
One thing strikes me as odd here- It keeps running because... length(-1) == 1; is this intentional? I thought it used to print something to the effect of "length of an atom is undefined" (which would have caught this bug).
The out of memory condition, though, definately sounds like an interpreter memory leak; memory is not being freed somewhere, either when the EOF error condition is hit in gets() or when the length() of an atom is read. Since the code forms an (unintentional) infinite loop, it eventually runs out of memory. To reproduce it (much) faster, just run this same code on an empty file.
4. Re: out of memory, even tho remembered nothing!
- Posted by jimcbrown (admin) Oct 18, 2012
- 1691 views
One thing strikes me as odd here- It keeps running because... length(-1) == 1; is this intentional? I thought it used to print something to the effect of "length of an atom is undefined"
This was an enhancement to 4.0, to remove countless "if sequence() then ... length()" checks.
5. Re: out of memory, even tho remembered nothing!
- Posted by mattlewis (admin) Oct 18, 2012
- 1827 views
See :
http://openeuphoria.org/pastey/159.wc for the source code
http://openeuphoria.org/pastey/160.wc for the err file.
Basically, open a huge file, gets() on it till done or crash. It crashed.
Windoze's Task Manager said eui.exe is using only 1,456 KB of memory. Nothing is stored, the length of the line gets()'d is added to the running total, and everything else is not remembered. Running out of memory seems a lil odd then.
So how does this work?
I ran this against a file here and I definitely saw a memory leak. I think the problem is in the call to gets(). We allocate a sequence at the beginning, but if we've hit EOF, it doesn't look like we ever free it. Actually, I think I've been chasing this for the last week or so, and this was the clue that I needed!
See ticket:799.
Matt
6. Re: out of memory, even tho remembered nothing!
- Posted by useless_ Oct 18, 2012
- 1645 views
One thing strikes me as odd here- It keeps running because... length(-1) == 1; is this intentional? I thought it used to print something to the effect of "length of an atom is undefined"
This was an enhancement to 4.0, to remove countless "if sequence() then ... length()" checks.
So why can't i do this any more?
readfile = open(readfilename,"rb") junk = gets(readfile) while junk do size += length(junk) puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
It fails to run because junk isn't an atom. But then, if it was an atom, it would be -1, and not execute the while loop. So i still must do the type checks on it.
useless
7. Re: out of memory, even tho remembered nothing!
- Posted by jimcbrown (admin) Oct 18, 2012
- 1613 views
One thing strikes me as odd here- It keeps running because... length(-1) == 1; is this intentional? I thought it used to print something to the effect of "length of an atom is undefined"
This was an enhancement to 4.0, to remove countless "if sequence() then ... length()" checks.
So why can't i do this any more?
readfile = open(readfilename,"rb") junk = gets(readfile) while junk do size += length(junk) puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
It fails to run because junk isn't an atom. But then, if it was an atom, it would be -1, and not execute the while loop. So i still must do the type checks on it.
8. Re: out of memory, even tho remembered nothing!
- Posted by jaygade Oct 18, 2012
- 1561 views
So why can't i do this any more?
readfile = open(readfilename,"rb") junk = gets(readfile) while junk do size += length(junk) puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
It fails to run because junk isn't an atom. But then, if it was an atom, it would be -1, and not execute the while loop. So i still must do the type checks on it.
useless
Wouldn't this hang as an infinite loop if junk is -1 (a true value) anyway?
http://openeuphoria.org/docs/lang_loop.html#_156_whilestatement
9. Re: out of memory, even tho remembered nothing!
- Posted by jimcbrown (admin) Oct 18, 2012
- 1575 views
So why can't i do this any more?
readfile = open(readfilename,"rb") junk = gets(readfile) while junk do size += length(junk) puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
It fails to run because junk isn't an atom. But then, if it was an atom, it would be -1, and not execute the while loop. So i still must do the type checks on it.
Wouldn't this hang as an infinite loop if junk is -1 (a true value) anyway?
http://openeuphoria.org/docs/lang_loop.html#_156_whilestatement
Yes. I tested this with Euphoria 2.3, and a "while -1 do" loop will loop forever even with that old interpreter.
10. Re: out of memory, even tho remembered nothing!
- Posted by useless_ Oct 18, 2012
- 1692 views
So why can't i do this any more?
readfile = open(readfilename,"rb") junk = gets(readfile) while junk do size += length(junk) puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
It fails to run because junk isn't an atom. But then, if it was an atom, it would be -1, and not execute the while loop. So i still must do the type checks on it.
Wouldn't this hang as an infinite loop if junk is -1 (a true value) anyway?
http://openeuphoria.org/docs/lang_loop.html#_156_whilestatement
Yes. I tested this with Euphoria 2.3, and a "while -1 do" loop will loop forever even with that old interpreter.
I have previously considered any value (whether bit byte, char, string, etc) over 0 as TRUE and any value of zero or less to be FALSE. I refer you to:
http://i53.tinypic.com/x6c6fd.jpg
useless
11. Re: out of memory, even tho remembered nothing!
- Posted by jaygade Oct 18, 2012
- 1643 views
I have previously considered any value (whether bit byte, char, string, etc) over 0 as TRUE and any value of zero or less to be FALSE. I refer you to:
http://i53.tinypic.com/x6c6fd.jpg
useless
Fair enough, but none of the languages I've ever used (BASIC, C, Euphoria) have behaved like that.
Not that I'm an expert, because I'm not.
Edit: I forget - did Euphoria prior to 4.0 do that?
12. Re: out of memory, even tho remembered nothing!
- Posted by jimcbrown (admin) Oct 18, 2012
- 1645 views
I have previously considered any value (whether bit byte, char, string, etc) over 0 as TRUE and any value of zero or less to be FALSE. I refer you to:
http://i53.tinypic.com/x6c6fd.jpg
Fair enough, but none of the languages I've ever used (BASIC, C, Euphoria) have behaved like that.
Not that I'm an expert, because I'm not.
I think there's a good case for -1 being the same as true (since in two bits complement, -1 is the same as having all bits set to 1, whereas the integer 1 is simply a single bit set to 1 and the rest set to 0).
Edit: I forget - did Euphoria prior to 4.0 do that?
As I stated previously in this thread, no.
13. Re: out of memory, even tho remembered nothing!
- Posted by useless_ Oct 18, 2012
- 1573 views
Well, anyhow, this fails to run properly also:
junk = gets(readfile) while sequence(junk) do size += length(junk) puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
I looked in io.e for what EOF is when gets() hits the end of the file:
From docs on gets() -
and from io.e -
And so i am still tripping over typecasting. To me, -1 is not a sequence, nor should it be TRUE. To avoid the horrors like this, the following convoluted while must be used:
junk = gets(readfile) while not equal(-1,junk) do size += length(junk) puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
If it's voteable, i vote for replacing "while not equal(-1,junk) do" with "while junk do".
Anyhoo, this is getting off my original point, and i am glad Matt is fixing the memory leak in gets().
useless
14. Re: out of memory, even tho remembered nothing!
- Posted by mattlewis (admin) Oct 18, 2012
- 1572 views
Well, anyhow, this fails to run properly also:
junk = gets(readfile) while sequence(junk) do size += length(junk) puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
This can be shortened slightly using a feature introduced in 4.0:
while sequence(junk) with entry do size += length(junk) puts(1,sprintf("%d",size)&"\r") entry junk = gets(readfile) end while
Execution of the loop always starts at entry, skipping the condition the first time through.
And so i am still tripping over typecasting. To me, -1 is not a sequence, nor should it be TRUE. To avoid the horrors like this, the following convoluted while must be used:
junk = gets(readfile) while not equal(-1,junk) do size += length(junk) puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
Yes, I typically check for a sequence, since you only get an atom returned from gets() after you've read all of the file.
If it's voteable, i vote for replacing "while not equal(-1,junk) do" with "while junk do".
Anyhoo, this is getting off my original point, and i am glad Matt is fixing the memory leak in gets().
Yeah, this is related to the long debated equals() vs = reform. But I don't see changing the truth value of atoms changing (the sequence as a boolean / truth value is more likely).
Matt
15. Re: out of memory, even tho remembered nothing!
- Posted by useless_ Oct 18, 2012
- 1590 views
Well, anyhow, this fails to run properly also:
junk = gets(readfile) while sequence(junk) do size += length(junk) puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
And then.........
Yes, I typically check for a sequence, since you only get an atom returned from gets() after you've read all of the file.
But that's my point in that last post, testing for junk being a SEQUENCE doesn't work. EOF is defined as a CONSTANT, and those are defined as OBJECTs, ergo gets() does not return an ATOM.
Yeah, this is related to the long debated equals() vs = reform. But I don't see changing the truth value of atoms changing (the sequence as a boolean / truth value is more likely).
I've defaulted to use equal() now, the debate has been going on over 10 years. Fact is, i'd wrap the boolean expression in parentheses anyhow, so leaving out the = and writing "equal" simply makes it stand out as not an assignment.
useless
16. Re: out of memory, even tho remembered nothing!
- Posted by jimcbrown (admin) Oct 18, 2012
- 1643 views
Well, anyhow, this fails to run properly also:
junk = gets(readfile) while sequence(junk) do size += length(junk) puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
I just tried this, and it appears to work correctly for me. It gets the number of bytes in the file and reports it correctly, and then quits.
What behavior is it exhibiting for you that causes you to declare that it has failed?
I looked in io.e for what EOF is when gets() hits the end of the file:
From docs on gets() -
and from io.e -
You already know this, but just so other readers of this thread are clear: gets() returns a value that is equal to the public constant EOF in io.e, but it doesn't actually use that constant. It merely returns -1.
And so i am still tripping over typecasting.
I don't see how.
To me, -1 is not a sequence,
Agreed. When I run your code, it seems to confirm that this is the case (that -1 is not a sequence).
nor should it be TRUE.
I disagree. -1 should be TRUE since -1 is the set of bits such that all bits are equal to 1.
To avoid the horrors like this, the following convoluted while must be used:
junk = gets(readfile) while not equal(-1,junk) do size += length(junk) puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
Avoid horrors like what? Why must this be used, instead of your original code above?
If it's voteable, i vote for replacing "while not equal(-1,junk) do" with "while junk do".
Same here. I'm sure we could enhance the parser so that, if gets() is being called inside an if or while statement (or something akin to that) it returns 0 on EOF instead of -1, as well as making any sequence from gets() counted as a TRUE value.
I'd hate to see -1 become a FALSE value, though.
17. Re: out of memory, even tho remembered nothing!
- Posted by jimcbrown (admin) Oct 18, 2012
- 1575 views
But that's my point in that last post, testing for junk being a SEQUENCE doesn't work. EOF is defined as a CONSTANT, and those are defined as OBJECTs, ergo gets() does not return an ATOM.
I tried this, and it works for me. gets() returns an atom when it hits the end-of-file. The constant also passes the atom() test:
if atom(EOF) then puts(1, "true\n") else puts(1, "false\n") end if
18. Re: out of memory, even tho remembered nothing!
- Posted by DerekParnell (admin) Oct 18, 2012
- 1579 views
But that's my point in that last post, testing for junk being a SEQUENCE doesn't work. EOF is defined as a CONSTANT, and those are defined as OBJECTs, ergo gets() does not return an ATOM.
I suspect that you have misunderstood the concept of OBJECT. The gets function returns an object. When that object contains a sequence, it's the read data and when that object contains an atom it signifies end-of-file. The CONSTANT called EOF is defined as an OBJECT with the ATOM value of -1. OBJECTs are containers that can contain either atoms or sequences.
The code you posted as not working will not work if 'junk' is defined as a sequence variable, but if defined as an object then it will work. The following should be working correctly ...
include std/console.e object junk junk = gets(readfile) while sequence(junk) do size += length(junk) display(size) --> simplier than ... puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
19. Re: out of memory, even tho remembered nothing!
- Posted by DerekParnell (admin) Oct 18, 2012
- 1536 views
I have previously considered any value (whether bit byte, char, string, etc) over 0 as TRUE and any value of zero or less to be FALSE.
The true situation is that FALSE is implemented as the value zero and TRUE is implemented as NOT FALSE, which happens to be any number that is not zero.
20. Re: out of memory, even tho remembered nothing!
- Posted by useless_ Oct 18, 2012
- 1582 views
Well, anyhow, this fails to run properly also:
junk = gets(readfile) while sequence(junk) do size += length(junk) puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
I just tried this, and it appears to work correctly for me. It gets the number of bytes in the file and reports it correctly, and then quits.
What behavior is it exhibiting for you that causes you to declare that it has failed?
It read a gigabyte past the end of the file again. Eu version 4.0.4.
useless
21. Re: out of memory, even tho remembered nothing!
- Posted by useless_ Oct 18, 2012
- 1547 views
But that's my point in that last post, testing for junk being a SEQUENCE doesn't work. EOF is defined as a CONSTANT, and those are defined as OBJECTs, ergo gets() does not return an ATOM.
I suspect that you have misunderstood the concept of OBJECT. The gets function returns an object. When that object contains a sequence, it's the read data and when that object contains an atom it signifies end-of-file. The CONSTANT called EOF is defined as an OBJECT with the ATOM value of -1. OBJECTs are containers that can contain either atoms or sequences.
The code you posted as not working will not work if 'junk' is defined as a sequence variable, but if defined as an object then it will work. The following should be working correctly ...
include std/console.e object junk junk = gets(readfile) while sequence(junk) do size += length(junk) display(size) --> simplier than ... puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
I had declared junk as an object in my code! I default to object a lot actually, just to avoid typecasting errors. It read past the end of file again.
useless
22. Re: out of memory, even tho remembered nothing!
- Posted by mattlewis (admin) Oct 18, 2012
- 1538 views
Well, anyhow, this fails to run properly also:
junk = gets(readfile) while sequence(junk) do size += length(junk) puts(1,sprintf("%d",size)&"\r") junk = gets(readfile) end while
And then.........
Yes, I typically check for a sequence, since you only get an atom returned from gets() after you've read all of the file.
But that's my point in that last post, testing for junk being a SEQUENCE doesn't work. EOF is defined as a CONSTANT, and those are defined as OBJECTs, ergo gets() does not return an ATOM.
Yes, -1 is an atom. Yes, gets() returns an atom on EOF. I've used that idiom for years.
Matt
23. Re: out of memory, even tho remembered nothing!
- Posted by jimcbrown (admin) Oct 18, 2012
- 1547 views
It read a gigabyte past the end of the file again. Eu version 4.0.4.
Ah, ok. It seems that I can't reproduce that issue.
24. Re: out of memory, even tho remembered nothing!
- Posted by DerekParnell (admin) Oct 18, 2012
- 1504 views
I had declared junk as an object in my code! I default to object a lot actually, just to avoid typecasting errors. It read past the end of file again.
With respect, may I suggest that you are not running the program code you think you're running.
How about updating the code you think you are running with a special message to identifiy it, and if it is still failing, post the entire code here for someone else to reproduce your issue.
25. Re: out of memory, even tho remembered nothing!
- Posted by useless_ Oct 18, 2012
- 1557 views
- Last edited Oct 19, 2012
I had declared junk as an object in my code! I default to object a lot actually, just to avoid typecasting errors. It read past the end of file again.
With respect, may I suggest that you are not running the program code you think you're running.
How about updating the code you think you are running with a special message to identifiy it, and if it is still failing, post the entire code here for someone else to reproduce your issue.
Yes, i was just getting to that, as it did it again. Going to take some runtime, it's a big file. Praps it's a big-file issue.
Ok, hours later .......
I took a screen shot of the app still running:
http://designerthinking.com/FT-ERROR.JPG
As i am typing this, it's 1.5 Gbytes past the end of file.
The source code is at:
http://openeuphoria.org/pastey/161.wc
Note:
object junk, readfile
This runs properly:
while not equal(-1,junk) do
This does NOT detect EOF:
while sequence(junk) do
useless
26. Re: out of memory, even tho remembered nothing!
- Posted by mattlewis (admin) Oct 19, 2012
- 1494 views
Going to take some runtime, it's a big file. Praps it's a big-file issue.
That sounds likely at this point.
Matt
27. Re: out of memory, even tho remembered nothing!
- Posted by petelomax Oct 19, 2012
- 1524 views
and from io.e -
YUK. I just automatically assumed that said
public constant EOF = {-1}
Yeah, you may need to get the magnifying glass out to see the difference. Whichever plonker put those totally unnecessary () in there should be thoroughly ashamed and publicly humiliated!
Pete
28. Re: out of memory, even tho remembered nothing!
- Posted by useless_ Oct 19, 2012
- 1493 views
and from io.e -
YUK. I just automatically assumed that said
public constant EOF = {-1}
Yeah, you may need to get the magnifying glass out to see the difference. Whichever plonker put those totally unnecessary () in there should be thoroughly ashamed and publicly humiliated!
Pete
I am unsure A) if you are saying that is the fix or not, B) if that is the fix or not, C) if i should risk breaking anything else by changing that. Because i don't know if there is a special way constant handles assignments using ( ).
useless
29. Re: out of memory, even tho remembered nothing!
- Posted by petelomax Oct 25, 2012
- 1419 views
I am unsure
All I know is it should be functionally identical to
public constant EOF = -1
And I'd be very surprised if it wasn't. I was trying to say that the () mislead me into thinking EOF had been defined as a sequence, when actually it is an atom.
Pete
30. Re: out of memory, even tho remembered nothing!
- Posted by DerekParnell (admin) Oct 25, 2012
- 1419 views
And I'd be very surprised if it wasn't. I was trying to say that the () mislead me into thinking EOF had been defined as a sequence, when actually it is an atom.
This is a throwback to a defensive programming practice by C coders. In C, when defining a macro it used to be a good idea to enclose the macro's definition in parenthesis to prevent accidental text appending in the source code.
#define EOF = -1 #define ABC = *2 #define XYZ = EOF ABCwould define 'XYZ' as '-1 *2' which the complier would evaluate as -2.
#define EOF = (-1) #define ABC = (*2) #define XYZ = EOF ABCwould define 'XYZ' as '(-1) (*2)' which the complier would treat as a syntax error.