1. Your program has run out of memory
- Posted by ghaberek (admin) Mar 10, 2012
- 1460 views
couldn't alloc 198532 bytes C:\Euphoria\include\std\eds.e:2756 in procedure db_replace_recid() Your program has run out of memory. One moment please...
Oops! I was up to about 5,130 records in one table, about 1,350 in another, and maybe a few dozen in 2-3 other tables when it topped out at about 2 GB of memory (32-bit limitation?). The database is only ~600 KB so it must be something in the code. I'm basically looping through CSV files and dumping the data into a few different tables with EuSQL. Even if I have 8,000 records, that's about 250 KB per record in memory. Does that seem right? Any ideas where my memory is going? (har har)
-Greg
2. Re: Your program has run out of memory
- Posted by mattlewis (admin) Mar 10, 2012
- 1428 views
couldn't alloc 198532 bytes C:\Euphoria\include\std\eds.e:2756 in procedure db_replace_recid() Your program has run out of memory. One moment please...
Oops! I was up to about 5,130 records in one table, about 1,350 in another, and maybe a few dozen in 2-3 other tables when it topped out at about 2 GB of memory (32-bit limitation?). The database is only ~600 KB so it must be something in the code. I'm basically looping through CSV files and dumping the data into a few different tables with EuSQL. Even if I have 8,000 records, that's about 250 KB per record in memory. Does that seem right? Any ideas where my memory is going? (har har)
Difficult to say. There was a report of possible memory leaks recently, but I haven't seen any of the details, so I can't really evaluate that.
It doesn't sound like you had a really long sequence. When doing stuff, it's common for a sequence to be required to be copied in memory, at least temporarily. Due to fragmentation of the memory space, it's possible that there was no contiguous block big enough to handle whatever you're doing.
Does this error happen consistently?
Can you run the program on Linux? If so, you might try translating it (in debug mode) and running it under valgrind. That's how I generally track down memory leaks. Technically, you can do it while interpreting, but the references to the code will be in the interpreter without a way to trace that back to a line of code in your program.
Matt
3. Re: Your program has run out of memory
- Posted by DerekParnell (admin) Mar 10, 2012
- 1429 views
Does that seem right? Any ideas where my memory is going? (har har)
No and No.
At first glance, I can't see anything in eds.e that would be a problem, but I guess as I'm current working on some enhancements to EDS I better create a ticket and see what I can find.
4. Re: Your program has run out of memory
- Posted by ghaberek (admin) Mar 10, 2012
- 1434 views
Can you run the program on Linux?
Just did that and I stopped it manually at 50% memory usage (~1.8 GB). It made it up to over 5,600 records in the main table this time, but otherwise the memory consumption is still present and still about 250 KB per record. Feel free to download and run it yourself. I'm just using FOIA Export-Import data obtained from data.gov. It will display every 10th record. Some stuff is commented out for testing - no point in processing all of the files if the first one mucks up the works.
Download: foia_deals.zip (6.92 MB)
Example output:
> eui foia_import.ex
----------------------------------------
participant_id = 10
unique_identifier = '07083927XX0002'
deal_number = 'AP083927XX'
role_id = 8 ('Supplier')
company_id = 5 ('General Electric Company')
-Greg
5. Re: Your program has run out of memory
- Posted by mattlewis (admin) Mar 10, 2012
- 1373 views
> eui foia_import.ex
----------------------------------------
participant_id = 10
unique_identifier = '07083927XX0002'
deal_number = 'AP083927XX'
role_id = 8 ('Supplier')
company_id = 5 ('General Electric Company')
Thanks! I ran it here using 4.0.3 on a 64-bit machine with 8GB RAM, and it died very close to 4GB:
participant_id = 8700
unique_identifier = '08489060XX0001'
deal_number = '489060'
role_id = 9 ('Insured')
company_id = 2343 ('General Electric Capital Corporation')
Watching the memory consumption, this definitely looks like there's a leak happening. I should be able to find it, but will require some detective work.
Matt
6. Re: Your program has run out of memory
- Posted by mattlewis (admin) Mar 10, 2012
- 1370 views
Watching the memory consumption, this definitely looks like there's a leak happening. I should be able to find it, but will require some detective work.
I translated it, and it ran much faster. I stopped after about 12,000 records. The memory usage was very flat, so it looks like this issue is confined to the interpreter.
Matt
7. Re: Your program has run out of memory
- Posted by ghaberek (admin) Mar 11, 2012
- 1324 views
I translated it, and it ran much faster. I stopped after about 12,000 records. The memory usage was very flat, so it looks like this issue is confined to the interpreter.
On my Windows box, compiled with Watcom, the memory consumption is about 10% compared to the interpreter, but otherwise it increases at the same rate (~25 KB per record versus ~250 KB). I stopped at 5,600 records using ~180 MB of memory. I'm still trying to get the confounded thing to even compile on Linux.
sprint.o: In function `_36pretty_sprint': sprint.c:(.text+0x9ac): undefined reference to `binary_op_a' sprint.c:(.text+0x9dd): undefined reference to `rhs_slice_target' sprint.c:(.text+0x9f2): undefined reference to `RHS_Slice' sprint.c:(.text+0xa04): undefined reference to `de_reference' sprint.o: In function `_36SprintEuString': sprint.c:(.text+0x158): undefined reference to `de_reference' sprint.o: In function `_36Sprint': sprint.c:(.text+0x5d4): undefined reference to `de_reference' collect2: ld returned 1 exit status
-Greg
8. Re: Your program has run out of memory
- Posted by jimcbrown (admin) Mar 11, 2012
- 1340 views
I translated it, and it ran much faster. I stopped after about 12,000 records. The memory usage was very flat, so it looks like this issue is confined to the interpreter.
On my Windows box, compiled with Watcom, the memory consumption is about 10% compared to the interpreter, but otherwise it increases at the same rate (~25 KB per record versus ~250 KB).
I wonder how MinGW would fare.
I'm still trying to get the confounded thing to even compile on Linux.
sprint.o: In function `_36pretty_sprint': sprint.c:(.text+0x9ac): undefined reference to `binary_op_a' sprint.c:(.text+0x9dd): undefined reference to `rhs_slice_target' sprint.c:(.text+0x9f2): undefined reference to `RHS_Slice' sprint.c:(.text+0xa04): undefined reference to `de_reference' sprint.o: In function `_36SprintEuString': sprint.c:(.text+0x158): undefined reference to `de_reference' sprint.o: In function `_36Sprint': sprint.c:(.text+0x5d4): undefined reference to `de_reference' collect2: ld returned 1 exit status
-Greg
That looks like it can't find eu.a (the runtime library).
9. Re: Your program has run out of memory
- Posted by ghaberek (admin) Mar 11, 2012
- 1313 views
I wonder how MinGW would fare.
Pretty much the same, about 25 KB per record, give-or-take some rounding.
That looks like it can't find eu.a (the runtime library).
You are correct sir! A quick "-lib /usr/lib/eu.a" and I'm in business! 
-Greg
10. Re: Your program has run out of memory
- Posted by ghaberek (admin) Mar 11, 2012
- 1368 views
I've added a function called process_memory_usage() so I can show the current memory usage with each record dump. I'm not too confident on my method for Linux (reading from /proc/self/stat) so we'll see how this goes...
Download: foia_deals.zip (6.92 MB)
----------------------------------------
memory_usage = 2,031,880 KB
participant_id = 4920
unique_identifier = '08085176XX0001'
deal_number = 'AP085176XX'
role_id = 8 ('Supplier')
company_id = 1298 ('Ergytech Inc')
couldn't alloc 256 bytes
C:\Euphoria\include\std\eds.e:369 in function decompress()
Your program has run out of memory.
One moment please...
-Greg
