RE: high file count in application

new topic     » goto parent     » topic index » view thread      » older message » newer message

On 1 Dec 2003, at 19:34, Matt Lewis wrote:

> 
> 
> Kat wrote:
> 
> <snip>
> 
> > ...just *one* subdirectory built last year has 414 megabytes in 
> > 118,306 files,, a bit more than EDS can deal with in ram. 
> > Loading that as one EDS file isn't possible...
> 
> That's probably more than most of us would ever want [for one process to 
> have] in RAM, period.  However, I don't see why EDS couldn't handle this 
> much data--by which I mean, it's possible, if not very efficient.  Since 
> EDS uses 32-bit pointers, a file can grow to 2Gb (some of that will be 
> overhead, of course).  The whole EDS file doesn't get loaded just by 
> opening.  That's kinda the point of a database.

So EDS won't load a 2 gig file into memory,  even assuming there is 2 gig of 
ram? I know the OS will page it out to virtual ram on a harddrive, but that 
gets really really slow.

Anyhow, 2 gig still isn't enough. The dictionary/translator database is 2 gigs 
by itself (altho i could prolly clean it up, down to 1.5gigs), the world news 
thrown in for pretty reading in IE, otherwise it's plain text), and the human-
readable general db is a lil over 4 gig of assorted text filetypes (no exe, 
music, or videos!, only .txt, .htm, .pdf, .ps, .zip, etc).

> Based on your figures, it looks like the files average about 3.6K per 
> file.  I'm curious (and I'm sure the original poster is, too), how are 
> you using the data, Kat?  What do the files look like?  How do you 
> access them?  

Some files are 3 megabytes of plain text. How i access them depends on 
the file type. I had started this road for Ai work in Tiggr, but as no one else
in
the world was interested, and when such things in Tiggr were switched on, 
people abused them, so i am merely logging the data. Someday maybe i 
can find people willing to work with me, and i can pick up where i left off 8 
years ago. Originally, getxml() in early versions of strtok.e did most of the 
hard work in reading the sgml files.

Kat,
watching her brain rot from non-use.

new topic     » goto parent     » topic index » view thread      » older message » newer message

Search



Quick Links

User menu

Not signed in.

Misc Menu