1. RE: high file count in application
Kat wrote:
<snip>
> ...just *one* subdirectory built last year has 414 megabytes in
> 118,306 files,, a bit more than EDS can deal with in ram.
> Loading that as one EDS file isn't possible...
That's probably more than most of us would ever want [for one process to
have] in RAM, period. However, I don't see why EDS couldn't handle this
much data--by which I mean, it's possible, if not very efficient. Since
EDS uses 32-bit pointers, a file can grow to 2Gb (some of that will be
overhead, of course). The whole EDS file doesn't get loaded just by
opening. That's kinda the point of a database.
Based on your figures, it looks like the files average about 3.6K per
file. I'm curious (and I'm sure the original poster is, too), how are
you using the data, Kat? What do the files look like? How do you
access them?
Matt Lewis
2. RE: high file count in application
On 1 Dec 2003, at 19:34, Matt Lewis wrote:
>
>
> Kat wrote:
>
> <snip>
>
> > ...just *one* subdirectory built last year has 414 megabytes in
> > 118,306 files,, a bit more than EDS can deal with in ram.
> > Loading that as one EDS file isn't possible...
>
> That's probably more than most of us would ever want [for one process to
> have] in RAM, period. However, I don't see why EDS couldn't handle this
> much data--by which I mean, it's possible, if not very efficient. Since
> EDS uses 32-bit pointers, a file can grow to 2Gb (some of that will be
> overhead, of course). The whole EDS file doesn't get loaded just by
> opening. That's kinda the point of a database.
So EDS won't load a 2 gig file into memory, even assuming there is 2 gig of
ram? I know the OS will page it out to virtual ram on a harddrive, but that
gets really really slow.
Anyhow, 2 gig still isn't enough. The dictionary/translator database is 2 gigs
by itself (altho i could prolly clean it up, down to 1.5gigs), the world news
thrown in for pretty reading in IE, otherwise it's plain text), and the human-
readable general db is a lil over 4 gig of assorted text filetypes (no exe,
music, or videos!, only .txt, .htm, .pdf, .ps, .zip, etc).
> Based on your figures, it looks like the files average about 3.6K per
> file. I'm curious (and I'm sure the original poster is, too), how are
> you using the data, Kat? What do the files look like? How do you
> access them?
Some files are 3 megabytes of plain text. How i access them depends on
the file type. I had started this road for Ai work in Tiggr, but as no one else
in
the world was interested, and when such things in Tiggr were switched on,
people abused them, so i am merely logging the data. Someday maybe i
can find people willing to work with me, and i can pick up where i left off 8
years ago. Originally, getxml() in early versions of strtok.e did most of the
hard work in reading the sgml files.
Kat,
watching her brain rot from non-use.
