1. RE: Speed of EDS (was Berkeley DB with Euphoria)
Matthew Lewis wrote:
>
> > From: Robert Craig [mailto:rds at RapidEuphoria.com]
>
> > I recently speeded up my copy of EDS.
> > It reads records twice as fast as before, and it
> > inserts and deletes into huge (100,000 record)
> > databases 3 times as fast. Even the released
> > EDS can insert/delete many times per second
> > on huge databases on a slow machine.
> > If you acquire new records via a human entering
> > data into a GUI, you'll never notice the time.
>
> Ooh, ooh! When can we see it?!
>
Actually, I've been meaning to ask --
When working with very large EDS databases, inserts get slower and
slower, but actually what I'm doing much of the time is:
-- checking to see if the key already exists
-- if not, do the insert
-- if so, grab the current record, *append* to it (making it bigger),
and replacing it.
Is it possible there would actually be less overhead in the second case
if after grabbing an existing record and adding to it, instead of doing
a replace I deleted the original record and then did an insert?
In other words, how expensive is doing a replace on the value?
2. RE: Speed of EDS (was Berkeley DB with Euphoria)
> From: Andy Serpa [mailto:renegade at earthling.net]
> Actually, I've been meaning to ask --
>
> When working with very large EDS databases, inserts get slower and
> slower, but actually what I'm doing much of the time is:
>
> -- checking to see if the key already exists
> -- if not, do the insert
> -- if so, grab the current record, *append* to it (making it bigger),
> and replacing it.
>
> Is it possible there would actually be less overhead in the
> second case
> if after grabbing an existing record and adding to it,
> instead of doing
> a replace I deleted the original record and then did an insert?
>
> In other words, how expensive is doing a replace on the value?
Based on looking at the code in EDS, it looks like you're better off doing a
replace than a delete and insert. The delete and insert method has to do
two updates to the key pointers, update the number of records, etc, while a
replace avoids all this. Also, if you can keep the data the same size, EDS
will reuse the same location for the data, rather than freeing and
allocating new space. It looks like it has to be within 8 bytes of the
original (from db_replace_data):
if new_size <= old_size and
new_size >= old_size - 8 then
-- keep the same data block
Not knowing what sort of data you're using, perhaps you could preallocate a
large chunk of data (a really big sequence) so that you have some breathing
room. Then, just replace the padding with your data as you go. You could
possibly figure out how often you need to increase the size of a record by
adding padding. Also, you could take out the
new_size >= old_size - 8 then
part of the test to allow for smaller records to always use the same space.
This is space inefficient, but could save you a lot of time, plus you could
get rid of the padding when you replace, since you don't have to be so
exact.
Matt Lewis
3. RE: Speed of EDS (was Berkeley DB with Euphoria)
On 30 Sep 2002, at 9:21, Matthew Lewis wrote:
> Based on looking at the code in EDS, it looks like you're better off doing a
> replace than a delete and insert. The delete and insert method has to do two
> updates to the key pointers, update the number of records, etc, while a
> replace
> avoids all this. Also, if you can keep the data the same size, EDS will reuse
> the same location for the data, rather than freeing and allocating new space.
> It looks like it has to be within 8 bytes of the original (from
> db_replace_data):
>
> if new_size <= old_size and
> new_size >= old_size - 8 then
> -- keep the same data block
Does this mean if you have large data, which continually shrinks by 7 bytes
over time, you will eventually end up never shrinking the database, and
haveing the original size totally consumed by the padding?
Kat
