1. RE: Speed of EDS (was Berkeley DB with Euphoria)
- Posted by Andy Serpa <renegade at earthling.net> Sep 30, 2002
- 414 views
Matthew Lewis wrote: > > > From: Robert Craig [mailto:rds at RapidEuphoria.com] > > > I recently speeded up my copy of EDS. > > It reads records twice as fast as before, and it > > inserts and deletes into huge (100,000 record) > > databases 3 times as fast. Even the released > > EDS can insert/delete many times per second > > on huge databases on a slow machine. > > If you acquire new records via a human entering > > data into a GUI, you'll never notice the time. > > Ooh, ooh! When can we see it?! > Actually, I've been meaning to ask -- When working with very large EDS databases, inserts get slower and slower, but actually what I'm doing much of the time is: -- checking to see if the key already exists -- if not, do the insert -- if so, grab the current record, *append* to it (making it bigger), and replacing it. Is it possible there would actually be less overhead in the second case if after grabbing an existing record and adding to it, instead of doing a replace I deleted the original record and then did an insert? In other words, how expensive is doing a replace on the value?
2. RE: Speed of EDS (was Berkeley DB with Euphoria)
- Posted by Matthew Lewis <matthewwalkerlewis at YAHOO.COM> Sep 30, 2002
- 410 views
> From: Andy Serpa [mailto:renegade at earthling.net] > Actually, I've been meaning to ask -- > > When working with very large EDS databases, inserts get slower and > slower, but actually what I'm doing much of the time is: > > -- checking to see if the key already exists > -- if not, do the insert > -- if so, grab the current record, *append* to it (making it bigger), > and replacing it. > > Is it possible there would actually be less overhead in the > second case > if after grabbing an existing record and adding to it, > instead of doing > a replace I deleted the original record and then did an insert? > > In other words, how expensive is doing a replace on the value? Based on looking at the code in EDS, it looks like you're better off doing a replace than a delete and insert. The delete and insert method has to do two updates to the key pointers, update the number of records, etc, while a replace avoids all this. Also, if you can keep the data the same size, EDS will reuse the same location for the data, rather than freeing and allocating new space. It looks like it has to be within 8 bytes of the original (from db_replace_data): if new_size <= old_size and new_size >= old_size - 8 then -- keep the same data block Not knowing what sort of data you're using, perhaps you could preallocate a large chunk of data (a really big sequence) so that you have some breathing room. Then, just replace the padding with your data as you go. You could possibly figure out how often you need to increase the size of a record by adding padding. Also, you could take out the new_size >= old_size - 8 then part of the test to allow for smaller records to always use the same space. This is space inefficient, but could save you a lot of time, plus you could get rid of the padding when you replace, since you don't have to be so exact. Matt Lewis
3. RE: Speed of EDS (was Berkeley DB with Euphoria)
- Posted by Kat <kat at kogeijin.com> Sep 30, 2002
- 435 views
On 30 Sep 2002, at 9:21, Matthew Lewis wrote: > Based on looking at the code in EDS, it looks like you're better off doing a > replace than a delete and insert. The delete and insert method has to do two > updates to the key pointers, update the number of records, etc, while a > replace > avoids all this. Also, if you can keep the data the same size, EDS will reuse > the same location for the data, rather than freeing and allocating new space. > It looks like it has to be within 8 bytes of the original (from > db_replace_data): > > if new_size <= old_size and > new_size >= old_size - 8 then > -- keep the same data block Does this mean if you have large data, which continually shrinks by 7 bytes over time, you will eventually end up never shrinking the database, and haveing the original size totally consumed by the padding? Kat