1. EDS - replacing record data slows down database
if you often use db_replace_data() or db_delete_record() database gets
fragmented, i.e it grows and there are a lot of small chunks of unused space, and
consequence is ever growing and ever slower database. you have to use
db_compress() to solve the problem.
solution would be to define maximum size of record keys and data. and then junk
data would be addded to every record. that way db_replace_data() wouldnt need
reallocating and db_insert() would use up whole free chunk.
we would need new functions:
db_insert_ex (object key, object data, integer key_max_size, integer
data_max_size)
or maybe db_set_max_record_size (integer max_record_size)
and db_compressed_object_size (object x) which will help us determinign maximum
size of keys and data.
am i right or am i missing something?
2. Re: EDS - replacing record data slows down database
On Tue, 16 Aug 2005 22:50:49 -0700, Tone =8Akoda
<guest at RapidEuphoria.com> wrote:
>if you often use db_replace_data() or db_delete_record() database gets fra=
gmented,
<snip>
>am i right or am i missing something?
Looks about right to me, the best way to find out for sure is to test
it. Try adding these (untested) routines to database.e (or better,
copying database.e to say edb2.e and modifying that):
global function db_insert_ex (object key, object data, integer
key_max_size, integer data_max_size)
integer l
l=length(compress(key))
key&=repeat(0,key_max_size-l)
l=length(compress(data))
data&=repeat(0,data_max_size-l)
return db_insert(key,data)
end function
global function db_find_key_ex(object key, integer key_max_size)
integer l
l=length(compress(key))
key&=repeat(0,key_max_size-l)
return db_find_key(key)
end function
global procedure db_replace_data_ex(integer rn, object data, integer
data_max_size)
integer l
l=length(compress(data))
data&=repeat(0,data_max_size-l)
db_replace_data(rn, data)
end procedure
global function db_compressed_object_size (object x)
return length(compress(x))
end function
You would also need to ensure your program copes with the excess 0's
on the end, and always getting back a sequence in place of an atom (eg
{1,0,0,0} instead of 1).
I'd be interested to see if this made any gains (in terms of database
size, fragmentation, and rate of performance degradation, rather than
purely in terms of raw speed).
Regards,
Pete
3. Re: EDS - replacing record data slows down database
Pete Lomax wrote:
>
> I'd be interested to see if this made any gains (in terms of database
> size, fragmentation, and rate of performance degradation, rather than
> purely in terms of raw speed).
I tried to create benchmark test but was having problem creating highly
fragmented database where file size (or speed) difference between non-fragmented
and fragmented file would be noticable. i guess test should be made with some
existing program.
but i had an .edb file which was so fragmented that it went from 95 mb to 8 kb
when compressed, and it got that fragmented in only a week or so.
4. Re: EDS - replacing record data slows down database
Tone Škoda wrote:
> I tried to create benchmark test but was having problem creating highly
> fragmented
> database where file size (or speed) difference between non-fragmented and
> fragmented
> file would be noticable. i guess test should be made with some existing
> program.
> but i had an .edb file which was so fragmented that it went from 95 mb to 8 kb
> when
> compressed, and it got that fragmented in only a week or so.
You must have a very interesting pattern of allocating and
freeing space in the database. Maybe you allocate 1Mb, then
free it, then allocate 1K (say), and EDS allocates the 1K from the
space formerly used by the 1Mb, so it only has .999Mb left,
which means it can't be used for your next 1Mb allocation, etc.
Or maybe you allocate/free 1Mb, 1.01 Mb, 1.02 Mb, so you can't use
a previously-freed block, and EDS can't merge the
previously-freed blocks because there are small in-use blocks in between.
This sort of thing somethimes happens with C's malloc and other
allocation systems. I'd be interested to know what
allocation/deallocation pattern you have. Some databases that
I've been updating daily for years, stabilize with just a few percent
of space on the free list. They allocate/free a variety of different
sizes, in a fairly random pattern.
Regards,
Rob Craig
Rapid Deployment Software
http://www.RapidEuphoria.com
5. Re: EDS - replacing record data slows down database
my mistake: using db_dump() i see now that there is one big 95 mb free block. it
looks like i deleted a big table before i used db_compress().
in database of another program i have 1350 free blocks, most of them from 10 to
25 bytes of size. this shouldn't slow down database. for this program i was
suspecting EDS was slowing it down, because sometimes it would take long to do a
task, but now i see something else must be the problem.
Robert Craig wrote:
>
> Tone Škoda wrote:
> > I tried to create benchmark test but was having problem creating highly
> > fragmented
> > database where file size (or speed) difference between non-fragmented and
> > fragmented
> > file would be noticable. i guess test should be made with some existing
> > program.
> > but i had an .edb file which was so fragmented that it went from 95 mb to 8
> > kb when
> > compressed, and it got that fragmented in only a week or so.
>
> You must have a very interesting pattern of allocating and
> freeing space in the database. Maybe you allocate 1Mb, then
> free it, then allocate 1K (say), and EDS allocates the 1K from the
> space formerly used by the 1Mb, so it only has .999Mb left,
> which means it can't be used for your next 1Mb allocation, etc.
> Or maybe you allocate/free 1Mb, 1.01 Mb, 1.02 Mb, so you can't use
> a previously-freed block, and EDS can't merge the
> previously-freed blocks because there are small in-use blocks in between.
> This sort of thing somethimes happens with C's malloc and other
> allocation systems. I'd be interested to know what
> allocation/deallocation pattern you have. Some databases that
> I've been updating daily for years, stabilize with just a few percent
> of space on the free list. They allocate/free a variety of different
> sizes, in a fairly random pattern.
>
> Regards,
> Rob Craig
> Rapid Deployment Software
> <a href="http://www.RapidEuphoria.com">http://www.RapidEuphoria.com</a>
>
