OpenEuphoria

1. EDB delete multiple records question

• Posted by ags <eu at 531pi.co.nz> Jul 05, 2005
• 610 views

If anyone has had much experience with the EDB system I would like to know
something:

I am keeping a large list of files in an EDB and scan them every few minutes
to add new ones, or remove missing ones.

The adding part is OK, but when it comes to deleting missing files the code
is terribly inefficient (though not too bad as I don't expect lots to go missing
at once...but you never know)

The deleting code snippet is:

-- check missing files
    msg("Checking missing...")
    bad_carts = {}                 --this is what I'm thinking, NOT USED
    go_again = 1 -- prime loop
    while go_again do
            rec_count = db_table_size()
	    for i = 1 to rec_count do
	    	rec = db_record_data(i)
	    	de = dir(rec[TRK_KEY]) -- TRK_KEY is the full path to the file
                                       -- I do store the key as well
	    	if atom(de) then   -- it'a an error code, not a record (sequence)
                                   -- that is, this file is missing or invalid
	    		msg(sprintf("Deleting record %s", {rec[TRK_NAME]}))
	    		db_delete_record(i)
	    		go_again = 1 -- loop again, 
	    		exit         -- since subsequent record numbers are invalid
	    	else
	    	    go_again = 0 -- ok till now, keep loop condition valid
	    	end if
	    end for    	
    end while
    files = db_table_size()
    tt = time() - t0
    msg(sprintf("Done. %d files, %4.2f seconds", {files, tt}))

My question(s) is/are:

Is it possible to use the "bad_cart" sequence to collect bad entries then
delete them all at once (my feeling is not, since you'll still be looping
through them with the same problem of subsequent record no.s being invalid)?

Is this the most efficient way of doing this?  With anything more than a few
hundred missing files this can take upwards of 20 seconds.

I'm not too worried about this since each scan will be 5 mins apart, but it
seems to me to be a missing feature of the EDB system, 
eg db_delete_multiple(sequence_of_ids)

Gary

3. Re: EDB delete multiple records question

• Posted by Robert Craig <rds at RapidEuphoria.com> Jul 05, 2005
• 517 views

ags wrote:
> I'm not too worried about this since each scan will be 5 mins apart, but it
> seems to me to be a missing feature of the EDB system, 
> eg db_delete_multiple(sequence_of_ids)

I can see where deleting a bunch of records at once would
save a bit of overhead, but I don't think it would be a major 
performance win.

Deletion takes longer when a table has a large number of records.
Maybe you could somehow have multiple smaller tables.
Or maybe you could just "mark" records as being deleted, and
perform the actual deletions when you have more time.

Regards,
   Rob Craig
   Rapid Deployment Software
   http://www.RapidEuphoria.com

5. Re: EDB delete multiple records question

• Posted by ags <eu at 531pi.co.nz> Jul 05, 2005
• 505 views

Robert Craig wrote:
> 
> ags wrote:
> > I'm not too worried about this since each scan will be 5 mins apart, but it
> > seems to me to be a missing feature of the EDB system, 
> > eg db_delete_multiple(sequence_of_ids)
> 
> I can see where deleting a bunch of records at once would
> save a bit of overhead, but I don't think it would be a major 
> performance win.
> 
> Deletion takes longer when a table has a large number of records.
> Maybe you could somehow have multiple smaller tables.
> Or maybe you could just "mark" records as being deleted, and
> perform the actual deletions when you have more time.

Deleting in reverse iteration seems to work fine.  That might be worth a
note in the next release though, I would think it a fairly common issue
given the scope and flexibility of the EDB system.

I thought about marking deleted items, but the system will run 24/7 and have
to perform the task every 5 minutes anyway.

Gary

7. Re: EDB delete multiple records question

• Posted by ags <eu at 531pi.co.nz> Jul 05, 2005
• 506 views

Greg Haberek wrote:
> 
> > I thought about marking deleted items, but the system will run 24/7 and h=
> ave
> > to perform the task every 5 minutes anyway.
> 
> What system are you running this on? If you're using Win32Lib, you
> could use the w32HIdle event to perform your 'marked' deletes.
> 
> ~Greg

It shouldn't actually matter as I have written the database updater as a 
separate process (which sends IPC messages to the main program to close
the database/reopen when it has been updated).

Gary