1. EDS
I looked at Rob's database.e and was quite impressed. I haven't had a =
chance to play around with it yet, but this certainly makes application =
programming much easier. If you could get it to handle simultaneous =
multi-user access (by locking individual records perhaps) and make it a =
standard include in the next version of Euphoria, it would be a very good =
thing.
Michael J. Sabal
mjs at osa.att.ne.jp=20
http://home.att.ne.jp/gold/mjs/index.html=20
ICQ: 32461146
2. Re: EDS
I am very impressed with EDS so far. Simple, extensible - it's a logical
extension of Euphoria.
I have a feeling it will turn out to be a very powerful tool. OK, now for the
wish list:
1) Positional read. I'd like to be able to use "ABCD" as my key for a
POSITIONAL read. The "file
pointer" (or whatever the beastie is) is put just before where key ABCD would
occur in the file,
whether it's there or not.
2) Sequential key read. Along the same lines, I'd like to have a sequential
(probably forward and
backward) read based of the key of the last record read. That way, if I do a
positional read for the
key "JO", I could get all the records whose keys start with JO, and then
logically exit the loop.
-- short example of some code that will display
-- every key that starts with the characters "JO"
object result
sequence mykey
result = db_positional_read("JO")
while 1 do
result = db_read_keysequential()
if atom(result) then
exit
end if
mykey = result[1][1..2]
if compare(mykey, "JO")
exit
end if
puts(1,"This key = " & result[1] & "\n")
end while
-- end code
3) Adding some kind of wildcard match on a key. For instance, if my key was
InvoiceNumber,InvoiceDate
and I wanted to retrieve all the records for a given date, it would be nice to
wildcard match a key.
4) Maybe even a better idea would be to allow multiple indices. This would get
around that problem and
probably be easier to implement.
Well, that's about all I can think of at the moment! It really looks excellent
Rob. Keep up the great work!
Brian Jackson
2FARGON Technologies
bjackson at 2fargon.com
http://www.2fargon.com
3. Re: EDS
- Posted by David Cuny <dcuny at LANSET.COM>
Mar 08, 2000
-
Last edited Mar 09, 2000
I finally got a chance to look at EDS. Pretty clever.
I'm thinking of using EDS as a font server for my DOS window manager. I use
Jiri's fonts (with Pete's rendering code), but it lacks some information,
such as a font's family, name, size, descent, and so on.
If I stored all the fonts in an EDS file, I could determine if a particular
font existed, or find a near replacement in the same family. It also looks
like it'll be easy to extend the font information without having to create a
new file format: the whole thing can be stored in EDS.
Another application of EDS might be to store bitmaps and their palettes. It
could work like a Windows resource file. Have you considered attaching an
EDS file to a bound Euphoria program (like my RES program does)?
Real exciting stuff!
-- David Cuny
4. EDS
------=_NextPart_000_0005_01BF8B38.EEBF3100
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
A few more suggestions:
A compression tool would be great - it could rearrange all the records =
so there are no free blocks and everything is contiguous. A sort of a =
defrag for a database. This would be ideal for when (if) you end up with =
a large collection of tiny blocks of free space that were partially =
filled by another item, but nothing else will be small enough.
Any ideas why the delete takes so long? Twice as long as insert on my =
computer. Does it wipe over all the unused space?
Nick
------=_NextPart_000_0005_01BF8B38.EEBF3100
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content=3D"text/html; charset=3Diso-8859-1" =
http-equiv=3DContent-Type>
<META content=3D"MSHTML 5.00.2614.3500" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV><FONT face=3DArial size=3D2>A few more suggestions:</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>A compression tool would be great - it =
could=20
rearrange all the records so there are no free blocks and everything is=20
contiguous. A sort of a defrag for a database. This would be ideal for =
when (if)=20
you end up with a large collection of tiny blocks of free space that =
were=20
partially filled by another item, but nothing else will be small=20
enough.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>Any ideas why the delete takes so long? =
Twice as=20
long as insert on my computer. Does it wipe over all the unused=20
space?</FONT></DIV>
<DIV> </DIV>
------=_NextPart_000_0005_01BF8B38.EEBF3100--
5. Re: EDS
Nick wrote:
> A compression tool would be great
It seems to me that rewriting all the contents of the file to a new file
would do the trick.
-- David Cuny
6. Re: EDS
Nick Johnson writes:
> A compression tool would be great
I was planning to write one eventually.
If your database tends to grow over time, as most do,
you probably won't have much need for it.
As David points out, you could write one yourself
by creating a new database and copying everything
over to it using db_insert() etc.
> Any ideas why the delete takes so long? Twice as
> long as insert on my computer.
It takes longer for me too.
I haven't tried to figure out why, but it doesn't seem
that unreasonable.
> Does it wipe over all the unused space?
No.
Earlier today I posted a new alpha release of EDS.
It fixes a couple of nasty bugs that showed up during
"stress" testing. The stress test is included.
Also, I've changed a few routines, so you'll have to
make some small changes to any code you've already written.
Regards,
Rob Craig
Rapid Deployment Software
http://www.RapidEuphoria.com
7. Re: EDS
On Fri, 10 Mar 2000, you wrote:
> A few more suggestions:
> A compression tool would be great - it could rearrange all the records so
> there are no free blocks and everything is contiguous. A sort of a defrag for a
> database. This would be ideal for when (if) you end up with a large collection of
> tiny blocks of free space that were partially filled by another item, but nothing
> else will be small enough.
> Any ideas why the delete takes so long? Twice as long as insert on my
> computer. Does it wipe over all the unused space?
I've been trying to decide whether to mention this or not. I don't want to
discourage progress with this new Euphoria database, but there are some
things that really should be considered:
I have been using Euphoria successfully with small databases containing
600-1000 customer records and 2000-3000 A/R transactions. Much more than that,
and the read/write times begin to get annoyingly long. Too long for automatic
updates after each change, which is the way I prefer to do things.
Rob's compact EDS file structure may improve read/write speeds somewhat; I
haven't been able to port any test data yet, so I don't know, but as it stands
this limits the use of Euphoria to only the smallest business apps.
Larger databases, the kind needed by all but my smallest client, need to track
anywhere from a couple thousand to 90,000 customers and their transactions per
month. Most clients also need multi-user capability. The "whole ball of wax"
method (load it all into memory, then write it all back) just won't work in
these situations.
I need the ability to lock individual records, make a change, and write the
change back to disk (quickly) before unlocking the record for someone else to
use. I see no way this can be done without some kind of fixed size record. Too
bad, because otherwise, Euphoria would be the language of choice. It does
everything else quite nicely, and lets me develop apps quickly (always an
important consideration).
Regards,
Irv
8. Re: EDS
- Posted by Robert Craig <rds at ATTCANADA.NET>
Mar 10, 2000
-
Last edited Mar 11, 2000
Irv Mullins writes:
> Rob's compact EDS file structure may improve
> read/write speeds somewhat
It not only uses less disk space (maybe half),
but it's also faster to read and write the compact format.
> The "whole ball of wax" method (load it all into memory,
> then write it all back) just won't work in these situations.
EDS doesn't load the "whole ball of wax".
When you select a new table, it reads in a
4-byte pointer per record in that table.
If you have 1000 records, it will read in 4000 contiguous
bytes from the disk, which is probably the minimum that
the O/S will read, even if you ask for a single byte.
After that, only the records that you
access are read/written to disk. With a binary search for a
key, you might also have to read a few key values before finding
the right record. I started off designing a complicated b-tree
arrangement that didn't require all the pointers to be read in,
but then I realized that this much simpler scheme should be ok
until you get to a really huge number of records (say a million).
I can always change the implementation if people start
creating million record databases.
> I need the ability to lock individual records,
> make a change, and write the change back to disk (quickly)
> before unlocking the record for someone else to use. I see no
> way this can be done without some kind of fixed size record.
Euphoria 2.2 has file and variable-length record locking
(only file locking so far in the case of Linux). I will eventually
add this in to EDS. I don't think that having fully-flexible
record and field lengths will cause a serious problem.
It just makes storage allocation and deallocation more
complicated and somewhat slower.
Regards,
Rob Craig
Rapid Deployment Software
http://www.RapidEuphoria.com
9. Re: EDS
- Posted by Everett Williams <rett at GVTC.COM>
Mar 10, 2000
-
Last edited Mar 11, 2000
Robert Craig wrote:
>I can always change the implementation if people start
>creating million record databases.
>
>> I need the ability to lock individual records,
>> make a change, and write the change back to disk (quickly)
>> before unlocking the record for someone else to use. I see no
>> way this can be done without some kind of fixed size record.
>
>Euphoria 2.2 has file and variable-length record locking
>(only file locking so far in the case of Linux). I will eventually
>add this in to EDS. I don't think that having fully-flexible
>record and field lengths will cause a serious problem.
>It just makes storage allocation and deallocation more
>complicated and somewhat slower.
>
As Euphoria starts to get an industrial strength IDE and windows
library along with a few other odds and ends, why not just go ahead
and do things as if we were headed for that industrial-strength world.
If we get namespace resolution and structures and the other few
odds and ends needed embedded into the language along with an
industrial strength DB and IDE and GUI libs, there will be few limits
to the things that can reasonably be attempted in Euphoria. The
request for a direct C call facility, eliminating the need for define_cfunc and
all that would allow us access to the rest of the world's code.
Maybe the attempt to do these industrial-strength things will point
out the needs that are left in the language.
Everett L.(Rett) Williams
rett at gvtc.com
10. EDS
Why is everyone sitting back and asking for this feature
and that feature to be added to the EDS. The source code is available
so why don't the users that want these feature add them and submit
them to Rob. It is always easier for someone to write the code for
you but you don't learn anything from doing that.
I would rather that Rob would dedicate his valuable time to adding
features to the langauge that would allow us to add are own features.
( namespaces, variable reference, conditional includes, etc. )
Bernie
11. EDS
------=_NextPart_000_000D_01BF8BF8.8C3E32E0
charset="iso-8859-1"
The attached zip contains an upgrade to EDS which provides for having
multiple databases open and choosing the current db with db_select().
Compression is provided by db_compress(), which renames the current db to
<name>.tmp and copies each table and record to a new db with the old name.
I don't know Linux--the system commands in db_compress() may need to be made
platform specific. I haven't updated the documentation. A test program for
compression is provided.
-- Mike Nelson
------=_NextPart_000_000D_01BF8BF8.8C3E32E0
name="Database.zip"
12. Re: EDS
------=_NextPart_000_0025_01BF8C17.BCE00E40
charset="iso-8859-1"
----- Original Message -----
From: "Michael Nelson" <mike-nelson-ODAAT at WORLDNET.ATT.NET>
To: <EUPHORIA at LISTSERV.MUOHIO.EDU>
Sent: Sunday, March 12, 2000 10:57 AM
Subject: EDS
> The attached zip contains an upgrade to EDS which provides for having
> multiple databases open and choosing the current db with db_select().
> Compression is provided by db_compress(), which renames the current db to
> <name>.tmp and copies each table and record to a new db with the old name.
>
> I don't know Linux--the system commands in db_compress() may need to be
made
> platform specific. I haven't updated the documentation. A test program for
> compression is provided.
Yes, you need to change ren to mv when on Linux. I've attached an updated
version which checks the platform and uses Linux commands if necessicary.
>
> -- Mike Nelson
>
>
>
Jeff Fielding
------=_NextPart_000_0025_01BF8C17.BCE00E40
name="db.zip"
13. Re: EDS
Mike Nelson writes:
> The attached zip contains an upgrade to EDS which
> provides for having multiple databases open and
> choosing the current db with db_select().
> Compression is provided by db_compress(), which
> renames the current db to <name>.tmp and copies
> each table and record to a new db with the old name.
Thanks. I was planning to do both of those.
I'll incorporate what you've done into the official version,
(perhaps with minor stylistic changes).
Thanks also to Jeff Fielding.
Regards,
Rob Craig
Rapid Deployment Software
http://www.RapidEuphoria.com
14. Re: EDS
Michael Nelson wrote:
>The attached zip contains an upgrade to EDS which provides for having
>multiple databases open and choosing the current db with db_select().
>Compression is provided by db_compress(), which renames the current db to
><name>.tmp and copies each table and record to a new db with the old name.
>
>I don't know Linux--the system commands in db_compress() may need to be made
>platform specific. I haven't updated the documentation. A test program for
>compression is provided.
>
>-- Mike Nelson
Who rubbed your bottle...you must be a geni..us 
I always get in trouble
for my bad puns. Your code appears to be exactly what the doctor ordered.
Compression, backup, and safety all in one swell foop. Bravo.
Everett L.(Rett) Williams
rett at gvtc.com
15. Re: EDS
> Compression, backup, and safety all in one swell foop. Bravo.
Wow, another person who likes wording with plays, mike le!
Nick
16. Re: EDS
----- Original Message -----
From: "Nick Johnson" <arachnid at MAD.SCIENTIST.COM>
To: <EUPHORIA at LISTSERV.MUOHIO.EDU>
Sent: Sunday, March 12, 2000 9:09 PM
Subject: Re: EDS
> > Compression, backup, and safety all in one swell foop. Bravo.
> Wow, another person who likes wording with plays, mike le!
Oh no,, i've been fighing dyslexia all my life and you guys are going to
make it recur! oh well, if nothing else, the Pee Little Thrigs was funny.
Kat
17. Re: EDS
On Fri, 10 Mar 2000, I wrote:
> I need the ability to lock individual records, make a change, and write the
> change back to disk (quickly) before unlocking the record for someone else to
> use....
The thought occurs to me that using Pete's RDC.e, I should be able to turn EDS
into a db server, and write client programs for my users. That would accomplish
two things: all the data would be accessed by one program, not several, and
only one function could be performed at any given time, so that the issue of
file/record locking would be moot, and multi-user capability would be more or
less free.
This seems too easy. Am I missing something?
Irv
18. Re: EDS
> The thought occurs to me that using Pete's RDC.e, I should be able to turn
EDS
> into a db server, and write client programs for my users. That would
accomplish
> two things: all the data would be accessed by one program, not several,
and
> only one function could be performed at any given time, so that the issue
of
> file/record locking would be moot, and multi-user capability would be more
> or less free.
This sounds brilliant! The only catch would be that RDC.e only supports
linux, doesn't it? But of course someone is bound to write one for winsock
at some stage.
Just think - you could web-serve your database, and have people using it
from anywhere in the world :)
Nick
19. Re: EDS
There is a small problem in the supplied createdb.ex progrm.
In the following code
for i = 1 to 1000 do
key = rand(1000)
rec = db_find_key(key)
if rec > 0 then
** data = db_record_data(key) **
else
printf(SCREEN, "Failed to find key: %d\n", key)
end if
end for
It should be ** data = db_record_data(rec) **
I think it works in the sample as everything is numeric, the rec and the
key.
Regards,
Prasanta.
20. Re: EDS
Irv Mullins wrote:
>On Fri, 10 Mar 2000, I wrote:
>
>> I need the ability to lock individual records, make a change, and write the
>> change back to disk (quickly) before unlocking the record for someone else to
>> use....
>
>The thought occurs to me that using Pete's RDC.e, I should be able to turn EDS
>into a db server, and write client programs for my users. That would accomplish
>two things: all the data would be accessed by one program, not several, and
>only one function could be performed at any given time, so that the issue of
>file/record locking would be moot, and multi-user capability would be more or
>less free.
>
>This seems too easy. Am I missing something?
>Irv
Other than tracking the IO of each user and their connection to the db, you're
not missing much. It is going to matter a lot if "x" reads record 10, and then
"y" reads record 10. One or both or neither of them may write that record
back, modified. The order may mildly affect the integrity of your data, and
both "x" and "y" may find some fault with your handling of "their" data.
Multiuser access is not made transparent by server mediation unless the
server has clearly known and enforceable rules for such situations. Record
locking is one of the hardest areas to handle regardless of implementation.
Without timers and timeouts on both server and client, deadly embraces are
the inevitable result of locking. There are more than a few other such subtle
situations that result among other times when multi-threaded clients access
a non-threaded server...which is a possibility in this situation. The bottom
line is that nothing changes about the need for certain types of constructs
and rules in database access...only the location of the code.
If you really do serialize all IO as was indicated by your premise, you will
make the database almost useless for multi-user access. The time between
a read and a write of a single piece of data is wall-clock, human time, not
computer cycle time.
Everett L.(Rett) Williams
rett at gvtc.com
21. Re: EDS
>Irv Mullins wrote:
>
>>On Fri, 10 Mar 2000, I wrote:
>>
>>> I need the ability to lock individual records, make a change, and write the
>>> change back to disk (quickly) before unlocking the record for someone else
>>> to
>>> use....
One contradictory thought. All said in my previous post does not apply if the
database is read-only during multi-user access.
Everett L.(Rett) Williams
rett at gvtc.com
22. Re: EDS
Prasanta Chakraborty writes:
> There is a small problem in the supplied createdb.ex progrm.
> ...
> It should be ** data = db_record_data(rec) **
Thanks!
I'll post a new version later today.
Rob Craig
Rapid Deployment Software
http://www.RapidEuphoria.com
23. Re: EDS
----- Original Message -----
From: "Everett Williams" <rett at GVTC.COM>
To: <EUPHORIA at LISTSERV.MUOHIO.EDU>
Sent: Tuesday, March 14, 2000 1:10 AM
Subject: Re: EDS
> Irv Mullins wrote:
>
> >On Fri, 10 Mar 2000, I wrote:
> >
> >> I need the ability to lock individual records, make a change, and write
the
> >> change back to disk (quickly) before unlocking the record for someone
else to
> >> use....
> >
> >The thought occurs to me that using Pete's RDC.e, I should be able to
turn EDS
> >into a db server, and write client programs for my users. That would
accomplish
> >two things: all the data would be accessed by one program, not several,
and
> >only one function could be performed at any given time, so that the issue
of
> >file/record locking would be moot, and multi-user capability would be
more or
> >less free.
> >
> >This seems too easy. Am I missing something?
> >Irv
> Other than tracking the IO of each user and their connection to the db,
you're
> not missing much. It is going to matter a lot if "x" reads record 10, and
then
> "y" reads record 10. One or both or neither of them may write that record
> back, modified. The order may mildly affect the integrity of your data,
and
> both "x" and "y" may find some fault with your handling of "their" data.
> Multiuser access is not made transparent by server mediation unless the
> server has clearly known and enforceable rules for such situations. Record
> locking is one of the hardest areas to handle regardless of
implementation.
> Without timers and timeouts on both server and client, deadly embraces are
> the inevitable result of locking. There are more than a few other such
subtle
> situations that result among other times when multi-threaded clients
access
> a non-threaded server...which is a possibility in this situation. The
bottom
> line is that nothing changes about the need for certain types of
constructs
> and rules in database access...only the location of the code.
>
> If you really do serialize all IO as was indicated by your premise, you
will
> make the database almost useless for multi-user access. The time between
> a read and a write of a single piece of data is wall-clock, human time,
not
> computer cycle time.
This was well documented in some early multi-cpu systems that used cpu
caches for retrieved data. After each cpu did it's thing, and modified the
data, it would write back the new data, and the file server would let all
the other cpus who read the data that their cache was no longer correct.
Then it was up to each cpu to request an updated version, see if the updates
impacted it, and if they did impact, then redo whatever they had done to the
old data unto the new data. Then of course the 2nd cpu could do a writeback
and the 1st cpu would get told it's data was now invalid. In some cases,
this would drop system speed to 10% or less of what was expected, ie a dual
pentium233 would run at single pentium60 output for that problem.
I ran into this problem in doing Ai, but with tight control over what is
serial and what is multithreaded, coupled with faster cpu than i had then,
the problem is reduced to figureing out what code is best threaded vs
nonthreaded. That takes it out of the realm of DB management, especially if
deletions are not permitted, and additions are time/origin stamped. It's up
to whoever has the data cached to decide to ask if the data has been
changed.
Kat
24. EDS
hi Robert Craig,
I would really like to see your enhancement to EDS ASAP please. in my
version i did db_table_rename() and did "pointer caching" an idea i got
from Derek Parnell. Derek told me, that he speeds up db_select_table()
by keeping the pointers in memory other than having to collect them on
every change of table. i did that to my version of EDS and he was very
correct. if tables have more than 100 records you'll notice how
fast/good his idea is. up to 20 times faster. the only problem with his
method is the memory consumption, but is still a better alternative.
I have other code fragments, i want you to look at, i'll send them to
you if you like.
Jordah Ferguson
25. Re: EDS
Jordah Ferguson writes:
> I would really like to see your enhancement to EDS ASAP please.
Sent.
> I have other code fragments, i want you to look at,
> i'll send them to you if you like.
OK. Thanks.
Rob Craig
Rapid Deployment Software
http://www.RapidEuphoria.com
