1. EDS
- Posted by Mike Sabal <MikeS at NOTATIONS.COM> Mar 08, 2000
- 653 views
I looked at Rob's database.e and was quite impressed. I haven't had a = chance to play around with it yet, but this certainly makes application = programming much easier. If you could get it to handle simultaneous = multi-user access (by locking individual records perhaps) and make it a = standard include in the next version of Euphoria, it would be a very good = thing. Michael J. Sabal mjs at osa.att.ne.jp=20 http://home.att.ne.jp/gold/mjs/index.html=20 ICQ: 32461146
2. Re: EDS
- Posted by Brian Jackson <bjackson at 2FARGON.COM> Mar 08, 2000
- 603 views
I am very impressed with EDS so far. Simple, extensible - it's a logical extension of Euphoria. I have a feeling it will turn out to be a very powerful tool. OK, now for the wish list: 1) Positional read. I'd like to be able to use "ABCD" as my key for a POSITIONAL read. The "file pointer" (or whatever the beastie is) is put just before where key ABCD would occur in the file, whether it's there or not. 2) Sequential key read. Along the same lines, I'd like to have a sequential (probably forward and backward) read based of the key of the last record read. That way, if I do a positional read for the key "JO", I could get all the records whose keys start with JO, and then logically exit the loop. -- short example of some code that will display -- every key that starts with the characters "JO" object result sequence mykey result = db_positional_read("JO") while 1 do result = db_read_keysequential() if atom(result) then exit end if mykey = result[1][1..2] if compare(mykey, "JO") exit end if puts(1,"This key = " & result[1] & "\n") end while -- end code 3) Adding some kind of wildcard match on a key. For instance, if my key was InvoiceNumber,InvoiceDate and I wanted to retrieve all the records for a given date, it would be nice to wildcard match a key. 4) Maybe even a better idea would be to allow multiple indices. This would get around that problem and probably be easier to implement. Well, that's about all I can think of at the moment! It really looks excellent Rob. Keep up the great work! Brian Jackson 2FARGON Technologies bjackson at 2fargon.com http://www.2fargon.com
3. Re: EDS
- Posted by David Cuny <dcuny at LANSET.COM> Mar 08, 2000
- 600 views
- Last edited Mar 09, 2000
I finally got a chance to look at EDS. Pretty clever. I'm thinking of using EDS as a font server for my DOS window manager. I use Jiri's fonts (with Pete's rendering code), but it lacks some information, such as a font's family, name, size, descent, and so on. If I stored all the fonts in an EDS file, I could determine if a particular font existed, or find a near replacement in the same family. It also looks like it'll be easy to extend the font information without having to create a new file format: the whole thing can be stored in EDS. Another application of EDS might be to store bitmaps and their palettes. It could work like a Windows resource file. Have you considered attaching an EDS file to a bound Euphoria program (like my RES program does)? Real exciting stuff! -- David Cuny
4. EDS
- Posted by Nick Johnson <arachnid at MAD.SCIENTIST.COM> Mar 11, 2000
- 604 views
------=_NextPart_000_0005_01BF8B38.EEBF3100 charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable A few more suggestions: A compression tool would be great - it could rearrange all the records = so there are no free blocks and everything is contiguous. A sort of a = defrag for a database. This would be ideal for when (if) you end up with = a large collection of tiny blocks of free space that were partially = filled by another item, but nothing else will be small enough. Any ideas why the delete takes so long? Twice as long as insert on my = computer. Does it wipe over all the unused space? Nick ------=_NextPart_000_0005_01BF8B38.EEBF3100 charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD> <META content=3D"text/html; charset=3Diso-8859-1" = http-equiv=3DContent-Type> <META content=3D"MSHTML 5.00.2614.3500" name=3DGENERATOR> <STYLE></STYLE> </HEAD> <BODY bgColor=3D#ffffff> <DIV><FONT face=3DArial size=3D2>A few more suggestions:</FONT></DIV> <DIV><FONT face=3DArial size=3D2>A compression tool would be great - it = could=20 rearrange all the records so there are no free blocks and everything is=20 contiguous. A sort of a defrag for a database. This would be ideal for = when (if)=20 you end up with a large collection of tiny blocks of free space that = were=20 partially filled by another item, but nothing else will be small=20 enough.</FONT></DIV> <DIV><FONT face=3DArial size=3D2>Any ideas why the delete takes so long? = Twice as=20 long as insert on my computer. Does it wipe over all the unused=20 space?</FONT></DIV> <DIV> </DIV> ------=_NextPart_000_0005_01BF8B38.EEBF3100--
5. Re: EDS
- Posted by "Cuny, David at DSS" <David.Cuny at DSS.CA.GOV> Mar 10, 2000
- 594 views
Nick wrote: > A compression tool would be great It seems to me that rewriting all the contents of the file to a new file would do the trick. -- David Cuny
6. Re: EDS
- Posted by Robert Craig <rds at ATTCANADA.NET> Mar 10, 2000
- 589 views
Nick Johnson writes: > A compression tool would be great I was planning to write one eventually. If your database tends to grow over time, as most do, you probably won't have much need for it. As David points out, you could write one yourself by creating a new database and copying everything over to it using db_insert() etc. > Any ideas why the delete takes so long? Twice as > long as insert on my computer. It takes longer for me too. I haven't tried to figure out why, but it doesn't seem that unreasonable. > Does it wipe over all the unused space? No. Earlier today I posted a new alpha release of EDS. It fixes a couple of nasty bugs that showed up during "stress" testing. The stress test is included. Also, I've changed a few routines, so you'll have to make some small changes to any code you've already written. Regards, Rob Craig Rapid Deployment Software http://www.RapidEuphoria.com
7. Re: EDS
- Posted by Irv Mullins <irv at ELLIJAY.COM> Mar 10, 2000
- 609 views
On Fri, 10 Mar 2000, you wrote: > A few more suggestions: > A compression tool would be great - it could rearrange all the records so > there are no free blocks and everything is contiguous. A sort of a defrag for a > database. This would be ideal for when (if) you end up with a large collection of > tiny blocks of free space that were partially filled by another item, but nothing > else will be small enough. > Any ideas why the delete takes so long? Twice as long as insert on my > computer. Does it wipe over all the unused space? I've been trying to decide whether to mention this or not. I don't want to discourage progress with this new Euphoria database, but there are some things that really should be considered: I have been using Euphoria successfully with small databases containing 600-1000 customer records and 2000-3000 A/R transactions. Much more than that, and the read/write times begin to get annoyingly long. Too long for automatic updates after each change, which is the way I prefer to do things. Rob's compact EDS file structure may improve read/write speeds somewhat; I haven't been able to port any test data yet, so I don't know, but as it stands this limits the use of Euphoria to only the smallest business apps. Larger databases, the kind needed by all but my smallest client, need to track anywhere from a couple thousand to 90,000 customers and their transactions per month. Most clients also need multi-user capability. The "whole ball of wax" method (load it all into memory, then write it all back) just won't work in these situations. I need the ability to lock individual records, make a change, and write the change back to disk (quickly) before unlocking the record for someone else to use. I see no way this can be done without some kind of fixed size record. Too bad, because otherwise, Euphoria would be the language of choice. It does everything else quite nicely, and lets me develop apps quickly (always an important consideration). Regards, Irv
8. Re: EDS
- Posted by Robert Craig <rds at ATTCANADA.NET> Mar 10, 2000
- 607 views
- Last edited Mar 11, 2000
Irv Mullins writes: > Rob's compact EDS file structure may improve > read/write speeds somewhat It not only uses less disk space (maybe half), but it's also faster to read and write the compact format. > The "whole ball of wax" method (load it all into memory, > then write it all back) just won't work in these situations. EDS doesn't load the "whole ball of wax". When you select a new table, it reads in a 4-byte pointer per record in that table. If you have 1000 records, it will read in 4000 contiguous bytes from the disk, which is probably the minimum that the O/S will read, even if you ask for a single byte. After that, only the records that you access are read/written to disk. With a binary search for a key, you might also have to read a few key values before finding the right record. I started off designing a complicated b-tree arrangement that didn't require all the pointers to be read in, but then I realized that this much simpler scheme should be ok until you get to a really huge number of records (say a million). I can always change the implementation if people start creating million record databases. > I need the ability to lock individual records, > make a change, and write the change back to disk (quickly) > before unlocking the record for someone else to use. I see no > way this can be done without some kind of fixed size record. Euphoria 2.2 has file and variable-length record locking (only file locking so far in the case of Linux). I will eventually add this in to EDS. I don't think that having fully-flexible record and field lengths will cause a serious problem. It just makes storage allocation and deallocation more complicated and somewhat slower. Regards, Rob Craig Rapid Deployment Software http://www.RapidEuphoria.com
9. Re: EDS
- Posted by Everett Williams <rett at GVTC.COM> Mar 10, 2000
- 605 views
- Last edited Mar 11, 2000
Robert Craig wrote: >I can always change the implementation if people start >creating million record databases. > >> I need the ability to lock individual records, >> make a change, and write the change back to disk (quickly) >> before unlocking the record for someone else to use. I see no >> way this can be done without some kind of fixed size record. > >Euphoria 2.2 has file and variable-length record locking >(only file locking so far in the case of Linux). I will eventually >add this in to EDS. I don't think that having fully-flexible >record and field lengths will cause a serious problem. >It just makes storage allocation and deallocation more >complicated and somewhat slower. > As Euphoria starts to get an industrial strength IDE and windows library along with a few other odds and ends, why not just go ahead and do things as if we were headed for that industrial-strength world. If we get namespace resolution and structures and the other few odds and ends needed embedded into the language along with an industrial strength DB and IDE and GUI libs, there will be few limits to the things that can reasonably be attempted in Euphoria. The request for a direct C call facility, eliminating the need for define_cfunc and all that would allow us access to the rest of the world's code. Maybe the attempt to do these industrial-strength things will point out the needs that are left in the language. Everett L.(Rett) Williams rett at gvtc.com
10. EDS
- Posted by Bernie Ryan <LockCityData at CS.COM> Mar 11, 2000
- 596 views
Why is everyone sitting back and asking for this feature and that feature to be added to the EDS. The source code is available so why don't the users that want these feature add them and submit them to Rob. It is always easier for someone to write the code for you but you don't learn anything from doing that. I would rather that Rob would dedicate his valuable time to adding features to the langauge that would allow us to add are own features. ( namespaces, variable reference, conditional includes, etc. ) Bernie
11. EDS
- Posted by Michael Nelson <mike-nelson-ODAAT at WORLDNET.ATT.NET> Mar 12, 2000
- 597 views
------=_NextPart_000_000D_01BF8BF8.8C3E32E0 charset="iso-8859-1" The attached zip contains an upgrade to EDS which provides for having multiple databases open and choosing the current db with db_select(). Compression is provided by db_compress(), which renames the current db to <name>.tmp and copies each table and record to a new db with the old name. I don't know Linux--the system commands in db_compress() may need to be made platform specific. I haven't updated the documentation. A test program for compression is provided. -- Mike Nelson ------=_NextPart_000_000D_01BF8BF8.8C3E32E0 name="Database.zip"
12. Re: EDS
- Posted by Jeffrey Fielding <JJProg at CYBERBURY.NET> Mar 12, 2000
- 609 views
------=_NextPart_000_0025_01BF8C17.BCE00E40 charset="iso-8859-1" ----- Original Message ----- From: "Michael Nelson" <mike-nelson-ODAAT at WORLDNET.ATT.NET> To: <EUPHORIA at LISTSERV.MUOHIO.EDU> Sent: Sunday, March 12, 2000 10:57 AM Subject: EDS > The attached zip contains an upgrade to EDS which provides for having > multiple databases open and choosing the current db with db_select(). > Compression is provided by db_compress(), which renames the current db to > <name>.tmp and copies each table and record to a new db with the old name. > > I don't know Linux--the system commands in db_compress() may need to be made > platform specific. I haven't updated the documentation. A test program for > compression is provided. Yes, you need to change ren to mv when on Linux. I've attached an updated version which checks the platform and uses Linux commands if necessicary. > > -- Mike Nelson > > > Jeff Fielding ------=_NextPart_000_0025_01BF8C17.BCE00E40 name="db.zip"
13. Re: EDS
- Posted by Robert Craig <rds at ATTCANADA.NET> Mar 12, 2000
- 593 views
Mike Nelson writes: > The attached zip contains an upgrade to EDS which > provides for having multiple databases open and > choosing the current db with db_select(). > Compression is provided by db_compress(), which > renames the current db to <name>.tmp and copies > each table and record to a new db with the old name. Thanks. I was planning to do both of those. I'll incorporate what you've done into the official version, (perhaps with minor stylistic changes). Thanks also to Jeff Fielding. Regards, Rob Craig Rapid Deployment Software http://www.RapidEuphoria.com
14. Re: EDS
- Posted by Everett Williams <rett at GVTC.COM> Mar 12, 2000
- 606 views
Michael Nelson wrote: >The attached zip contains an upgrade to EDS which provides for having >multiple databases open and choosing the current db with db_select(). >Compression is provided by db_compress(), which renames the current db to ><name>.tmp and copies each table and record to a new db with the old name. > >I don't know Linux--the system commands in db_compress() may need to be made >platform specific. I haven't updated the documentation. A test program for >compression is provided. > >-- Mike Nelson Who rubbed your bottle...you must be a geni..us I always get in trouble for my bad puns. Your code appears to be exactly what the doctor ordered. Compression, backup, and safety all in one swell foop. Bravo. Everett L.(Rett) Williams rett at gvtc.com
15. Re: EDS
- Posted by Nick Johnson <arachnid at MAD.SCIENTIST.COM> Mar 13, 2000
- 600 views
> Compression, backup, and safety all in one swell foop. Bravo. Wow, another person who likes wording with plays, mike le! Nick
16. Re: EDS
- Posted by Kat <gertie at ZEBRA.NET> Mar 13, 2000
- 578 views
----- Original Message ----- From: "Nick Johnson" <arachnid at MAD.SCIENTIST.COM> To: <EUPHORIA at LISTSERV.MUOHIO.EDU> Sent: Sunday, March 12, 2000 9:09 PM Subject: Re: EDS > > Compression, backup, and safety all in one swell foop. Bravo. > Wow, another person who likes wording with plays, mike le! Oh no,, i've been fighing dyslexia all my life and you guys are going to make it recur! oh well, if nothing else, the Pee Little Thrigs was funny. Kat
17. Re: EDS
- Posted by Irv Mullins <irv at ELLIJAY.COM> Mar 13, 2000
- 616 views
On Fri, 10 Mar 2000, I wrote: > I need the ability to lock individual records, make a change, and write the > change back to disk (quickly) before unlocking the record for someone else to > use.... The thought occurs to me that using Pete's RDC.e, I should be able to turn EDS into a db server, and write client programs for my users. That would accomplish two things: all the data would be accessed by one program, not several, and only one function could be performed at any given time, so that the issue of file/record locking would be moot, and multi-user capability would be more or less free. This seems too easy. Am I missing something? Irv
18. Re: EDS
- Posted by Nick Johnson <arachnid at MAD.SCIENTIST.COM> Mar 14, 2000
- 612 views
> The thought occurs to me that using Pete's RDC.e, I should be able to turn EDS > into a db server, and write client programs for my users. That would accomplish > two things: all the data would be accessed by one program, not several, and > only one function could be performed at any given time, so that the issue of > file/record locking would be moot, and multi-user capability would be more > or less free. This sounds brilliant! The only catch would be that RDC.e only supports linux, doesn't it? But of course someone is bound to write one for winsock at some stage. Just think - you could web-serve your database, and have people using it from anywhere in the world :) Nick
19. Re: EDS
- Posted by Prasanta Chakraborty <prasanta at WRITEME.COM> Mar 14, 2000
- 624 views
There is a small problem in the supplied createdb.ex progrm. In the following code for i = 1 to 1000 do key = rand(1000) rec = db_find_key(key) if rec > 0 then ** data = db_record_data(key) ** else printf(SCREEN, "Failed to find key: %d\n", key) end if end for It should be ** data = db_record_data(rec) ** I think it works in the sample as everything is numeric, the rec and the key. Regards, Prasanta.
20. Re: EDS
- Posted by Everett Williams <rett at GVTC.COM> Mar 14, 2000
- 627 views
Irv Mullins wrote: >On Fri, 10 Mar 2000, I wrote: > >> I need the ability to lock individual records, make a change, and write the >> change back to disk (quickly) before unlocking the record for someone else to >> use.... > >The thought occurs to me that using Pete's RDC.e, I should be able to turn EDS >into a db server, and write client programs for my users. That would accomplish >two things: all the data would be accessed by one program, not several, and >only one function could be performed at any given time, so that the issue of >file/record locking would be moot, and multi-user capability would be more or >less free. > >This seems too easy. Am I missing something? >Irv Other than tracking the IO of each user and their connection to the db, you're not missing much. It is going to matter a lot if "x" reads record 10, and then "y" reads record 10. One or both or neither of them may write that record back, modified. The order may mildly affect the integrity of your data, and both "x" and "y" may find some fault with your handling of "their" data. Multiuser access is not made transparent by server mediation unless the server has clearly known and enforceable rules for such situations. Record locking is one of the hardest areas to handle regardless of implementation. Without timers and timeouts on both server and client, deadly embraces are the inevitable result of locking. There are more than a few other such subtle situations that result among other times when multi-threaded clients access a non-threaded server...which is a possibility in this situation. The bottom line is that nothing changes about the need for certain types of constructs and rules in database access...only the location of the code. If you really do serialize all IO as was indicated by your premise, you will make the database almost useless for multi-user access. The time between a read and a write of a single piece of data is wall-clock, human time, not computer cycle time. Everett L.(Rett) Williams rett at gvtc.com
21. Re: EDS
- Posted by Everett Williams <rett at GVTC.COM> Mar 14, 2000
- 633 views
>Irv Mullins wrote: > >>On Fri, 10 Mar 2000, I wrote: >> >>> I need the ability to lock individual records, make a change, and write the >>> change back to disk (quickly) before unlocking the record for someone else >>> to >>> use.... One contradictory thought. All said in my previous post does not apply if the database is read-only during multi-user access. Everett L.(Rett) Williams rett at gvtc.com
22. Re: EDS
- Posted by Robert Craig <rds at ATTCANADA.NET> Mar 14, 2000
- 604 views
Prasanta Chakraborty writes: > There is a small problem in the supplied createdb.ex progrm. > ... > It should be ** data = db_record_data(rec) ** Thanks! I'll post a new version later today. Rob Craig Rapid Deployment Software http://www.RapidEuphoria.com
23. Re: EDS
- Posted by Kat <gertie at ZEBRA.NET> Mar 14, 2000
- 618 views
----- Original Message ----- From: "Everett Williams" <rett at GVTC.COM> To: <EUPHORIA at LISTSERV.MUOHIO.EDU> Sent: Tuesday, March 14, 2000 1:10 AM Subject: Re: EDS > Irv Mullins wrote: > > >On Fri, 10 Mar 2000, I wrote: > > > >> I need the ability to lock individual records, make a change, and write the > >> change back to disk (quickly) before unlocking the record for someone else to > >> use.... > > > >The thought occurs to me that using Pete's RDC.e, I should be able to turn EDS > >into a db server, and write client programs for my users. That would accomplish > >two things: all the data would be accessed by one program, not several, and > >only one function could be performed at any given time, so that the issue of > >file/record locking would be moot, and multi-user capability would be more or > >less free. > > > >This seems too easy. Am I missing something? > >Irv > Other than tracking the IO of each user and their connection to the db, you're > not missing much. It is going to matter a lot if "x" reads record 10, and then > "y" reads record 10. One or both or neither of them may write that record > back, modified. The order may mildly affect the integrity of your data, and > both "x" and "y" may find some fault with your handling of "their" data. > Multiuser access is not made transparent by server mediation unless the > server has clearly known and enforceable rules for such situations. Record > locking is one of the hardest areas to handle regardless of implementation. > Without timers and timeouts on both server and client, deadly embraces are > the inevitable result of locking. There are more than a few other such subtle > situations that result among other times when multi-threaded clients access > a non-threaded server...which is a possibility in this situation. The bottom > line is that nothing changes about the need for certain types of constructs > and rules in database access...only the location of the code. > > If you really do serialize all IO as was indicated by your premise, you will > make the database almost useless for multi-user access. The time between > a read and a write of a single piece of data is wall-clock, human time, not > computer cycle time. This was well documented in some early multi-cpu systems that used cpu caches for retrieved data. After each cpu did it's thing, and modified the data, it would write back the new data, and the file server would let all the other cpus who read the data that their cache was no longer correct. Then it was up to each cpu to request an updated version, see if the updates impacted it, and if they did impact, then redo whatever they had done to the old data unto the new data. Then of course the 2nd cpu could do a writeback and the 1st cpu would get told it's data was now invalid. In some cases, this would drop system speed to 10% or less of what was expected, ie a dual pentium233 would run at single pentium60 output for that problem. I ran into this problem in doing Ai, but with tight control over what is serial and what is multithreaded, coupled with faster cpu than i had then, the problem is reduced to figureing out what code is best threaded vs nonthreaded. That takes it out of the realm of DB management, especially if deletions are not permitted, and additions are time/origin stamped. It's up to whoever has the data cached to decide to ask if the data has been changed. Kat
24. EDS
- Posted by jordah ferguson <jorfergie03 at yahoo.com> Sep 30, 2002
- 591 views
hi Robert Craig, I would really like to see your enhancement to EDS ASAP please. in my version i did db_table_rename() and did "pointer caching" an idea i got from Derek Parnell. Derek told me, that he speeds up db_select_table() by keeping the pointers in memory other than having to collect them on every change of table. i did that to my version of EDS and he was very correct. if tables have more than 100 records you'll notice how fast/good his idea is. up to 20 times faster. the only problem with his method is the memory consumption, but is still a better alternative. I have other code fragments, i want you to look at, i'll send them to you if you like. Jordah Ferguson
25. Re: EDS
- Posted by Robert Craig <rds at RapidEuphoria.com> Sep 30, 2002
- 624 views
Jordah Ferguson writes: > I would really like to see your enhancement to EDS ASAP please. Sent. > I have other code fragments, i want you to look at, > i'll send them to you if you like. OK. Thanks. Rob Craig Rapid Deployment Software http://www.RapidEuphoria.com