1. Storing Data for Web Site
If you think about the blogging sites, say, blogger.com, and all the users
they have blogging every day, they've gotta have outrageous storage
requirements,
no? We're talking GBs at a minimum... Is EDB storage pretty efficient? I wonder
how much data could be stored in, say, 60GB of hard disk space. Or what is the
minimum storage capacity of a EDB? Is it still 2GB? Rob, you might want to
consider upping that for me. 8)
-=ck
"Programming in a state of EUPHORIA."
http://www.cklester.com/euphoria/
2. Re: Storing Data for Web Site
- Posted by Robert Craig <rds at RapidEuphoria.com>
Sep 09, 2005
-
Last edited Sep 10, 2005
cklester wrote:
> If you think about the blogging sites, say, blogger.com, and all the users
> they have blogging every day, they've gotta have outrageous storage
> requirements,
> no? We're talking GBs at a minimum... Is EDB storage pretty efficient? I
> wonder
> how much data could be stored in, say, 60GB of hard disk space. Or what is the
> minimum storage capacity of a EDB? Is it still 2GB? Rob, you might want to
> consider upping that for me. 8)
A Euphoria database is pretty efficient. It stores
small integers in one byte, not four. The overhead
on sequences is also pretty low.
I think you'll run into trouble at around 2Gb.
You would have to store your data in multiple databases at that point.
Someday I (or someone else) should try to make the limit
much larger.
Regards,
Rob Craig
Rapid Deployment Software
http://www.RapidEuphoria.com
3. Re: Storing Data for Web Site
- Posted by Greg Haberek <ghaberek at gmail.com>
Sep 09, 2005
-
Last edited Sep 10, 2005
> If you think about the blogging sites, say, blogger.com, and all the user=
s
> they have blogging every day, they've gotta have outrageous storage requi=
rements,
> no? We're talking GBs at a minimum... Is EDB storage pretty efficient? I =
wonder
> how much data could be stored in, say, 60GB of hard disk space. Or what i=
s the
> minimum storage capacity of a EDB? Is it still 2GB? Rob, you might want t=
o
> consider upping that for me. 8)
Try looking into Tsunami Record Manager. Its quite a big step up from
EDS, Euman wrote a wrapper for it. It has built-in compression, which
should keep those gigabytes of data down, as well as easier to manage,
since it supports 16 terabyte files. I hope your server is on Windows
though, I don't think they have a Linux version.
Snippit from http://www.trm-ug.com/
* Max File Size........... 16 Terabytes 1
* Max Record Size.......... 2 Gigabytes 1
* Max Record Count......... 2,147,483,647 2
* Max Key Size........... 255 Bytes
* Max Segments Per Key... 128
* Max Indexes Per File... 128
* Max Open Files......... 255
(1) Subject to compiler and operating system limits.
(2) Subject to physical file size limits.
* Very fast, self-balancing b-tree indexing.
* Records may contain any text and/or binary data.
* Records may be any mixture of fixed/variable length.
* Record and key compression options to reduce file size.
* All indexes are stored in the same file with the records.
* Duplicate keys allowed/disallowed on an index-by-index basis.
* Key-only searches supported for all index-based Get operations.
* Case sensitive/insensitive keys on a segment-by-segment basis.
* Integer, long integer and quad integer binary keys supported.
* Automatic re-use of free space produced by record deletions.
* Completely thread safe, offering true design flexibility.
* 65K DLL supports single-user & multi-user file access.
* Includes a 97 page user's guide in Adobe PDF format.
4. Re: Storing Data for Web Site
- Posted by cklester <cklester at yahoo.com>
Sep 09, 2005
-
Last edited Sep 10, 2005
Greg Haberek wrote:
>
> Try looking into Tsunami Record Manager. Its quite a big step up from
> EDS, Euman wrote a wrapper for it. It has built-in compression, which
> should keep those gigabytes of data down, as well as easier to manage,
> since it supports 16 terabyte files. I hope your server is on Windows
> though, I don't think they have a Linux version.
Looks good... DOH! My server is a Linux box. :(
-=ck
"Programming in a state of EUPHORIA."
http://www.cklester.com/euphoria/
5. Re: Storing Data for Web Site
- Posted by Ryan W. Johnson <ryanj at fluidae.com>
Sep 09, 2005
-
Last edited Sep 10, 2005
I was thinking, what if a database used multiple files instead of a single
database file? In other words, the database was in a folder, and each table
was in a separate file. Do you think that would work well, compared to a
single file such as EDB or Microsoft Access?
~Ryan W. Johnson
6. Re: Storing Data for Web Site
- Posted by Chris Burch <chriscrylex at aol.com>
Sep 09, 2005
-
Last edited Sep 10, 2005
cklester wrote:
>
> Greg Haberek wrote:
> >
> > Try looking into Tsunami Record Manager. Its quite a big step up from
> > EDS, Euman wrote a wrapper for it. It has built-in compression, which
> > should keep those gigabytes of data down, as well as easier to manage,
> > since it supports 16 terabyte files. I hope your server is on Windows
> > though, I don't think they have a Linux version.
>
> Looks good... DOH! My server is a Linux box. :(
>
> -=ck
> "Programming in a state of EUPHORIA."
> <a
> href="http://www.cklester.com/euphoria/">http://www.cklester.com/euphoria/</a>
>
Urm, at the sake of repeating myself, ( 
) try eusqlite, unfortunately it
only seems to do do 2 terrabyte databases (miniscule I know) - sorry, but it does
work on Linux.
Chris
Ps I still have no idea how to work it on a web server.
http://members.aol.com/chriscrylex/euphoria.htm
http://uboard.proboards32.com/
http://members.aol.com/chriscrylex/EUSQLite/eusql.html
7. Re: Storing Data for Web Site
- Posted by cklester <cklester at yahoo.com>
Sep 09, 2005
-
Last edited Sep 10, 2005
Ryan W. Johnson wrote:
>
> I was thinking, what if a database used multiple files instead of a single
> database file? In other words, the database was in a folder, and each table
> was in a separate file. Do you think that would work well, compared to a
> single file such as EDB or Microsoft Access?
I don't mind either paradigm. SQLite uses one file while IBM's open-source
DB (forget the name offhand) uses multiple files.
I ran a test the other day and using EuSQL
SELECT * FROM MEMBERS WHERE VALIDATED=1
and got an average of 135/sec. I don't know if that's good or bad, but
it's a potential of 11,664,000 per day. I'm going to run this test with
SQLite and some others (Linux box, so no Tsunami) and see what happens.
-=ck
"Programming in a state of EUPHORIA."
http://www.cklester.com/euphoria/
8. Re: Storing Data for Web Site
- Posted by Ryan W. Johnson <ryanj at fluidae.com>
Sep 09, 2005
-
Last edited Sep 10, 2005
cklester wrote:
> I ran a test the other day and using EuSQL
>
> SELECT * FROM MEMBERS WHERE VALIDATED=1
>
> and got an average of 135/sec. I don't know if that's good or bad, but
> it's a potential of 11,664,000 per day. I'm going to run this test with
> SQLite and some others (Linux box, so no Tsunami) and see what happens.
Cool, I've been wondering how fast EDB is compared to other databases.
~Ryan W. Johnson
9. Re: Storing Data for Web Site
Robert Craig wrote:
>
> A Euphoria database is pretty efficient. It stores
> small integers in one byte, not four. The overhead
> on sequences is also pretty low.
> I think you'll run into trouble at around 2Gb.
> You would have to store your data in multiple databases at that point.
> Someday I (or someone else) should try to make the limit
> much larger.
>
Rob how would someone go about increasing the limit to 4 GB max? According to
the EDS docs, the limited it caused by C file routines used in the interpreter
internally?
Also, how is that PD-source going, almost ready to release it for community
testing?
> Regards,
> Rob Craig
> Rapid Deployment Software
> <a href="http://www.RapidEuphoria.com">http://www.RapidEuphoria.com</a>
>
Regards,
Vincent
10. Re: Storing Data for Web Site
Vincent wrote:
>
> Robert Craig wrote:
> >
> > A Euphoria database is pretty efficient. It stores
> > small integers in one byte, not four. The overhead
> > on sequences is also pretty low.
> > I think you'll run into trouble at around 2Gb.
> > You would have to store your data in multiple databases at that point.
> > Someday I (or someone else) should try to make the limit
> > much larger.
> >
>
> Rob how would someone go about increasing the limit to 4 GB max? According to
> the EDS
> docs, the limited it caused by C file routines used in the interpreter
> internally?
You'd think the limit would be 4GB, but I recall that some of the
C library routines will fail after 2GB.
Windows/Linux/FreeBSD have newer file routines that go beyond
4-byte file offsets. I would need to start using those.
EDS would also have to be adjusted to use greater than
4-byte offsets.
> Also, how is that PD-source going, almost ready to release it for community
> testing?
The new cooperative multitasking feature is working fine.
It was used in the game of Language War that I just won (expert level )
I'll release it pretty soon, maybe next week.
I'm just mulling over possible improvements,
and thinking of other demos I could write to show it off.
As you recall, it goes beyond what has been done already
in Language War, and other multitasking systems in User Contributions,
since each task gets its own separate call stack, program counter
and private variables. For example, a task can be buried many levels deep
in subroutine calls, and easily transfer control back and forth
with other tasks, while maintaining its own private data.
I have an example where two instances of quick sort run in parallel
with two instances of shell sort, and everything comes out correct in
the end.
Regards,
Rob Craig
Rapid Deployment Software
http://www.RapidEuphoria.com
11. Re: Storing Data for Web Site
Robert Craig wrote:
> Vincent wrote:
> > Robert Craig wrote:
>
> Windows/Linux/FreeBSD have newer file routines that go beyond
> 4-byte file offsets. I would need to start using those.
>
> EDS would also have to be adjusted to use greater than
> 4-byte offsets.
>
That should be as easy as changing this
use4byte_offsets()
to
use8byte_offsets()
right?! 8)
> The new cooperative multitasking feature is working fine.
> It was used in the game of Language War that I just won (expert level )
Shouldn't you be programming, not playing games?! Oh, testing. Gotcha. hehe.
Actually, I think I'll play some CS:S. I'm tired and need some vegetative
mind distraction...
> As you recall, it goes beyond what has been done already
> in Language War, and other multitasking systems in User Contributions...
That sounds wild. I can't wait to at least take a look at it, even if I don't
know exactly how it's going to benefit me in the long run. 8D
-=ck
"Programming in a state of EUPHORIA."
http://www.cklester.com/euphoria/
12. Re: Storing Data for Web Site
Robert Craig wrote:
>
> Vincent wrote:
> >
> > Robert Craig wrote:
> > >
> > > A Euphoria database is pretty efficient. It stores
> > > small integers in one byte, not four. The overhead
> > > on sequences is also pretty low.
> > > I think you'll run into trouble at around 2Gb.
> > > You would have to store your data in multiple databases at that point.
> > > Someday I (or someone else) should try to make the limit
> > > much larger.
> > >
> >
> > Rob how would someone go about increasing the limit to 4 GB max? According
> > to the EDS
> > docs, the limited it caused by C file routines used in the interpreter
> > internally?
>
> You'd think the limit would be 4GB, but I recall that some of the
> C library routines will fail after 2GB.
>
> Windows/Linux/FreeBSD have newer file routines that go beyond
> 4-byte file offsets. I would need to start using those.
> EDS would also have to be adjusted to use greater than
> 4-byte offsets.
>
Well if it isn't to difficult to use newer routines, that would be quite useful
for EDS (2x larger limit), and reading & writing huge files. Maybe you can
consider it for the next release?
> > Also, how is that PD-source going, almost ready to release it for community
> > testing?
>
> The new cooperative multitasking feature is working fine.
> It was used in the game of Language War that I just won (expert level )
>
So did I, except the Euphoria ship had 5 million units of energy instead of
50,000 8^D.
> I'll release it pretty soon, maybe next week.
> I'm just mulling over possible improvements,
> and thinking of other demos I could write to show it off.
>
You said that last week but ok 
. It's more possible for issues to arise when
implemented in the C backend.
> As you recall, it goes beyond what has been done already
> in Language War, and other multitasking systems in User Contributions,
> since each task gets its own separate call stack, program counter
> and private variables. For example, a task can be buried many levels deep
> in subroutine calls, and easily transfer control back and forth
> with other tasks, while maintaining its own private data.
> I have an example where two instances of quick sort run in parallel
> with two instances of shell sort, and everything comes out correct in
> the end.
>
I'll be interested what routines this cooperative tasking system comprises of,
and it's performance.
> Regards,
> Rob Craig
> Rapid Deployment Software
> <a href="http://www.RapidEuphoria.com">http://www.RapidEuphoria.com</a>
>
Regards,
Vincent
----------------------------------------------
___ __________ ___
/__/\ /__________\ |\ _\
\::\'\ //::::::::::\\ |'|::|
\::\'\ //:::_::::_:::\\ |'|::|
\::\'\ //::/ |::| \::\\ |'|::|
\::\'\ //::/ |::| \::\\|'|::|
\::\'\__//::/ |::| \::\|'|::|
\::\','/::/ |::| \::\\|::|
\::\_/::/ |::| \::\|::|
\::,::/ |::| \:::::|
\___/ |__| \____|
.``.
',,'
----------------------------------------------
13. Re: Storing Data for Web Site
> You'd think the limit would be 4GB, but I recall that some of the
> C library routines will fail after 2GB.
>
> Windows/Linux/FreeBSD have newer file routines that go beyond
> 4-byte file offsets. I would need to start using those.
> EDS would also have to be adjusted to use greater than
> 4-byte offsets.
Actually, at one point, I was looking into implementing variable-byte
offsets. That way you could squeeze an extra byte to two out of every
record. One wouldn't need to bloat the database with 8-byte offsets
for *every* record. One could also make use of only 2-byte offsets.
Typical 4 byte offsets would now be 5 bytes, but 8-byte offsets
(actually 9 bytes) would only be used when necessary.
14. Re: Storing Data for Web Site
Greg Haberek wrote:
>
> > You'd think the limit would be 4GB, but I recall that some of the
> > C library routines will fail after 2GB.
> >
> > Windows/Linux/FreeBSD have newer file routines that go beyond
> > 4-byte file offsets. I would need to start using those.
> > EDS would also have to be adjusted to use greater than
> > 4-byte offsets.
>
> Actually, at one point, I was looking into implementing variable-byte
> offsets. That way you could squeeze an extra byte to two out of every
> record. One wouldn't need to bloat the database with 8-byte offsets
> for *every* record. One could also make use of only 2-byte offsets.
> Typical 4 byte offsets would now be 5 bytes, but 8-byte offsets
> (actually 9 bytes) would only be used when necessary.
>
That seems less efficient than it would need to be. Why not use a similar
system to how Rob implements in compress/decompress, and 'only' allow
up to 7-byte offsets (or whatever). Frankly, it's hard to imagine needing
more than that big of a file (5-bytes would give you what, like 512gb--
"No one should ever need more than 640GB" :). That should give plenty of
room to grow.
It's actually pretty easy (on Windows, I've never tried on Linux) to use
large files. SetFilePointer actually uses two 4-byte integers to move around
in a file:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createfile.asp
I imagine there's a similar API on Linux. You'd likely break DOS
compatibility, but it'd probably be pretty rare that anyone would use such
big files, so there's probably not a whole lot of utility being lost.
Matt Lewis
15. Re: Storing Data for Web Site
One thing I think many people are overlooking is that files in DOS, Windows 9x,
and many Linux distros only allow files to be 2 Gb in size (whether due to the
kernel or filesystem). Windows 2000 and XP and BSD seem to allow for files larger
than 2 gigs and Linux (usually) has commands to change the maximum file size
(depends on filesystem type though). If any changes are made we need to be sure
that compatability isn't broken.
The Euphoria Standard Library project :
http://esl.sourceforge.net/
The Euphoria Standard Library mailing list :
https://lists.sourceforge.net/lists/listinfo/esl-discussion
