1. Eunet or EuLibnet or TCP socket library (or socket.e)
- Posted by Spock May 16, 2016
- 2648 views
I have to write a custom invoice maker for our company. (They realized that the client-facing Excel or Word documents later handed to Finance to key into the accounting system were not a good idea). Anyway, I want to utilise a multi-client server system. My question is: Which communications library would suit my purposes best?
The Server will manage a small database of transactions. The Client will need to read various data items from the Server and send confirmed transactions to the Server. The size of the data strings sent to the Server will vary from eg 30 to 2000 bytes. The Server could return data between, say, 20 bytes to 20 Kbytes. Number of users could vary from 1 to 10. The basic requirements are:
- Simplicity
- Performance
- Robustness
Another factor to consider is Synchronous vs Asynchronous transmissions. Personally I would favour Synchronous since the Client code should be simpler. However, a problem with the Server could stop the Client dead in its tracks. So I'm just a bit cautious about committing to it.
Has someone had experience with this sort of project that could advise? TIA.
EDIT: I'm thinking that the Server could simply be a console app that can respond quickly to requests but the Client will always be a Windows GUI program.
EDIT2: Actually, a synchronous call with a timeout would solve the blocking issue. Looks like others have had the same issue. This solution uses UDP.
Spock
2. Re: Eunet or EuLibnet or TCP socket library (or socket.e)
- Posted by ghaberek (admin) May 18, 2016
- 2815 views
I have to write a custom invoice maker for our company. (They realized that the client-facing Excel or Word documents later handed to Finance to key into the accounting system were not a good idea). Anyway, I want to utilise a multi-client server system. My question is: Which communications library would suit my purposes best?
I would recommend ZeroMQ as the "best" solution for this type of communication. However, the current wrapper in The Archive is for version 2.0 (version 3.x and 4.x are much better) and it is incomplete and does not work correctly on Windows without some modification (needs '+' added to the define_c_func calls for CDECL). Ultimately, somebody needs to wrap the CZMQ API for Euphoria, but I've had trouble getting things build correctly on my system to even get started with that. But that's neither here nor there I guess.
The Server will manage a small database of transactions. The Client will need to read various data items from the Server and send confirmed transactions to the Server. The size of the data strings sent to the Server will vary from eg 30 to 2000 bytes. The Server could return data between, say, 20 bytes to 20 Kbytes. Number of users could vary from 1 to 10. The basic requirements are:
- Simplicity
- Performance
- Robustness
You can accomplish this via Core Sockets. I like to combine this with Serialization to send/receive Euphoria objects on the wire. I've created a demo for you here: Socket object demo.
Another factor to consider is Synchronous vs Asynchronous transmissions. Personally I would favour Synchronous since the Client code should be simpler. However, a problem with the Server could stop the Client dead in its tracks. So I'm just a bit cautious about committing to it.
With the data sizes you're talking about, a synchronous server shouldn't cause any perceivable delays. You can read/write megabytes of data on the database in a fraction of a second. My biggest concern is that Euphoria is not very resilient when it comes to crashes or errors. If something causes a hard error (an invalid data type, for instance) then the server would crash and you'd have to restart it. Now, there are ways to get around that, but when the server exits, your existing TCP sockets would be dropped and need to be reconnected, and your client application would have to somehow negotiate that.
You could accomplish a pseudo-asynchronous server by combining the select function with Multi-tasking where you have one task to listen for incoming connections, that would then pass of those connections to a separate sub-task that talks to the client. Using select allows you determine if a socket has data waiting and, if not, continue on to other tasks. I could mock up a demo for you if you'd like.
EDIT: I'm thinking that the Server could simply be a console app that can respond quickly to requests but the Client will always be a Windows GUI program.
That would be the correct approach to take. You could also even implement the console app as an actual service. There is a Windows Services Library in The Archive that might be of some help here.
EDIT2: Actually, a synchronous call with a timeout would solve the blocking issue. Looks like others have had the same issue.
I wouldn't be worried about people having issues in the browser with AJAX calls. That's completely different from how core sockets are implemented in Euphoria. The JavaScript engines used in web browsers (and Node.js) are designed to be natively asynchronous. Euphoria is not. You will not have the same problems they are having.
This solution uses UDP.
Wow. Nope. DO NOT use UDP for this. UDP is inherently unreliable. You could lose packets on the wire and you would never know. It's not something that's appropriate for this type of communication. Even if you did write some additional code to determine when packets were lost and request re-transmission from the other end, all you're doing is reinventing the wheel when all of these problems are corrected by TCP. Just use TCP.
Has someone had experience with this sort of project that could advise? TIA.
I have quite a lot of experience with this. My best advice would be to avoid writing your own data server and also avoid writing your own TCP data protocol. These types of low-level "inventions" are prone to human error and can lead to lots and lots of feature bloat or spaghetti code over time. What's most concerning, is that this needs to be a production system for several people, but you'd effectively be developing it in the field. You can only ever do so much development testing, and as soon as you have your first server crash in production, you'd have to develop and implement a fix to keep that from happening again, in real time.
My ultimate recommendation would be to use a MySQL or Postgres server and Jeremy's EDBI library to connect your clients to it. This solves all of your server-related problems and you won't have to write a single bit of server code. No protocols to invent, no worries about synchronicity, no server-side error recovery problems, none of that. Just design your client to connect to the database and manage the shared data from there.
-Greg
3. Re: Eunet or EuLibnet or TCP socket library (or socket.e)
- Posted by Spock May 18, 2016
- 2528 views
Hi Greg, Thanks for the very informative response. It has helped clarify my thinking on the matter.
I have to write a custom invoice maker... a multi-client server system.. Which communications library would suit my purposes best?
The Server will manage a small database of transactions. The Client will need to read various data items from the Server and send confirmed transactions to the Server. The size of the data strings sent to the Server will vary from eg 30 to 2000 bytes. The Server could return data between, say, 20 bytes to 20 Kbytes. Number of users could vary from 1 to 10. The basic requirements are:
- Simplicity
- Performance
- Robustness
You can accomplish this via Core Sockets. I like to combine this with Serialization to send/receive Euphoria objects on the wire. I've created a demo for you here: Socket object demo.
I like the idea of using core sockets since it seems simple enough and I won't have to install any other software and I am obliged to have a better technical understanding. There is just the synchronicity/connectivity issue to solve.
With the data sizes you're talking about, a synchronous server shouldn't cause any perceivable delays. You can read/write megabytes of data on the database in a fraction of a second. My biggest concern is that Euphoria is not very resilient when it comes to crashes or errors. If something causes a hard error (an invalid data type, for instance) then the server would crash and you'd have to restart it. Now, there are ways to get around that, but when the server exits, your existing TCP sockets would be dropped and need to be reconnected, and your client application would have to somehow negotiate that.
The client-server communications will be quite sparse (and the user load light). At client initialisation the previous few months transactions (for that user) will be loaded into the client as a single read-only table. When a new transaction is created in the client it gets sent to the server which then confirms the event. That is really the core of the transmissions. So on the client side I think it would work as a synchronous socket call that times out. The dropped TCP connection could be managed by preceding each call with a check that the connection is still up. Can select be used to check connectivity? I presume so since socket.e refers to ERR_NOTCONN indicating that I can test for that condition.
You could accomplish a pseudo-asynchronous server by combining the select function with Multi-tasking where you have one task to listen for incoming connections, that would then pass of those connections to a separate sub-task that talks to the client. Using select allows you determine if a socket has data waiting and, if not, continue on to other tasks.
I would prefer to not use tasks, if at all possible. Would some form of polling work instead?
My best advice would be to avoid writing your own data server and also avoid writing your own TCP data protocol. These types of low-level "inventions" are prone to human error and can lead to lots and lots of feature bloat or spaghetti code over time. What's most concerning, is that this needs to be a production system for several people, but you'd effectively be developing it in the field. You can only ever do so much development testing, and as soon as you have your first server crash in production, you'd have to develop and implement a fix to keep that from happening again, in real time.
I understand what you're saying.. In my case the main user/beneficiary sits in the desk just opposite me so any crashes during the initial production will be well contained prior to the wider deployment.
Spock
4. Re: Eunet or EuLibnet or TCP socket library (or socket.e)
- Posted by ghaberek (admin) May 19, 2016
- 2530 views
The client-server communications will be quite sparse (and the user load light). At client initialisation the previous few months transactions (for that user) will be loaded into the client as a single read-only table. When a new transaction is created in the client it gets sent to the server which then confirms the event. That is really the core of the transmissions. So on the client side I think it would work as a synchronous socket call that times out. The dropped TCP connection could be managed by preceding each call with a check that the connection is still up.
I don't think you need to keep the TCP connections open between the clients and server. This is the process I imagine you'd take:
| # | Client | Server |
|---|---|---|
| 1 | Connect to server | |
| 2 | Accept connection from client | |
| 3 | Send request for new transaction ID | |
| 4 | Receive request for new transaction ID | |
| 5 | Send new transaction ID to client | |
| 6 | Receive new transaction ID | |
| 7 | Send transaction ID and data | |
| 8 | Receive transaction ID and data | |
| 9 | Send SUCCESS or FAILED status | |
| 10 | Close client socket | |
| 11 | Receive status from server | |
| 12 | Close local socket |
This is how a lot of simple TCP protocols, like HTTP, work: the connection only lives for the duration of the request. More advanced protocols (especially databases) utilize connection pooling, which allows the programmer to design the application as if the connection were always open, but underneath, the connection can idle out, time out, error out, create multiple channels, etc. without the programmer ever knowing. But we don't really have that luxury with core sockets.
Can select be used to check connectivity? I presume so since socket.e refers to ERR_NOTCONN indicating that I can test for that condition.
That's exactly what select() is for: to poll the state of one or more sockets and act accordingly. However, as stated, I would not recommend relying on the TCP connection to remain open. It's a lot easier if your connections have the same lifetime as your transactions.
I would prefer to not use tasks, if at all possible. Would some form of polling work instead?
You certainly don't need to use tasks. I use tasks to provide a stronger separation of concerns in my networking model. That is, there the primary task is listening for new connections and additional tasks (or one task processing a queue) is handling the clients' data in chunks at a time. But since your data sizes are so small, I doubt you'll need to break things down into chunks.
-Greg
5. Re: Eunet or EuLibnet or TCP socket library (or socket.e)
- Posted by Spock May 19, 2016
- 2517 views
The client-server communications will be quite sparse (and the user load light). At client initialisation the previous few months transactions (for that user) will be loaded into the client as a single read-only table. When a new transaction is created in the client it gets sent to the server which then confirms the event. That is really the core of the transmissions. So on the client side I think it would work as a synchronous socket call that times out. The dropped TCP connection could be managed by preceding each call with a check that the connection is still up.
I don't think you need to keep the TCP connections open between the clients and server. This is the process I imagine you'd take:
| # | Client | Server |
|---|---|---|
| 1 | Connect to server | |
| 2 | Accept connection from client | |
| 3 | Send request for new transaction ID | |
| 4 | Receive request for new transaction ID | |
| 5 | Send new transaction ID to client | |
| 6 | Receive new transaction ID | |
| 7 | Send transaction ID and data | |
| 8 | Receive transaction ID and data | |
| 9 | Send SUCCESS or FAILED status | |
| 10 | Close client socket | |
| 11 | Receive status from server | |
| 12 | Close local socket |
This is how a lot of simple TCP protocols, like HTTP, work: the connection only lives for the duration of the request..
-Greg
This is even better by making each event independent. In the above table there is a user delay of a few minutes between items 6 & 7. So I could then break the transmission into 2 distinct chunks. And the server loop itself will be much simpler with no past/future to worry about - only the here and now.
Edit - In testing the client/server I discovered there is a buffer limit (in my case 1023 bytes). So files of arbitrary size can't be sent all in one go. Is there a standard solution to this?
Spock
6. Re: Eunet or EuLibnet or TCP socket library (or socket.e)
- Posted by acEduardo May 27, 2016
- 2403 views
I've read the Euphoria source code, it uses a fixed length buffer for the socket IO, of 1024 bytes, and sacrificing the last byte. I had too perceived this 1023 limitation while developing a scgi server.
Seems to me the solution is, first send a "start file trans" packet, containing the size and checksum of the data, then send various "file-chunk" packets, each one carrying a "chunk id", maybe a "file id".
7. Re: Eunet or EuLibnet or TCP socket library (or socket.e)
- Posted by Spock May 31, 2016
- 2407 views
I've read the Euphoria source code, it uses a fixed length buffer for the socket IO, of 1024 bytes, and sacrificing the last byte. I had too perceived this 1023 limitation while developing a scgi server.
Seems to me the solution is, first send a "start file trans" packet, containing the size and checksum of the data, then send various "file-chunk" packets, each one carrying a "chunk id", maybe a "file id".
I mostly agree with this. This guy has a similar solution but using only the length prefix. I just wish there was a shrink-wrapped solution in Euphoria available for download.
Spock
8. Re: Eunet or EuLibnet or TCP socket library (or socket.e)
- Posted by Spock Jun 03, 2016
- 2381 views
I have to write a custom invoice maker for our company. (They realized that the client-facing Excel or Word documents later handed to Finance to key into the accounting system were not a good idea). Anyway, I want to utilise a multi-client server system.
With the data sizes you're talking about, a synchronous server shouldn't cause any perceivable delays. You can read/write megabytes of data on the database in a fraction of a second...
I added message framing and a receive loop to the experimental code. The basic socket code always could send an arbitrary length data. My receive function can now receive arbitrary lengths - Yippee ! You are right about the speed. I tested the throughput (2-way) up to 2,000,000 bytes and it takes only 1/6 second running the server and client on a single computer. Over an actual network the speed will drop but my modest app would be happy with even 5% of that rate.
You could accomplish a pseudo-asynchronous server by combining the select function with Multi-tasking ..
I have been reading about Finite State Machines. They seem well suited for solving the problem of parallel but independent actions. Once the single instance server core is stable I'll add an FSM to allow multiple connections in parallel. Muhahaha...
Spock
9. Re: Eunet or EuLibnet or TCP socket library (or socket.e)
- Posted by Spock Jun 09, 2016
- 2319 views
You could accomplish a pseudo-asynchronous server by combining the select function with Multi-tasking ..
I have been reading about Finite State Machines. They seem well suited for solving the problem of parallel but independent actions. Once the single instance server core is stable I'll add an FSM to allow multiple connections in parallel. Muhahaha...
Spock
Ok. FSM has been added and seems to be working ok for multiple connections. What I now need is some guidance on timeouts for receiving large files. The basic process is
1. Connect to other machine
2. Receive message prefix indicating total length of the data
3. Receive next block of data
4. repeat 3 until data buffer reaches the length value
5. Close connection
There are several places for timeouts.
i) When the connection is made till the prefix is received
ii) Between each subsequent block of data
iii) The total time to receive all the data (optional?)
Are there any suggestions as to how long the maximum length of each stage chould be? I am also thinking that (barring network delays) the timeouts should be based somewhat on the number of active connections.
Spock
10. Re: Eunet or EuLibnet or TCP socket library (or socket.e)
- Posted by ghaberek (admin) Jun 09, 2016
- 2279 views
Are there any suggestions as to how long the maximum length of each stage chould be? I am also thinking that (barring network delays) the timeouts should be based somewhat on the number of active connections.
Maybe have a look at execute_request() in std/net/http.e to see how it handles timeouts.
-Greg
11. Re: Eunet or EuLibnet or TCP socket library (or socket.e)
- Posted by Spock Jun 09, 2016
- 2249 views
Are there any suggestions as to how long the maximum length of each stage chould be? I am also thinking that (barring network delays) the timeouts should be based somewhat on the number of active connections.
Maybe have a look at execute_request() in std/net/http.e to see how it handles timeouts.
-Greg
Thanks for the reference. In that example the default timeout for the HTTP send-receive is 15 seconds but can be altered by the user. Other timeouts I have seen range from, eg, 30 seconds upwards. I'll use those figures as starting estimates and refine the concept over time.
Spock
12. Re: Eunet or EuLibnet or TCP socket library (or socket.e)
- Posted by Spock Jun 15, 2016
- 2184 views
Are there any suggestions as to how long the maximum length of each stage chould be? I am also thinking that (barring network delays) the timeouts should be based somewhat on the number of active connections.
Maybe have a look at execute_request() in std/net/http.e to see how it handles timeouts.
-Greg
Thanks for the reference. In that example the default timeout for the HTTP send-receive is 15 seconds but can be altered by the user. Other timeouts I have seen range from, eg, 30 seconds upwards. I'll use those figures as starting estimates and refine the concept over time.
Spock
I'm pleased with the server's performance now. I can firebomb it with dozens of simultaneous connections sending random strings (1k to 100k) of data and it handles them all without dropping any.. except when the string is less than the buffer size (Maximum Transmission Unit) thanks to Nagle's algorithm which coalesces subsequent "sends" until the buffer is full and then actually sends the data out. In my case it just times out the connection. I haven't been able to fix it and not sure where to start. I can sidestep the issue by ensuring that each string of data has a minimum size but that's really just a hack.
Spock
Forked into: Libharu dll trouble
