1. Server-Sent Events

Server-Sent Events


I haven’t seen much discussion here about Server-Sent Events, which is not surprising given that the number of people doing CGI processing seems to be fairly small. There’s but one mention of this, by acEduardo in this thead .

I’ve been playing with this, successfully, to drive progress bars, and to populate log sections on web pages. I’m posting the results in case anyone else has a need for this function.

First, a few words on what it is. A short chronology, which many can skip.

  • The first web pages were completely static. You entered a url, and back came a web page. Embedded in that page would be hyperlinks, images, video, sound … but a page was a page.
  • With the development of CGI functionality, the ability to create the page on the fly allowed for the model in common use, including the page on which you’re reading this now, whereby the url included attributes describing how the page should be constructed. This typically involved looking up a database, and assembling the page with static elements and constructed elements based on processing the database results. As well as the base url, additional query options are sent as part of a GET or POST.
  • One problem with this is that as page sizes got larger, the overhead in asking for and receiving a huge page to make a minor change became cumbersome. Javascript came to the rescue, with the AJAX function. Javascript is of course a programming language embedded in the HTML of the web page. An AJAX request is one that is made, to a url, on behalf of just one element of the page. This request will also involve query components.
    • A good example of AJAX is when you start filling in a form field, and based on what you have typed the field is populated (or a drop-down is populated) with suggestions. After each keystroke, an AJAX request is sent which returns all appropriate words starting with that (or those) characters.
  • In all of this the communications were initiated by the browser. The server could respond to any kind of request, but could never initiate a communication. To implement online chat, for example, in which case both parties send messages via a server, it’s necessary for the server to advise client B that there is a new message from client A. The traditional way has been by way of polling. Every interval the client page would, with an AJAX request, ask the server if there were any new data. This is certainly better than refreshing an entire page every few seconds, but the constant polling has its own undesirable overhead.
  • That’s where Server-Set Events come in. A Javascript object is created that initiates a listener. When the server has something to say, it sends data to the listener, which has a function attached to it which will (perhaps after processing) put the new information somewhere on the web page. It might be the latest chat message. Or an updated stock ticker.

Implementation

Javascript


This is example code, updating a progress bar (“test”) as well as final status in a field (“Result3”). Changing the value in “test” will update the length of the progress bar.

 
{view:"button", id:"test_progress", value:"Test Progress", type:"form", autowidth:true, 
     click: function(){ 
                                var source = new EventSource("./find_fix_tables.eui?action=test_progress"); 
				source.onmessage = function(event) { 
			        .setValue(event.data * 1)}; // turn string into number 
 
			        source.onopen = function(event) { 
				          console.log("OPEN socket"); 
				          $ $("test").show(); 
				          $ $("Result3").setHTML("Processing")} 
 
				source.onerror = function(event) { 
				          console.log("ERROR - CLOSING socket"); 
					  source.close();  
					  $ $("test").hide(); 
					  $ $("Result3").setHTML("Completed")} 
			         } 
}, 
// I have had to write '$ $("test")' because if the two dollar signs are adjacent the 
// forum formatting code strips each such item out. 

Euphoria


Here is the server code. For testing, it’s just a loop counting from 1 to 100, advising the client on each iteration.

Notes:

  • The no-cache header, recommended in all the documentation for Server-Set Events, prevents this from working. It’s commented out.
  • The usual Content-Type header, such as ‘text/html’, ’application/json’ etc. is now ‘text/event-stream’. This must be followed by two blank lines. With normal text headers, you need *at least* two, and can have more. In this case however you need *exactly* two.
  • The data must be in the format ‘data:your_data’. In this case the loop counter is used, so the result is, form example ‘data: 5’. The Javascript receiver will parse only lines with ‘data: …’ and use the value of 5.
  • Most CGI scripts written in Euphoria have the headers and final ‘puts()’ at the end of the program. The program then terminates. That serves to send the output to Apache or other web server. In this case, there is no termination, so it’s necessary to insert a flush() statement to ensure that the data are sent to Apache.
--******************************************** 
--*      function test_progress()            * 
--******************************************** 
function test_progress() 
--puts(1,"Cache-Control: no-cache") -- This header would prevent the function from working. 
puts(1,"Content-Type: text/event-stream\n\n") 
for i = 1 to 100 do 
		sleep(0.1) 
		puts(1, sprintf("data: %d \n\n", i)) -- must have the two newlines 
		flush(1) 
end for 
puts(1, "Status: 415 OK") 
return {"Finished test"} 
end function 

That’s it. It works. I’m happy to answer any questions or provide any help.

Craig

new topic     » topic index » view message » categorize

2. Re: Server-Sent Events

CraigWelch said...
--******************************************** 
--*      function test_progress()            * 
--******************************************** 
function test_progress() 
--puts(1,"Cache-Control: no-cache") -- This header would prevent the function from working. 
puts(1,"Content-Type: text/event-stream\n\n") 
for i = 1 to 100 do 
		sleep(0.1) 
		puts(1, sprintf("data: %d \n\n", i)) -- must have the two newlines 
		flush(1) 
end for 
puts(1, "Status: 415 OK") 
return {"Finished test"} 
end function 

I think the problem is that you're not ending your Cache-Control header with the expected "\r\n" terminator.

So this code:

puts(1,"Cache-Control: no-cache") 
puts(1,"Content-Type: text/event-stream\n\n") 

Will output this text, which is incorrect:

Cache-Control: no-cacheContent-Type: text/event-stream\n\n 

The correct code would look like this:

puts(1,"Cache-Control: no-cache\r\n") 
puts(1,"Content-Type: text/event-stream\r\n") 
puts(1,"\r\n") 

And the output looks like this:

Cache-Control: no-cache\r\n 
Content-Type: text/event-stream\r\n 
\r\n 

See: http://stackoverflow.com/a/5757349/2300395

-Greg

new topic     » goto parent     » topic index » view message » categorize

3. Re: Server-Sent Events

You're absolutely right.

I've been using "\n\n" with the "Content-Type" header for years. The earliest sample from Rob Craig had that. In the example above, the "Cache-Control" now works with "\r\n" but does not work with "\n\n" (as I had at first tried). However "\n\n" still works with the "Content-Type" header. However, for the sake of good practice (as per the StackOverflow article you pointed me to), it shall be "\r\n" for all my CGI from now on.

Many thanks,

Craig

new topic     » goto parent     » topic index » view message » categorize

4. Re: Server-Sent Events

CraigWelch said...

That’s it. It works. I’m happy to answer any questions or provide any help.

I'm probably missing something cause Firefox allows me only one possibility: to save the source file. It does not execute as with other source code on the same server.

Here is my code:

include std/os.e 
include std/io.e 
 
--********************************************  
--*      function test_progress()            *  
--********************************************  
function test_progress()  
  puts(1,"Cache-Control: no-cache\r\n")  
  puts(1,"Content-Type: text/event-stream\r\n")  
  puts(1,"\r\n") 
  for i = 1 to 100 do  
    sleep(0.1)  
    puts(1, sprintf("data: %d \n\n", i)) -- must have the two newlines  
    flush(1)  
  end for  
  puts(1, "Status: 415 OK")  
  return {"Finished test"}  
end function  
 
test_progress() 

Jean-Marc

new topic     » goto parent     » topic index » view message » categorize

5. Re: Server-Sent Events

Problem seems to be the content-type. Following code does not the job as expected but executes (EU4.1 on Linux Debian 64):

include std/os.e 
include std/io.e 
 
puts(1,"Cache-Control: no-cache\n")  
puts(1, "Content-Type: text/plain\n\n")  
--  puts(1,"Content-Type: text/event-stream\n\n")  
for i = 1 to 100 do  
  sleep(0.1)  
  puts(1, sprintf("data: %d \n\n", i)) -- must have the two newlines  
  flush(1)  
end for  
puts(1, "Status: 415 OK")  

Jean-Marc

new topic     » goto parent     » topic index » view message » categorize

6. Re: Server-Sent Events

It is not that simple: it works with Chrome 58 and not with Firefox ESR 45.9 (default on Debian Jessie).

new topic     » goto parent     » topic index » view message » categorize

7. Re: Server-Sent Events

Same behaviour with latest Firefox ESR 52.1 64 on Debian Jessie with EU4.1. Something must be missing to allow server-side events with Firefox that is installed by default in Chrome.

new topic     » goto parent     » topic index » view message » categorize

8. Re: Server-Sent Events

That's interesting - Firefox has apparently supported Server-Side Events since 6.0.

Does this thread or this one help?

Are you looking at the output with an empty browser, or do you have some Javascript code to establish a receiver? If so, can you post that code?

new topic     » goto parent     » topic index » view message » categorize

9. Re: Server-Sent Events

I'm just running an empty browser with code published above.

Both threads didn't help. EU code complies to the first one and as the script is not executed, I don't know how to send an event before it should run.

Jean-Marc

new topic     » goto parent     » topic index » view message » categorize

10. Re: Server-Sent Events

I'd adapted a example I found in PHP, and it simply works.

sse.html:

<!DOCTYPE html> 
<html> 
<head> 
  <title> SSE Example </title> 
  <meta charset="utf-8" /> 
</head> 
<body> 
  <script> 
    var source = new EventSource('sse.ex'); 
    source.onmessage = function(e) { 
      document.body.innerHTML += e.data + '<br>'; 
    }; 
  </script> 
</body> 
</html> 

sse.ex:

include std/datetime.e 
 
constant months = 
   {"Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"} 
 
constant days_week = 
   {"Sun","Mon","Tue","Wed","Thu","Fri","Sat"} 
 
export enum year, month, day, hour, minute, second, weekday 
 
export function now() 
   sequence _now = now_gmt() 
   _now &= weeks_day(_now) 
   return _now 
end function 
 
export function timestamp (sequence _date, sequence timezone = "UTC", integer offset = 0) 
   integer  i = 1 
   sequence weekday_name = days_week[_date[7]], 
            month_name = months[_date[2]] 
   return sprintf("%s, %02d %s %04d %02d:%02d:%02d %s", 
      {weekday_name, _date[3], month_name, _date[1], _date[4], _date[5], _date[6], timezone} ) 
end function 
 
puts (1, 
    "Content-Type: text/event-stream\r\n" & 
    "Cache-Control: no-cache\r\n" & 
    "\r\n") 
 
printf (1, 
    "id: %d\n" & 
    "data: %s\n", 
    { time(), timestamp(now()) }) 
new topic     » goto parent     » topic index » view message » categorize

11. Re: Server-Sent Events

There must be something different between your configuration and mine: I get a blank page with only the right title on both Firefox and Google. I use Debian 8.6 64-bit with OpenEuphoria 4.1.0 beta 2 64-bit.

Jean-Marc

new topic     » goto parent     » topic index » view message » categorize

12. Re: Server-Sent Events

Maybe some Apache2 module?

Here are mine:

ls /etc/apache2/mods-enabled/ 
access_compat.load  authn_core.load  autoindex.load  env.load          negotiation.conf  setenvif.conf 
actions.conf        authn_file.load  cgi.load        filter.load       negotiation.load  setenvif.load 
actions.load        authz_core.load  deflate.conf    mime.conf         php5.conf         status.conf 
alias.conf          authz_host.load  deflate.load    mime.load         php5.load         status.load 
alias.load          authz_user.load  dir.conf        mpm_prefork.conf  reqtimeout.conf 
auth_basic.load     autoindex.conf   dir.load        mpm_prefork.load  reqtimeout.load 

new topic     » goto parent     » topic index » view message » categorize

13. Re: Server-Sent Events

I get the same blank page on RaspberryPy 3.

new topic     » goto parent     » topic index » view message » categorize

14. Re: Server-Sent Events

I'm using Lighttpd and running Euphoria scripts as CGI.

To put it to work: lighttpd -f ./config/lighttpd-something.conf -D

Example configuration (save it as lighttpd-something.conf):

server.modules = ( 
  "mod_cgi", 
  "mod_simple_vhost" 
) 
server.document-root = "/home/aceduardo/apps/lighttpd/base/www/" 
server.port = 8080 
server.bind = "127.0.0.1" 
server.pid-file = "/home/aceduardo/apps/lighttpd/running/server.pid" 
server.errorfile-prefix = "/home/aceduardo/apps/lighttpd/base/status/status-" 
server.upload-dirs = ( "/home/aceduardo/apps/lighttpd/base/uploads/" ) 
 
server.network-backend = "linux-sendfile" 
server.follow-symlink = "enable" 
 
dir-listing.activate = "enable" 
dir-listing.encoding = "utf-8" 
 
index-file.names = ( 
  "index.xhtml", 
  "index.html", 
  "index.xml", 
  "index.il", 
  "index.ex", 
  "index.pl", 
  "index.cgi", 
  "index.sh", 
) 
 
simple-vhost.server-root = "/home/aceduardo/apps/lighttpd/base/" 
simple-vhost.default-host = "www" 
simple-vhost.document-root = "public" 
 
url.access-deny = ( "~", ".inc", ".cfg", ".err", ".log", ".e" ) 
 
cgi.assign = ( 
  ".sh"  => "/bin/sh", 
#-- Most relevant line: 
  ".ex"  => "/home/aceduardo/apps/euphoria41/bin/eui",  
#-- Shrouded Euphoria scripts as CGI, runs faster:   
  ".il"  => "/home/aceduardo/apps/euphoria41/bin/eub",  
  ".pl"  => "/usr/bin/perl", 
  ".cgi" => "", 
) 
 
new topic     » goto parent     » topic index » view message » categorize

15. Re: Server-Sent Events

No change, still blank page. It does not try to execute the script.

Here is my configuration:

root@raspberrypi:/home/pi# ls -lR apps/ 
apps/: 
total 4 
drwxrwxrwx 4 pi pi 4096 mai   18 12:59 lighttpd 
 
apps/lighttpd: 
total 8 
drwxrwxrwx 5 pi pi 4096 mai   18 13:00 base 
drwxrwxrwx 2 pi pi 4096 mai   18 13:16 running 
 
apps/lighttpd/base: 
total 12 
drwxrwxrwx 2 pi pi 4096 mai   18 13:00 status 
drwxrwxrwx 2 pi pi 4096 mai   18 13:00 uploads 
drwxrwxrwx 2 pi pi 4096 mai   18 13:09 www 
 
apps/lighttpd/base/status: 
total 0 
 
apps/lighttpd/base/uploads: 
total 0 
 
apps/lighttpd/base/www: 
total 8 
-rwxrwxrwx 1 root root 935 mai   12 16:02 sse.ex 
-rwxrwxrwx 1 root root 294 mai   12 16:01 sse.html 
 
apps/lighttpd/running: 
total 0 
root@raspberrypi:/home/pi# cat ./config/lighttpd.conf  
server.modules = (  
  "mod_cgi",  
  "mod_simple_vhost"  
)  
server.document-root = "/home/pi/apps/lighttpd/base/www/"  
server.port = 80  
server.bind = "127.0.0.1"  
server.pid-file = "/home/pi/apps/lighttpd/running/server.pid"  
server.errorfile-prefix = "/home/pi/apps/lighttpd/base/status/status-"  
server.upload-dirs = ( "/home/pi/apps/lighttpd/base/uploads/" )  
  
server.network-backend = "linux-sendfile"  
server.follow-symlink = "enable"  
  
dir-listing.activate = "enable"  
dir-listing.encoding = "utf-8"  
  
index-file.names = (  
  "index.xhtml",  
  "index.html",  
  "index.xml",  
  "index.il",  
  "index.ex",  
  "index.exu",  
  "index.pl",  
  "index.cgi",  
  "index.sh",  
)  
  
simple-vhost.server-root = "/home/pi/apps/lighttpd/base/"  
simple-vhost.default-host = "www"  
simple-vhost.document-root = "public"  
  
url.access-deny = ( "~", ".inc", ".cfg", ".err", ".log", ".e" )  
  
cgi.assign = (  
  ".sh"  => "/bin/sh",  
#-- Most relevant line:  
  ".ex"  => "/usr/local/euphoria-4.1.0-RaspberryPi/bin/eui",   
  ".exu"  => "/usr/local/euphoria-4.1.0-RaspberryPi/bin/eui",   
#-- Shrouded Euphoria scripts as CGI, runs faster:    
  ".il"  => "/usr/local/euphoria-4.1.0-RaspberryPi/bin/eub",   
  ".pl"  => "/usr/bin/perl",  
  ".cgi" => "",  
) root@raspberrypi:/home/pi# lighttpd -f ./config/lighttpd.conf -D 
2017-05-18 13:19:23: (log.c.164) server started  
2017-05-18 13:19:23: (server.c.1045) WARNING: unknown config-key: url.access-deny (ignored)  

If I point directly to http://localhost/sse.ex, I can see an error in apps/lighttpd/base/www:

/home/pi/apps/lighttpd/base/www/sse.ex:1 
<0052>:: can't find 'std/datetime.e' in any of ... 
	/home/pi/apps/lighttpd/base/www 
	/home/pi/apps/lighttpd/base/www/sse.ex 
 
include std/datetime.e  
                       ^ 

EUDIR is declared for root and for the user account (pi).

Jean-Marc

new topic     » goto parent     » topic index » view message » categorize

16. Re: Server-Sent Events

I guess the result of following command is decisive:

# grep EUDIR ~/.bashrc ~/.profile ~/.bash_profile ~/.bash_login /etc/profile /etc/environment /etc/bash.bashrc 

Here is what I get:

/root/.bashrc:export EUDIR=/usr/local/euphoria-4.1.0-RaspberryPi 
grep: /root/.bash_profile: Aucun fichier ou dossier de ce type 
grep: /root/.bash_login: Aucun fichier ou dossier de ce type 
/etc/profile:EUDIR="/usr/local/euphoria-4.1.0-RaspberryPi" 
/etc/profile:export EUDIR 
/etc/environment:EUDIR=/usr/local/euphoria-4.1.0-RaspberryPi 

Jean-Marc

new topic     » goto parent     » topic index » view message » categorize

Search



Quick Links

User menu

Not signed in.

Misc Menu