OpenEuphoria

1. Euphoria JSON parser

• Posted by ghaberek (admin) Mar 13, 2015
• 3250 views

Good news, everyone!

I have implemented an experimental JSON parser powered by JSMN (pronounced like 'jasmine'). I created a new branch for this in SCM named jsmn. I think I did a pretty good job of "Euphoriaizing" the functionality. I've done a lot of testing so far with JSON data from many sources and I have verified the JSON output with several online validators.

Basically, JSMN parses a JSON string into a series of tokens that simply describe the contents of string. The tokens contain the following values:

• J_TYPE (a JSON type constant, see below)
• J_START (start of the JSON value in the string)
• J_END (end of the JSON value in the string)
• J_SIZE (number of sub-tokens following this token)
• J_PARENT (index of this token's parent token)

We then take those tokens and assemble Euphoria objects such as maps, sequences, strings, and atoms.

These values are always stored in a sequence with their respective type:

• JSON_OBJECT (a key/value map)
• JSON_ARRAY (an sequence of values)
• JSON_STRING (a basic string)
• JSON_PRIMITIVE (a number or true/false/null)

Documentation is complete but may need to be touched up. I have not tested building on all platforms.

Example 1

-- 
-- Parse a raw JSON string into a sequence of tokens. 
-- 
include std/console.e 
include std/json.e 
 
sequence js = `{"key": "hello", "value": "world"}` 
 
object tokens = json:parse( js ) 
if atom( tokens ) then 
    -- token is an error number 
end if 
 
for i = 1 to length( tokens ) do 
     
    sequence t = tokens[i] 
 
    integer j_type   = t[J_TYPE] 
    integer j_start  = t[J_START] 
    integer j_end    = t[J_END] 
    integer j_size   = t[J_SIZE] 
    integer j_parent = t[J_PARENT] 
    sequence j_value = js[j_start..j_end] 
 
    display( "token [1]", {i} ) 
    display( "  j_type   = [1]", {j_type} ) 
    display( "  j_start  = [1]", {j_start} ) 
    display( "  j_end    = [1]", {j_end} ) 
    display( "  j_size   = [1]", {j_size} ) 
    display( "  j_parent = [1]", {j_parent} ) 
    display( "  j_value  = [1]", {j_value} ) 
 
end for 

Output

token 1 
  j_type   = 1 
  j_start  = 1 
  j_end    = 34 
  j_size   = 2 
  j_parent = 0 
  j_value  = {"key": "hello", "value": "world"} 
token 2 
  j_type   = 3 
  j_start  = 3 
  j_end    = 5 
  j_size   = 1 
  j_parent = 1 
  j_value  = key 
... 

Example 2

-- 
-- Parse a sequence of JSON tokens into a Euphoria object. 
-- 
include std/console.e 
include std/json.e 
include std/map.e 
  
object result = json:value( `{"key": "hello", "value": "world"}` ) 
if atom( result ) then 
    -- result is an error number 
end if 
 
map m = result[J_VALUE] 
 
object key   = map:get( m, "key" ) 
object value = map:get( m, "value" ) 
 
display( `key = "[1]", value = "[2]"`, 
    {key[J_VALUE],value[J_VALUE]} ) 

Output

key = "hello", value = "world" 

Example 3

-- 
-- Create a simple JSON object. 
-- 
include std/json.e 
 
-- a JSON_OBJECT is a map of key/value pairs 
object test = json:new(JSON_OBJECT, { 
    -- a JSON_OBJECT is a map of key/value pairs 
    -- values get converted to JSON_STRING automatically 
    {"key", "hello"}, 
    {"value", "world"} 
}) 
 
json:write(1, test) 

Output

{ 
    "key": "hello", 
    "value": "world" 
} 

Example 4

-- 
-- Create a complex JSON object. 
-- 
include std/json.e 
 
object test = json:new(JSON_OBJECT, { 
    -- a JSON_OBJECT is a map of key/value pairs 
    {"array", json:new(JSON_ARRAY, { 
        -- JSON_ARRAY is a sequence of values, which can be nested JSON_OBJECTs 
        json:new( JSON_OBJECT, { {"name","one"}, {"value",1} } ), 
        json:new( JSON_OBJECT, { {"name","two"}, {"value",2} } ), 
        json:new( JSON_OBJECT, { {"name","three"}, {"value",3} } ), 
        json:new( JSON_OBJECT, { {"name","four"}, {"value",4} } ), 
        json:new( JSON_OBJECT, { {"name","five"}, {"value",5} } ) 
    })} 
}) 
 
json:write(1, test) 

Output

{ 
    "array": [ 
        { 
            "value": 1, 
            "name": "one" 
        }, 
        { 
            "value": 2, 
            "name": "two" 
        }, 
        { 
            "value": 3, 
            "name": "three" 
        }, 
        { 
            "value": 4, 
            "name": "four" 
        }, 
        { 
            "value": 5, 
            "name": "five" 
        } 
    ] 
} 

-Greg

3. Re: Euphoria JSON parser

• Posted by ghaberek (admin) Mar 14, 2015
• 3305 views

It's a JSON parser. It parses JSON. It also allows you to create JSON objects in memory and output them as JSON text.

With this, you can make use of REST APIs that send and receive JSON data by combining this with http_get() and http_post() calls. Or you could just store some data in a portable human-readable format. Or support importing and exporting data that is compatible with other applications and services.

There have been several unsuccessfuly attempts at adding JSON to Euphoria. I've also run into my own roadblocks trying to get a good JSON parser written in Euphoria. Sometimes it's better to avoid reinventing the wheel and just seek out something that Just Works.

• XML to JSON in Euphoria (April 2007)
• JSON to euphoria (April 2010)
• json (August 2012)

There is also cjson in The Archive, which was added by "ras" in May 2012. That library probably works well but it has a few short-comings:

• It requires an external library. If the libraries provided are not compatible with the user's system, they will have to be recompiled accordingly (not a trivial task). I have compiled JSMN into the backend of Euphoria.
• It has a lot of functions to manipulate JSON objects in memory. I have implemented JSMN such that all of the JSON object manipulation is as "close to the bone" as possible in native Euphoria.
• It has not been updated since 2012 and most likely will not be. With JSMN built into the backend, we will pull updates from upstream when updating Euphoria.

I original started along the same vein as cjson: I compiled JSMN into a shared library and wrapped its two functions, then wrote all of the additional code to format Euphoria objects from the tokens it provides. However, given how small the library is, it was only a few hours work to get the code compiled into the backend so that there are no external dependancies.

This, I believe, is key to adding a lot of good features to Euphoria: zero external dependancies.

-Greg

5. Re: Euphoria JSON parser

• Posted by jimcbrown (admin) Oct 16, 2015
• 2832 views

I'm in favor.

7. Re: Euphoria JSON parser

• Posted by ghaberek (admin) Oct 16, 2015
• 2756 views

Right, for wrapping Mini-XML, I was thinking I could put all of the function addresses into a table and just peek them all from memory, get the pointer to that table with one machine_func() call, and then wrap the functions with define_c_func()/proc() as if I were wrapping a shared library. That would save me from having to burn through ~70 machine_func() constants in the backend. But, if I can just use define_c_func()/proc() to access backend functions directly, then I can just skip the machine_func() nonsense entirely. (And if that's true, why do we still have machine_func()? Just for legacy purposes?)

True, but JSMN is only two functions and my wrapper is only one function. It was easier to just expose that via machine_func() and be done with it. Most of the magic happens directly within json.e anyway. In fact, all of the "build a new JSON object and write it to a file" parts are 100% Euphoria using well-structured map objects.

-Greg

9. Re: Euphoria JSON parser

• Posted by jimcbrown (admin) Oct 16, 2015
• 2937 views

Does this work on Windoze though? Using -1 in place of open_dll("") works fine on nix because they all use the same dlsym() - which supports use of RTLD_DEFAULT and RTLD_NEXT in place of an dlopen handle. (See http://www.scs.stanford.edu/histar/src/pkg/uclibc/include/dlfcn.h and http://linux.die.net/man/3/dlsym for details on how this works.)

I can't find any documentation suggesting that GetProcAddress() on the stupid platform supports anything similar. I even looked through WINE sources ( http://source.winehq.org/source/dlls/ntdll/loader.c and http://source.winehq.org/source/dlls/kernel32/module.c ) but afaict WINE doesn't implement such behavior either.

Another reason this works is because on nix, both shared objects and executables are ELF objects, and you can dlopen() an executable with no problems. However, I've seen evidence that the stupid platform can't do this (e.g. http://stackoverflow.com/questions/19110747/loadlibrary-an-exe )

Huh?

The main reason we did it this way was for DOS support. Since DOS was dropped I've advocated for loading PCRE via a shared library instead of having it built in, but no one has put in the effort to actually separate it back out again.

That's a really clever idea!

Probably. A big part of this was that the define_c() stuff didn't work on DOS. (At least not this way. You could use define_c_func() to call a machine code routine, but it'd be a heck of a lot harder to call a backend routine in DOS this way. At the very least, you'd need a way to look up the memory address of the routine yourself, as there's no linker (ala dlopen/LoadLibrary) that can do it for you in the DOS version of Euphoria.)

11. Re: Euphoria JSON parser

• Posted by ghaberek (admin) Mar 19, 2017
• 2177 views

I'm not exactly sure how to answer that. My hope was that by adding it to the repository as a branch would allow it to be pulled into the master branch as part of a future release.

Ultimately we need somebody to do a release soon. It look like Shawn fixed the two outstanding issues in May. (Further discussions about releasing should continue in that thread.)

-Greg

13. Re: Euphoria JSON parser

• Posted by jmduro Mar 20, 2017
• 2058 views

These JSON sequences

constant JSON_DATA = { 
  "{\"key1\" : {\"key1.1\" : 2}, \"key2\" : 2}", 
  "[120,183,\"alpha\",\"beta\",\"gamma\"]", 
  "{\"age\" : 51.5, \"details\" : [120,183], \"embedded\" : { \"three\" : 3.0, \"arr\" : [1,2,3,\"alpha\",\"beta\",\"gamma\"]} }", 
  "{\"him\" : true, \"her\" : false, \"nothing\" : null }", 
  "{\"dog\" : {\"name\" : \"Rover\", \"gender\" : \"male\"}}", 
  "{\"configure\":[{\"access-control\":[{\"access-list\":[]}]},{\"slot\":[]},{\"system\":[{\"date-and-time\":[{\"sntp\":[{\"server\":[]}]}]},{\"syslog\":[]}]},{\"management\":[{\"login-user\":[]},{\"snmp\":[{\"community\":[]}]},{\"access\":[]},{\"tacacsplus\":[{\"group\":[]}]}]},{\"qos\":[{\"policer-profile\":[]},{\"shaper-profile\":[]},{\"wred-profile\":[]},{\"queue-block-profile\":[{\"queue\":[]}]},{\"queue-group-profile\":[{\"queue-block\":[]}]}]},{\"port\":[{\"l2cp-profile\":[]},{\"ethernet\":[]},{\"svi\":[]}]},{\"bridge\":[{\"port\":[]},{\"vlan\":[]}]},{\"flows\":[{\"classifier-profile\":[]},{\"flow\":[]}]},{\"router\":[{\"interface\":[{\"dhcp-client\":[]}]}]},{\"oam\":[{\"cfm\":[{\"maintenance-domain\":[{\"maintenance-association\":[{\"mep\":[]}]}]}]}]},{\"test\":[{\"rfc2544\":[{\"profile-name\":[]}]}]}]}" 
} 
 

are converted to this

result =  
.  [1] 
.  .  [1] "key1" 
.  .  [2] 
.  .  .  [1] 
.  .  .  .  [1] "key1.1" 
.  .  .  .  [2] 2 
.  [2] 
.  .  [1] "key2" 
.  .  [2] 2 
result =  
.  [1] 120 
.  [2] 183 
.  [3] "alpha" 
.  [4] "beta" 
.  [5] "gamma" 
result =  
.  [1] 
.  .  [1] "age" 
.  .  [2] 51.500000 
.  [2] 
.  .  [1] "details" 
.  .  [2] "x·" 
.  [3] 
.  .  [1] "embedded" 
.  .  [2] 
.  .  .  [1] 
.  .  .  .  [1] "three" 
.  .  .  .  [2] 3 
.  .  .  [2] 
.  .  .  .  [1] "arr" 
.  .  .  .  [2] 
.  .  .  .  .  [1] 1 
.  .  .  .  .  [2] 2 
.  .  .  .  .  [3] 3 
.  .  .  .  .  [4] "alpha" 
.  .  .  .  .  [5] "beta" 
.  .  .  .  .  [6] "gamma" 
result =  
.  [1] 
.  .  [1] "him" 
.  .  [2] "true" 
.  [2] 
.  .  [1] "her" 
.  .  [2] "false" 
.  [3] 
.  .  [1] "nothing" 
.  .  [2] "null" 
result =  
.  [1] 
.  .  [1] "dog" 
.  .  [2] 
.  .  .  [1] 
.  .  .  .  [1] "name" 
.  .  .  .  [2] "Rover" 
.  .  .  [2] 
.  .  .  .  [1] "gender" 
.  .  .  .  [2] "male" 
result =  
.  [1] 
.  .  [1] "configure" 
.  .  [2] 
.  .  .  [1] 
.  .  .  .  [1] 
.  .  .  .  .  [1] "access-control" 
.  .  .  .  .  [2] 
.  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "access-list" 
.  .  .  .  .  .  .  .  [2] "" 
.  .  .  [2] 
.  .  .  .  [1] 
.  .  .  .  .  [1] "slot" 
.  .  .  .  .  [2] "" 
.  .  .  [3] 
.  .  .  .  [1] 
.  .  .  .  .  [1] "system" 
.  .  .  .  .  [2] 
.  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "date-and-time" 
.  .  .  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  .  [1] "sntp" 
.  .  .  .  .  .  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  .  .  .  .  [1] "server" 
.  .  .  .  .  .  .  .  .  .  .  .  .  .  [2] "" 
.  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "syslog" 
.  .  .  .  .  .  .  .  [2] "" 
.  .  .  [4] 
.  .  .  .  [1] 
.  .  .  .  .  [1] "management" 
.  .  .  .  .  [2] 
.  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "login-user" 
.  .  .  .  .  .  .  .  [2] "" 
.  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "snmp" 
.  .  .  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  .  [1] "community" 
.  .  .  .  .  .  .  .  .  .  .  [2] "" 
.  .  .  .  .  .  [3] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "access" 
.  .  .  .  .  .  .  .  [2] "" 
.  .  .  .  .  .  [4] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "tacacsplus" 
.  .  .  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  .  [1] "group" 
.  .  .  .  .  .  .  .  .  .  .  [2] "" 
.  .  .  [5] 
.  .  .  .  [1] 
.  .  .  .  .  [1] "qos" 
.  .  .  .  .  [2] 
.  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "policer-profile" 
.  .  .  .  .  .  .  .  [2] "" 
.  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "shaper-profile" 
.  .  .  .  .  .  .  .  [2] "" 
.  .  .  .  .  .  [3] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "wred-profile" 
.  .  .  .  .  .  .  .  [2] "" 
.  .  .  .  .  .  [4] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "queue-block-profile" 
.  .  .  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  .  [1] "queue" 
.  .  .  .  .  .  .  .  .  .  .  [2] "" 
.  .  .  .  .  .  [5] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "queue-group-profile" 
.  .  .  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  .  [1] "queue-block" 
.  .  .  .  .  .  .  .  .  .  .  [2] "" 
.  .  .  [6] 
.  .  .  .  [1] 
.  .  .  .  .  [1] "port" 
.  .  .  .  .  [2] 
.  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "l2cp-profile" 
.  .  .  .  .  .  .  .  [2] "" 
.  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "ethernet" 
.  .  .  .  .  .  .  .  [2] "" 
.  .  .  .  .  .  [3] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "svi" 
.  .  .  .  .  .  .  .  [2] "" 
.  .  .  [7] 
.  .  .  .  [1] 
.  .  .  .  .  [1] "bridge" 
.  .  .  .  .  [2] 
.  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "port" 
.  .  .  .  .  .  .  .  [2] "" 
.  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "vlan" 
.  .  .  .  .  .  .  .  [2] "" 
.  .  .  [8] 
.  .  .  .  [1] 
.  .  .  .  .  [1] "flows" 
.  .  .  .  .  [2] 
.  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "classifier-profile" 
.  .  .  .  .  .  .  .  [2] "" 
.  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "flow" 
.  .  .  .  .  .  .  .  [2] "" 
.  .  .  [9] 
.  .  .  .  [1] 
.  .  .  .  .  [1] "router" 
.  .  .  .  .  [2] 
.  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "interface" 
.  .  .  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  .  [1] "dhcp-client" 
.  .  .  .  .  .  .  .  .  .  .  [2] "" 
.  .  .  [10] 
.  .  .  .  [1] 
.  .  .  .  .  [1] "oam" 
.  .  .  .  .  [2] 
.  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "cfm" 
.  .  .  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  .  [1] "maintenance-domain" 
.  .  .  .  .  .  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  .  .  .  .  [1] "maintenance-association" 
.  .  .  .  .  .  .  .  .  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  [1] "mep" 
.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  [2] "" 
.  .  .  [11] 
.  .  .  .  [1] 
.  .  .  .  .  [1] "test" 
.  .  .  .  .  [2] 
.  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  [1] "rfc2544" 
.  .  .  .  .  .  .  .  [2] 
.  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  [1] 
.  .  .  .  .  .  .  .  .  .  .  [1] "profile-name" 
.  .  .  .  .  .  .  .  .  .  .  [2] "" 

15. Re: Euphoria JSON parser

• Posted by CraigWelch Mar 28, 2017
• 2086 views

Thanks to the various replies - plenty there for me to work with. Appreciated.