1. "dumping" a map

Hi,

I'm new to OE and learning the ropes as it were. I've written a few simple programs, and I'm now exploring maps, which I want to use ultimately for a largish personal project that I have.

So, in my tests I want to be able to dump a text representation of a complex map, which includes nested maps, and none of the map functions in section 8.32 of the manual allow me to do this. I therefore would like to write a function that reads all key/value pairs in order to display them on screen, and calls itself recursively if a value is a nested map, but I don't know how to check if a value is a map. Using the map() function returns 1 for integers too, so doesn't help me.

Can anyone point me in the right direction, either to a function that already does what I want, or to tell me how to check if a value is a map or not ?

new topic     » topic index » view message » categorize

2. Re: "dumping" a map

cjnwl said...

So, in my tests I want to be able to dump a text representation of a complex map, which includes nested maps, and none of the map functions in section 8.32 of the manual allow me to do this. I therefore would like to write a function that reads all key/value pairs in order to display them on screen, and calls itself recursively if a value is a nested map, but I don't know how to check if a value is a map. Using the map() function returns 1 for integers too, so doesn't help me.

Can anyone point me in the right direction, either to a function that already does what I want, or to tell me how to check if a value is a map or not ?

You can't. Since, as you discovered, "maps" are simply references to in-memory constructs, you cannot "store a map inside a map" on disk. At least, not easily.

The best you can do is convert maps into JSON, which can give you "nested maps." However, even then, it is near impossible to reform a map because there is no way to know what kind of data you stored (not without explicit fields telling you what it is, which I guess is possible, actually). When I had questions about this (or a similar) need, Greg Haberek, current Lord of Euphoria, said this is difficult if not impossible in Euphoria's current incarnation.

You could also serialize the map and store it that way, but, again, you can't have a map of maps and just "save it to disk" without some indication of what your data was before you serialized it. There's gonna be a lot of processing required.

I guess the point is, maps have a specific use-case in Euphoria right now. You might want to consider another data store... Maybe an SQLite database would be better for your use-case.

P.S. I can't remember, but Pete's Phix might make map handling (called dictionaries in Phix) easier. What say you, Pete?

new topic     » goto parent     » topic index » view message » categorize

3. Re: "dumping" a map

The only way you could achieve that sort of thing is to tag any nested dictionaries properly,
so instead of storing 7, you store {"map",7} and it is your responsibility to ensure that nothing
ever naturally occurs which could accidentally be mistaken for that. Equally you could have say
{"building_plot", width, depth, price, dictionary_id} whereby [4] is always a dict (or 0/absent).

Tags are a pretty minimal overhead, and will more than easily repay their rent by helping with debugging.

As for saving to disk, then instead of saving {"map",7} you would have to store something like {"map","customers"},
and one thing Phix does have is the ability to give a name to a dictionary when it is created, and retrieve the
dictionary-id from a name, but otherwise not much else. Dictionaries are meant to be transient anyway, so if you are
looking for more permanent storage you would be far better off just using an SQLite database or similar.

new topic     » goto parent     » topic index » view message » categorize

4. Re: "dumping" a map

cjnwl said...

I'm new to OE and learning the ropes as it were. I've written a few simple programs, and I'm now exploring maps, which I want to use ultimately for a largish personal project that I have.

So, in my tests I want to be able to dump a text representation of a complex map, which includes nested maps, and none of the map functions in section 8.32 of the manual allow me to do this. I therefore would like to write a function that reads all key/value pairs in order to display them on screen, and calls itself recursively if a value is a nested map, but I don't know how to check if a value is a map. Using the map() function returns 1 for integers too, so doesn't help me.

Can anyone point me in the right direction, either to a function that already does what I want, or to tell me how to check if a value is a map or not ?

There's a lot going on here so I'll break it down topic by topic.

User defined types: Euphoria (currently) lacks any actual named-typing mechanism. The type syntax creates a user-defined type which is simply a function that 1) requires the base type specified in its parameter and 2) either passes or fails the type-check based on the output of the function (0 fails, anything else passes). When you declare a variable with a user-defined type (or UDT) it's still just an object and when type_check is enabled (which is on by default, use without type_check to disable it) the interpreter emits "type check" operations after every variable assignment, which triggers the behavior I just described and subsequently crashes with an error when a type check fails. However, once execution begins there's nothing stored on an object itself that identifies its original declared type. Everything in memory is either an atom, sequence, or integer by way of some internal bit-twiddling. The interpreter itself simply cannot know if you declared that 1 as an atom or an object or a map or whatever else. It's up to the UDT function to check the value of the object and return a pass/fail result.

Nested maps: When you create a map with new it just allocates a slot in Pseudo Memory (a.k.a. "eumem") which is just a way to create global "handles" for pass-by-reference. The map UDT just checks that the value is an atom, is a valid ID in eumem, and the does some internal bounds-checking. The nested_put routine is just a recursive call that "loops" through a sequence of keys, creating sub-maps as required. As long as the value stored in a key looks like a map ID, nested_put() assumes it is a map and uses that map for the next key in the sequence, even if the value is supposed to be just a number.

Saving maps: The save_map isn't ever concerned with nested maps. It just wants to write one-map-per-file, period. In fact, load_map very specifically resets the file position to zero each time it is called, so it's impossible to read more than one map from single file using that routine the way it is. It's probably best to think of these routines as a way to cache a map on disk, perhaps for storing a complex lookup table without having to regenerate it each time your application starts. If I was involved at the time, I think I would have argued against adding both nested maps and save/load since the two features are mutually exclusive.

So short answer to your question is: maps do not work that way. Sorry. But, if you're only looking to store and load a map-of-maps, then that is relatively straight-forward, and I made a demo for you: https://openeuphoria.org/pastey/357.wc

-Greg

new topic     » goto parent     » topic index » view message » categorize

5. Re: "dumping" a map

euphoric said...

You can't. Since, as you discovered, "maps" are simply references to in-memory constructs, you cannot "store a map inside a map" on disk. At least, not easily.

The best you can do is convert maps into JSON, which can give you "nested maps." However, even then, it is near impossible to reform a map because there is no way to know what kind of data you stored (not without explicit fields telling you what it is, which I guess is possible, actually).

You hit the nail on the head. I was halfway through drafting my in-depth explanation above when I saw you chimed in.

euphoric said...

When I had questions about this (or a similar) need, Greg Haberek, current Lord of Euphoria, said this is difficult if not impossible in Euphoria's current incarnation.

Hey hey hey. I prefer to use the term proprietor, thank you very much. I'm running the shop and keeping the lights on, not trying to pass decree from on high. Although "Lord" does have a certain ring to it...

euphoric said...

You could also serialize the map and store it that way, but, again, you can't have a map of maps and just "save it to disk" without some indication of what your data was before you serialized it. There's gonna be a lot of processing required.

I wouldn't say "a lot" of processing is required if we're only talking about a single level of nesting. But for multi-level or heterogeneous nesting yeah, it's nearly impossible as-is.

euphoric said...

I guess the point is, maps have a specific use-case in Euphoria right now. You might want to consider another data store... Maybe an SQLite database would be better for your use-case.

Don't discount EDS either. It's a perfectly cromulent data store for simple things. SQLite should hopefully ship with 4.2, whenever we can get it out the door.

-Greg

new topic     » goto parent     » topic index » view message » categorize

6. Re: "dumping" a map

In the olden daze, we'd ask Jiri how he did it. <bows head for a moment>

Pretty sure he'd mention "linked lists".

Sequences are marvelous things.

You could

sequence key_key = {} 
sequence key_data = {} 
sequence key_types = {} 
... 
sequence key_x = {} 

and then nest and search and sort the daylights out of them. In strtok, you could not only search/sort by the obvious top level, but nested sub-search/sorts too, in one call.

Sometimes the built-ins in OE are eye candy for one problem, an advertising gimmick, and you code your own for your problem. Maps never solved my problems, everything valuable is always more than "key-data" formats.

Kat

new topic     » goto parent     » topic index » view message » categorize

7. Re: "dumping" a map

One other thing, that confirms almost everything just said, is that the Phix docs are littered with links to this generic "unwise" statement.

I have also added this to the Phix docs recently:

It might not be entirely unfair to say that Phix user defined types are an explict embodiment of duck typing:

type duck(object x) 
    return gait(x)=WADDLE and noise(x)=QUACK and shape(x)=DUCK 
end type 
duck d = wolf -- crashes, eg `typecheck error: d is "wolf"` 
              -- aka you cannot put a wolf in a duck house 

However duck typing usually means the polar opposite: taking control away from the programmer and bravely/stupidly carrying on long after things go wrong.
In contrast, Phix types are designed to stop with a clear and readable (and catchable) runtime error as soon after and as close as possible to the error/mistake.

new topic     » goto parent     » topic index » view message » categorize

8. Re: "dumping" a map

Do platypus, do platypus!

Yeas, that was cute way to typecast a duck.

Until a duck is bred that's different, like it's fur never stops growing and is never shed , wait, that's a domestic sheep. Is "goose down" a "fur"? Is there a key in the map for "feathers resemble wool"? Anyhow, how can your program learn, if to add a new type it must write new source code, shut itself down, and hope it fires up again with no simple fatal crashes?

Kat

new topic     » goto parent     » topic index » view message » categorize

9. Re: "dumping" a map

Thanks all for your replies and advice.

At the moment I don't want to save anything to file, just display on screen in a readable format.

To give more context, I have an existing app which reads a text file and based on its contents generates text output. The existing app is written in gawk and comprises some 3500 lines of code. For some time I have wanted to port it to a multi-platform compilable version, and I'm looking at either OE/phix, or python. The current app relies heavily on gawk's associative arrays, with string keys, so I need an easy way to replicate this in OE. Maps seem to fit the bill, but for development and trouble-shooting purposes it would be great to be able to visualize the contents of a complex multi-level map structure.

And this is really just a learning exercise so that I can next leverage my new-found OE or Python knowledge to port an Excel/VBA which also relies heavily on data structures, but this is a much more ambitious project as I need to recreate Excel's tabbed GUI (though I don't need any formulas, just tabbed grids).

Anyway, your replies have given me food for thought, I will run some tests during the day to see if they lead anywhere.

new topic     » goto parent     » topic index » view message » categorize

10. Re: "dumping" a map

Is there any mechanism to catch runtime errors, like python's Try...Except ?

new topic     » goto parent     » topic index » view message » categorize

11. Re: "dumping" a map

Hi

The simple answer to that is no.

I think Eu has just relied on checking syntax, and reporting errors as they arise. You can implement in a check a variable is 'right' before operating on it, but that's about it.

Cheers

Chris

new topic     » goto parent     » topic index » view message » categorize

12. Re: "dumping" a map

Euphoria does not, but Phix however does have try/catch.

new topic     » goto parent     » topic index » view message » categorize

13. Re: "dumping" a map

cjnwl said...

To give more context, I have an existing app which reads a text file and based on its contents generates text output. The existing app is written in gawk and comprises some 3500 lines of code. For some time I have wanted to port it to a multi-platform compilable version, and I'm looking at either OE/phix, or python. The current app relies heavily on gawk's associative arrays, with string keys, so I need an easy way to replicate this in OE. Maps seem to fit the bill, but for development and trouble-shooting purposes it would be great to be able to visualize the contents of a complex multi-level map structure.

Awhile ago I wrote mvc/mapdbg.e for Euphoria MVC (a web framework I started writing for redoing this website). Mapdbg helps trace out what I was doing with maps and to ensure I was properly freeing objects from eumem. It's meant to be a drop-in replacement for std/map.e that uses uses the debugging features in Euphoria 4.1 to parse your code for map allocations and it logs these events to the console using mvc/logger.e. It also includes a print_map() routine that attempts to nicely display the key/value pairs, and an print_maps() routine that will print all of the maps currently allocated. I plan to pull mvc/logger.e into Euphoria 4.2 but if it's useful, I will pull in mvc/mapdbg.e as well. (These would be moved to the std/ include directory.) If you want to try it out, just copy mvc/mapdbg.e and mvc/logger.e into an mvc/ directory in your project directory and then change your include std/map.e line to include mvc/mapdbg.e. You also need to define MAPDBG for it to do its job and LOG_DEBUG for it to show debug logs.

-- enable debug logging. optional. shows mapdbg looking through your code. 
with define LOG_DEBUG 
include mvc/logger.e 
 
-- enable map debugging. required. tracks map usage and allows print_map() below. 
with define MAPDBG 
include mvc/mapdbg.e as map 
 
object users = map:new() 
map:nested_put( users, {"alice","id"}, "alice" ) 
map:nested_put( users, {"alice","name"}, "Alice" ) 
map:nested_put( users, {"alice","email"}, "alice@example.com" ) 
map:nested_put( users, {"bob","id"}, "bob" ) 
map:nested_put( users, {"bob","name"}, "Bob" ) 
map:nested_put( users, {"bob","email"}, "bob@example.com" ) 
map:nested_put( users, {"carol","id"}, "carol" ) 
map:nested_put( users, {"carol","name"}, "Carol" ) 
map:nested_put( users, {"carol","email"}, "carol@example.com" ) 
 
ifdef MAPDBG then 
    print_map( users, /* nested = */ 1 ) 
end ifdef 

Output:

users@/maptest3.ex:9 (map #3) = { 
     3: {0x24D62C12,"bob",  map #5 = { 
       1: {0x369D9230,"id","bob"}, 
      10: {0x36DEAD19,"email","bob@example.com"}, 
      12: {0x0EAAA6CB,"name","Bob"}, 
  } 
}, 
    10: {0x1C0FF609,"carol",  map #6 = { 
       1: {0x369D9230,"id","carol"}, 
      10: {0x36DEAD19,"email","carol@example.com"}, 
      12: {0x0EAAA6CB,"name","Carol"}, 
  } 
}, 
    15: {0x053F7D1E,"alice",  map #4 = { 
       1: {0x369D9230,"id","alice"}, 
      10: {0x36DEAD19,"email","alice@example.com"}, 
      12: {0x0EAAA6CB,"name","Alice"}, 
  } 
}, 
} 

cjnwl said...

And this is really just a learning exercise so that I can next leverage my new-found OE or Python knowledge to port an Excel/VBA which also relies heavily on data structures, but this is a much more ambitious project as I need to recreate Excel's tabbed GUI (though I don't need any formulas, just tabbed grids).

I've been revisiting IUP lately and it has a few options for easily creating a tabbed interface with grids of data. I'd be glad to help you out with that when you're ready.

cjnwl said...

Is there any mechanism to catch runtime errors, like python's Try...Except ?

Nope, sorry. Runtime errors crash. But you can use crash_routine to "catch" that crash before your program exits completely. When your crash routine is called, Euphoria has already written its ex.err so you can open and read that file if you need.

-Greg

new topic     » goto parent     » topic index » view message » categorize

14. Re: "dumping" a map

cjnwl said...

At the moment I don't want to save anything to file, just display on screen in a readable format.

If that's the case, then you can do it easily with Euphoria. grin

For any key that points to a map, just use a string like "something-map."

When reading it back, if any key you encounter contains "-map," you know that the value is a map, and you can deal with it appropriately.

You could also use a key formatted like {key,type}, e.g., {"my_map",EU_MAP}, or somesuch. Not sure about that last one, but I'm just thinking out loud for now.

new topic     » goto parent     » topic index » view message » categorize

15. Re: "dumping" a map

Thanks @ghabarek and @euphoric for your replies, using "map_xxx" as the key works for now, and I'll definitely try your mapdbg routines going forward. I'll also take you up on your offer of help with IUP when I reach that point, as I've never worked on a GUI before, my main programming experience goes back 30 years to dBase and COBOL, and since then I've mainly written short scripts. My gawk app is in essence only that, a script that reads an input file and generates output, so no GUI or user interaction. And the Excel/VBA app is similar in that I use the Excel front-end to enter data, and then it's a VBA script that generates output based on the contents of the different tabs.

new topic     » goto parent     » topic index » view message » categorize

16. Re: "dumping" a map

For what it's worth, here is my quick and dirty map_dump function. It relies on the key being prefixed with "m_" :

-- map investigation 
 
include std/io.e 
include std/map.e 
 
procedure map_dump(map m, integer offset) 
    integer s = map:size(m) 
    sequence k = map:keys(m) 
    sequence v = map:values(m) 
    for i = 1 to s do 
        if sequence(k[i]) and equal("m_", k[i][1..2]) then 
            writefln("[]>[]", {repeat(32, offset), k[i]}) 
            map_dump(v[i], offset+2) 
        else 
            writefln("[][] = []", {repeat(32, offset), k[i], v[i]}) 
        end if 
    end for 
    return 
end procedure 
 
 
map      tables = new()  -- map to hold tables data 
map      kids   = new()  -- map to hold kids 
map      smap   = new()  -- map to hold submap 
 
map:put(tables, "id", 1) 
map:put(tables, "name", "aaa") 
 
map:put(kids, 1, "alice") 
map:put(kids, 2, "tom") 
map:put(kids, 3, "wendy") 
map:put(tables, "m_kids", kids) 
 
map:put(smap, "test", "test") 
map:put(smap, "description", "long description") 
map:put(kids, "m_test", smap) 
 
map_dump(tables, 0) 

This gives the following output :

id = 1 
>m_kids 
  1 = alice 
  >m_test 
    description = long description 
    test = test 
  3 = wendy 
  2 = tom 
name = aaa 

new topic     » goto parent     » topic index » view message » categorize

17. Re: "dumping" a map

I can now use a nested_put|get to set or retrieve data, which is what I wanted :

 
map:nested_put(tables, {"m_kids", "m_test", "value"}, 125) 
writefln("\n" & map:nested_get(tables, {"m_kids", "m_test", "description"}) & "\n") 
 
map_dump(tables, 0) 

new topic     » goto parent     » topic index » view message » categorize

18. Re: "dumping" a map

cjnwl said...

I can now use a nested_put|get to set or retrieve data, which is what I wanted :

 
map:nested_put(tables, {"m_kids", "m_test", "value"}, 125) 
writefln("\n" & map:nested_get(tables, {"m_kids", "m_test", "description"}) & "\n") 
 
map_dump(tables, 0) 

Very nice, and thank you for this!

I wonder if we want to add this, or a variation of it at least, to the stdlib. Would be nice to be able to say that we support nested maps (even with some limitations).

new topic     » goto parent     » topic index » view message » categorize

19. Re: "dumping" a map

With some global variables and a couple of wrapper functions :

-- map investigation 
 
include std/io.e 
include std/map.e 
 
 
integer  indent  = 4       -- spaces per sub-level 
sequence m_      = "m_"    -- submap indicator 
sequence prefix  = ">> "   -- prefix before submap name 
sequence postfix = " >"    -- postfix after submap name 
 
map      tables = new()  -- map to hold tables data 
map      kids   = new()  -- map to hold kids 
map      smap   = new()  -- map to hold submap 
 
 
procedure map_dump(map m, integer offset) 
    integer s = map:size(m) 
    sequence k = map:keys(m) 
    sequence v = map:values(m) 
 
    for i = 1 to s do 
        if sequence(k[i]) and equal(m_, k[i][1..2]) then 
            writefln("[][][][]", {repeat(32, offset), prefix, k[i], postfix}) 
            map_dump(v[i], offset+indent) 
        else 
            writefln("[][] = []", {repeat(32, offset), k[i], v[i]}) 
        end if 
    end for 
    return 
end procedure 
 
procedure my_nested_put(map m, sequence s, object v) 
    integer l = length(s) 
    sequence temp = s 
    for i = 1 to l-1 do 
	temp[i] = m_ & temp[i] 
    end for 
    map:nested_put(m, temp, v) 
    return 
end procedure 
 
function my_nested_get(map m, sequence s) 
    integer l = length(s) 
    sequence temp = s 
    for i = 1 to l-1 do 
	temp[i] = m_ & temp[i] 
    end for 
    return map:nested_get(m, temp) 
end function 
    
 
map:put(tables, "id", 1) 
map:put(tables, "name", "aaa") 
 
map:put(kids, 1, "alice") 
map:put(kids, 2, "tom") 
map:put(kids, 3, "wendy") 
map:put(tables, "m_kids", kids) 
 
map:put(smap, "test", "test") 
map:put(smap, "description", "long description") 
map:put(kids, "m_test", smap) 
map:put(tables, "m_test_2", smap) 
 
map_dump(tables, 0) 
 
--map:nested_put(tables, {"m_kids", "m_test", "value"}, 125) 
--writefln("\n" & map:nested_get(tables, {"m_kids", "m_test", "description"}) & "\n") 
 
--map_dump(tables, 0) 
 
writefln("\n============\n") 
 
object users = map:new()  
map:nested_put( users, {"m_alice","id"}, "alice" )  
map:nested_put( users, {"m_alice","name"}, "Alice" )  
map:nested_put( users, {"m_alice","email"}, "alice@example.com" )  
map:nested_put( users, {"m_bob","id"}, "bob" )  
map:nested_put( users, {"m_bob","name"}, "Bob" )  
map:nested_put( users, {"m_bob","email"}, "bob@example.com" )  
map:nested_put( users, {"m_carol","id"}, "carol" )  
map:nested_put( users, {"m_carol","name"}, "Carol" )  
map:nested_put( users, {"m_carol","email"}, "carol@example.com" )  
 
my_nested_put(users, {"carol", "age"}, 25) 
 
map_dump(users, 0) 
 
writefln("\n" & my_nested_get(users, {"carol", "email"})) 
new topic     » goto parent     » topic index » view message » categorize

20. Re: "dumping" a map

FYI you can use <eucode></eucode> tags instead of {{{ }}} to get syntax-colored code blocks.

-Greg

new topic     » goto parent     » topic index » view message » categorize

21. Re: "dumping" a map

Thanks, I'd missed that possibility.

new topic     » goto parent     » topic index » view message » categorize

22. Re: "dumping" a map

So, I've been using my map_dump function for over a week now and it works fine.

I did have to add a to_string call in the two wrapper functions, so that I get e readable string when the submap key is an integer :

temp[i] = m_ & to_string(temp[i]) 

Note too that when I use want to insert an already-created map as a submap I have to explicitly include the m_ prefix before the final key in the call to my_nested_put:

my_nested_put(main_map, {key_1, m_ & key_2}, sub_map) 
new topic     » goto parent     » topic index » view message » categorize

Search



Quick Links

User menu

Not signed in.

Misc Menu