1. "dumping" a map
- Posted by cjnwl Feb 17, 2023
- 2049 views
Hi,
I'm new to OE and learning the ropes as it were. I've written a few simple programs, and I'm now exploring maps, which I want to use ultimately for a largish personal project that I have.
So, in my tests I want to be able to dump a text representation of a complex map, which includes nested maps, and none of the map functions in section 8.32 of the manual allow me to do this. I therefore would like to write a function that reads all key/value pairs in order to display them on screen, and calls itself recursively if a value is a nested map, but I don't know how to check if a value is a map. Using the map() function returns 1 for integers too, so doesn't help me.
Can anyone point me in the right direction, either to a function that already does what I want, or to tell me how to check if a value is a map or not ?
2. Re: "dumping" a map
- Posted by euphoric (admin) Feb 17, 2023
- 2002 views
So, in my tests I want to be able to dump a text representation of a complex map, which includes nested maps, and none of the map functions in section 8.32 of the manual allow me to do this. I therefore would like to write a function that reads all key/value pairs in order to display them on screen, and calls itself recursively if a value is a nested map, but I don't know how to check if a value is a map. Using the map() function returns 1 for integers too, so doesn't help me.
Can anyone point me in the right direction, either to a function that already does what I want, or to tell me how to check if a value is a map or not ?
You can't. Since, as you discovered, "maps" are simply references to in-memory constructs, you cannot "store a map inside a map" on disk. At least, not easily.
The best you can do is convert maps into JSON, which can give you "nested maps." However, even then, it is near impossible to reform a map because there is no way to know what kind of data you stored (not without explicit fields telling you what it is, which I guess is possible, actually). When I had questions about this (or a similar) need, Greg Haberek, current Lord of Euphoria, said this is difficult if not impossible in Euphoria's current incarnation.
You could also serialize the map and store it that way, but, again, you can't have a map of maps and just "save it to disk" without some indication of what your data was before you serialized it. There's gonna be a lot of processing required.
I guess the point is, maps have a specific use-case in Euphoria right now. You might want to consider another data store... Maybe an SQLite database would be better for your use-case.
P.S. I can't remember, but Pete's Phix might make map handling (called dictionaries in Phix) easier. What say you, Pete?
3. Re: "dumping" a map
- Posted by petelomax Feb 17, 2023
- 1981 views
The only way you could achieve that sort of thing is to tag any nested dictionaries properly,
so instead of storing 7, you store {"map",7} and it is your responsibility to ensure that nothing
ever naturally occurs which could accidentally be mistaken for that. Equally you could have say
{"building_plot", width, depth, price, dictionary_id} whereby [4] is always a dict (or 0/absent).
Tags are a pretty minimal overhead, and will more than easily repay their rent by helping with debugging.
As for saving to disk, then instead of saving {"map",7} you would have to store something like {"map","customers"},
and one thing Phix does have is the ability to give a name to a dictionary when it is created, and retrieve the
dictionary-id from a name, but otherwise not much else. Dictionaries are meant to be transient anyway, so if you are
looking for more permanent storage you would be far better off just using an SQLite database or similar.
4. Re: "dumping" a map
- Posted by ghaberek (admin) Feb 17, 2023
- 1971 views
I'm new to OE and learning the ropes as it were. I've written a few simple programs, and I'm now exploring maps, which I want to use ultimately for a largish personal project that I have.
So, in my tests I want to be able to dump a text representation of a complex map, which includes nested maps, and none of the map functions in section 8.32 of the manual allow me to do this. I therefore would like to write a function that reads all key/value pairs in order to display them on screen, and calls itself recursively if a value is a nested map, but I don't know how to check if a value is a map. Using the map() function returns 1 for integers too, so doesn't help me.
Can anyone point me in the right direction, either to a function that already does what I want, or to tell me how to check if a value is a map or not ?
There's a lot going on here so I'll break it down topic by topic.
User defined types: Euphoria (currently) lacks any actual named-typing mechanism. The type syntax creates a user-defined type which is simply a function that 1) requires the base type specified in its parameter and 2) either passes or fails the type-check based on the output of the function (0 fails, anything else passes). When you declare a variable with a user-defined type (or UDT) it's still just an object and when type_check is enabled (which is on by default, use without type_check to disable it) the interpreter emits "type check" operations after every variable assignment, which triggers the behavior I just described and subsequently crashes with an error when a type check fails. However, once execution begins there's nothing stored on an object itself that identifies its original declared type. Everything in memory is either an atom, sequence, or integer by way of some internal bit-twiddling. The interpreter itself simply cannot know if you declared that 1 as an atom or an object or a map or whatever else. It's up to the UDT function to check the value of the object and return a pass/fail result.
Nested maps: When you create a map with new it just allocates a slot in Pseudo Memory (a.k.a. "eumem") which is just a way to create global "handles" for pass-by-reference. The map UDT just checks that the value is an atom, is a valid ID in eumem, and the does some internal bounds-checking. The nested_put routine is just a recursive call that "loops" through a sequence of keys, creating sub-maps as required. As long as the value stored in a key looks like a map ID, nested_put() assumes it is a map and uses that map for the next key in the sequence, even if the value is supposed to be just a number.
Saving maps: The save_map isn't ever concerned with nested maps. It just wants to write one-map-per-file, period. In fact, load_map very specifically resets the file position to zero each time it is called, so it's impossible to read more than one map from single file using that routine the way it is. It's probably best to think of these routines as a way to cache a map on disk, perhaps for storing a complex lookup table without having to regenerate it each time your application starts. If I was involved at the time, I think I would have argued against adding both nested maps and save/load since the two features are mutually exclusive.
So short answer to your question is: maps do not work that way. Sorry. But, if you're only looking to store and load a map-of-maps, then that is relatively straight-forward, and I made a demo for you: https://openeuphoria.org/pastey/357.wc
-Greg
5. Re: "dumping" a map
- Posted by ghaberek (admin) Feb 17, 2023
- 1956 views
You can't. Since, as you discovered, "maps" are simply references to in-memory constructs, you cannot "store a map inside a map" on disk. At least, not easily.
The best you can do is convert maps into JSON, which can give you "nested maps." However, even then, it is near impossible to reform a map because there is no way to know what kind of data you stored (not without explicit fields telling you what it is, which I guess is possible, actually).
You hit the nail on the head. I was halfway through drafting my in-depth explanation above when I saw you chimed in.
When I had questions about this (or a similar) need, Greg Haberek, current Lord of Euphoria, said this is difficult if not impossible in Euphoria's current incarnation.
Hey hey hey. I prefer to use the term proprietor, thank you very much. I'm running the shop and keeping the lights on, not trying to pass decree from on high. Although "Lord" does have a certain ring to it...
You could also serialize the map and store it that way, but, again, you can't have a map of maps and just "save it to disk" without some indication of what your data was before you serialized it. There's gonna be a lot of processing required.
I wouldn't say "a lot" of processing is required if we're only talking about a single level of nesting. But for multi-level or heterogeneous nesting yeah, it's nearly impossible as-is.
I guess the point is, maps have a specific use-case in Euphoria right now. You might want to consider another data store... Maybe an SQLite database would be better for your use-case.
Don't discount EDS either. It's a perfectly cromulent data store for simple things. SQLite should hopefully ship with 4.2, whenever we can get it out the door.
-Greg
6. Re: "dumping" a map
- Posted by katsmeow Feb 17, 2023
- 1921 views
In the olden daze, we'd ask Jiri how he did it. <bows head for a moment>
Pretty sure he'd mention "linked lists".
Sequences are marvelous things.
You could
sequence key_key = {} sequence key_data = {} sequence key_types = {} ... sequence key_x = {}
and then nest and search and sort the daylights out of them. In strtok, you could not only search/sort by the obvious top level, but nested sub-search/sorts too, in one call.
Sometimes the built-ins in OE are eye candy for one problem, an advertising gimmick, and you code your own for your problem. Maps never solved my problems, everything valuable is always more than "key-data" formats.
Kat
7. Re: "dumping" a map
- Posted by petelomax Feb 17, 2023
- 1924 views
One other thing, that confirms almost everything just said, is that the Phix docs are littered with links to this generic "unwise" statement.
I have also added this to the Phix docs recently:
It might not be entirely unfair to say that Phix user defined types are an explict embodiment of duck typing:
type duck(object x) return gait(x)=WADDLE and noise(x)=QUACK and shape(x)=DUCK end type duck d = wolf -- crashes, eg `typecheck error: d is "wolf"` -- aka you cannot put a wolf in a duck house
However duck typing usually means the polar opposite: taking control away from the programmer and bravely/stupidly carrying on long after things go wrong.
In contrast, Phix types are designed to stop with a clear and readable (and catchable) runtime error as soon after and as close as possible to the error/mistake.
8. Re: "dumping" a map
- Posted by katsmeow Feb 17, 2023
- 1925 views
Do platypus, do platypus!
Yeas, that was cute way to typecast a duck.
Until a duck is bred that's different, like it's fur never stops growing and is never shed , wait, that's a domestic sheep. Is "goose down" a "fur"? Is there a key in the map for "feathers resemble wool"? Anyhow, how can your program learn, if to add a new type it must write new source code, shut itself down, and hope it fires up again with no simple fatal crashes?
Kat
9. Re: "dumping" a map
- Posted by cjnwl Feb 20, 2023
- 1613 views
Thanks all for your replies and advice.
At the moment I don't want to save anything to file, just display on screen in a readable format.
To give more context, I have an existing app which reads a text file and based on its contents generates text output. The existing app is written in gawk and comprises some 3500 lines of code. For some time I have wanted to port it to a multi-platform compilable version, and I'm looking at either OE/phix, or python. The current app relies heavily on gawk's associative arrays, with string keys, so I need an easy way to replicate this in OE. Maps seem to fit the bill, but for development and trouble-shooting purposes it would be great to be able to visualize the contents of a complex multi-level map structure.
And this is really just a learning exercise so that I can next leverage my new-found OE or Python knowledge to port an Excel/VBA which also relies heavily on data structures, but this is a much more ambitious project as I need to recreate Excel's tabbed GUI (though I don't need any formulas, just tabbed grids).
Anyway, your replies have given me food for thought, I will run some tests during the day to see if they lead anywhere.
10. Re: "dumping" a map
- Posted by cjnwl Feb 20, 2023
- 1614 views
Is there any mechanism to catch runtime errors, like python's Try...Except ?
11. Re: "dumping" a map
- Posted by ChrisB (moderator) Feb 20, 2023
- 1553 views
Hi
The simple answer to that is no.
I think Eu has just relied on checking syntax, and reporting errors as they arise. You can implement in a check a variable is 'right' before operating on it, but that's about it.
Cheers
Chris
12. Re: "dumping" a map
- Posted by petelomax Feb 20, 2023
- 1569 views
Euphoria does not, but Phix however does have try/catch.
13. Re: "dumping" a map
- Posted by ghaberek (admin) Feb 20, 2023
- 1555 views
To give more context, I have an existing app which reads a text file and based on its contents generates text output. The existing app is written in gawk and comprises some 3500 lines of code. For some time I have wanted to port it to a multi-platform compilable version, and I'm looking at either OE/phix, or python. The current app relies heavily on gawk's associative arrays, with string keys, so I need an easy way to replicate this in OE. Maps seem to fit the bill, but for development and trouble-shooting purposes it would be great to be able to visualize the contents of a complex multi-level map structure.
Awhile ago I wrote mvc/mapdbg.e for Euphoria MVC (a web framework I started writing for redoing this website). Mapdbg helps trace out what I was doing with maps and to ensure I was properly freeing objects from eumem. It's meant to be a drop-in replacement for std/map.e that uses uses the debugging features in Euphoria 4.1 to parse your code for map allocations and it logs these events to the console using mvc/logger.e. It also includes a print_map() routine that attempts to nicely display the key/value pairs, and an print_maps() routine that will print all of the maps currently allocated. I plan to pull mvc/logger.e into Euphoria 4.2 but if it's useful, I will pull in mvc/mapdbg.e as well. (These would be moved to the std/ include directory.) If you want to try it out, just copy mvc/mapdbg.e and mvc/logger.e into an mvc/ directory in your project directory and then change your include std/map.e line to include mvc/mapdbg.e. You also need to define MAPDBG for it to do its job and LOG_DEBUG for it to show debug logs.
-- enable debug logging. optional. shows mapdbg looking through your code. with define LOG_DEBUG include mvc/logger.e -- enable map debugging. required. tracks map usage and allows print_map() below. with define MAPDBG include mvc/mapdbg.e as map object users = map:new() map:nested_put( users, {"alice","id"}, "alice" ) map:nested_put( users, {"alice","name"}, "Alice" ) map:nested_put( users, {"alice","email"}, "alice@example.com" ) map:nested_put( users, {"bob","id"}, "bob" ) map:nested_put( users, {"bob","name"}, "Bob" ) map:nested_put( users, {"bob","email"}, "bob@example.com" ) map:nested_put( users, {"carol","id"}, "carol" ) map:nested_put( users, {"carol","name"}, "Carol" ) map:nested_put( users, {"carol","email"}, "carol@example.com" ) ifdef MAPDBG then print_map( users, /* nested = */ 1 ) end ifdef
Output:
users@/maptest3.ex:9 (map #3) = { 3: {0x24D62C12,"bob", map #5 = { 1: {0x369D9230,"id","bob"}, 10: {0x36DEAD19,"email","bob@example.com"}, 12: {0x0EAAA6CB,"name","Bob"}, } }, 10: {0x1C0FF609,"carol", map #6 = { 1: {0x369D9230,"id","carol"}, 10: {0x36DEAD19,"email","carol@example.com"}, 12: {0x0EAAA6CB,"name","Carol"}, } }, 15: {0x053F7D1E,"alice", map #4 = { 1: {0x369D9230,"id","alice"}, 10: {0x36DEAD19,"email","alice@example.com"}, 12: {0x0EAAA6CB,"name","Alice"}, } }, }
And this is really just a learning exercise so that I can next leverage my new-found OE or Python knowledge to port an Excel/VBA which also relies heavily on data structures, but this is a much more ambitious project as I need to recreate Excel's tabbed GUI (though I don't need any formulas, just tabbed grids).
I've been revisiting IUP lately and it has a few options for easily creating a tabbed interface with grids of data. I'd be glad to help you out with that when you're ready.
Is there any mechanism to catch runtime errors, like python's Try...Except ?
Nope, sorry. Runtime errors crash. But you can use crash_routine to "catch" that crash before your program exits completely. When your crash routine is called, Euphoria has already written its ex.err so you can open and read that file if you need.
-Greg
14. Re: "dumping" a map
- Posted by euphoric (admin) Feb 20, 2023
- 1552 views
At the moment I don't want to save anything to file, just display on screen in a readable format.
If that's the case, then you can do it easily with Euphoria.
For any key that points to a map, just use a string like "something-map."
When reading it back, if any key you encounter contains "-map," you know that the value is a map, and you can deal with it appropriately.
You could also use a key formatted like {key,type}, e.g., {"my_map",EU_MAP}, or somesuch. Not sure about that last one, but I'm just thinking out loud for now.
15. Re: "dumping" a map
- Posted by cjnwl Feb 22, 2023
- 1464 views
Thanks @ghabarek and @euphoric for your replies, using "map_xxx" as the key works for now, and I'll definitely try your mapdbg routines going forward. I'll also take you up on your offer of help with IUP when I reach that point, as I've never worked on a GUI before, my main programming experience goes back 30 years to dBase and COBOL, and since then I've mainly written short scripts. My gawk app is in essence only that, a script that reads an input file and generates output, so no GUI or user interaction. And the Excel/VBA app is similar in that I use the Excel front-end to enter data, and then it's a VBA script that generates output based on the contents of the different tabs.
16. Re: "dumping" a map
- Posted by cjnwl Feb 22, 2023
- 1415 views
For what it's worth, here is my quick and dirty map_dump function. It relies on the key being prefixed with "m_" :
-- map investigation include std/io.e include std/map.e procedure map_dump(map m, integer offset) integer s = map:size(m) sequence k = map:keys(m) sequence v = map:values(m) for i = 1 to s do if sequence(k[i]) and equal("m_", k[i][1..2]) then writefln("[]>[]", {repeat(32, offset), k[i]}) map_dump(v[i], offset+2) else writefln("[][] = []", {repeat(32, offset), k[i], v[i]}) end if end for return end procedure map tables = new() -- map to hold tables data map kids = new() -- map to hold kids map smap = new() -- map to hold submap map:put(tables, "id", 1) map:put(tables, "name", "aaa") map:put(kids, 1, "alice") map:put(kids, 2, "tom") map:put(kids, 3, "wendy") map:put(tables, "m_kids", kids) map:put(smap, "test", "test") map:put(smap, "description", "long description") map:put(kids, "m_test", smap) map_dump(tables, 0)
This gives the following output :
id = 1 >m_kids 1 = alice >m_test description = long description test = test 3 = wendy 2 = tom name = aaa
17. Re: "dumping" a map
- Posted by cjnwl Feb 22, 2023
- 1408 views
I can now use a nested_put|get to set or retrieve data, which is what I wanted :
map:nested_put(tables, {"m_kids", "m_test", "value"}, 125) writefln("\n" & map:nested_get(tables, {"m_kids", "m_test", "description"}) & "\n") map_dump(tables, 0)
18. Re: "dumping" a map
- Posted by jimcbrown (admin) Feb 22, 2023
- 1396 views
I can now use a nested_put|get to set or retrieve data, which is what I wanted :
map:nested_put(tables, {"m_kids", "m_test", "value"}, 125) writefln("\n" & map:nested_get(tables, {"m_kids", "m_test", "description"}) & "\n") map_dump(tables, 0)
Very nice, and thank you for this!
I wonder if we want to add this, or a variation of it at least, to the stdlib. Would be nice to be able to say that we support nested maps (even with some limitations).
19. Re: "dumping" a map
- Posted by cjnwl Feb 22, 2023
- 1382 views
- Last edited Feb 24, 2023
With some global variables and a couple of wrapper functions :
-- map investigation include std/io.e include std/map.e integer indent = 4 -- spaces per sub-level sequence m_ = "m_" -- submap indicator sequence prefix = ">> " -- prefix before submap name sequence postfix = " >" -- postfix after submap name map tables = new() -- map to hold tables data map kids = new() -- map to hold kids map smap = new() -- map to hold submap procedure map_dump(map m, integer offset) integer s = map:size(m) sequence k = map:keys(m) sequence v = map:values(m) for i = 1 to s do if sequence(k[i]) and equal(m_, k[i][1..2]) then writefln("[][][][]", {repeat(32, offset), prefix, k[i], postfix}) map_dump(v[i], offset+indent) else writefln("[][] = []", {repeat(32, offset), k[i], v[i]}) end if end for return end procedure procedure my_nested_put(map m, sequence s, object v) integer l = length(s) sequence temp = s for i = 1 to l-1 do temp[i] = m_ & temp[i] end for map:nested_put(m, temp, v) return end procedure function my_nested_get(map m, sequence s) integer l = length(s) sequence temp = s for i = 1 to l-1 do temp[i] = m_ & temp[i] end for return map:nested_get(m, temp) end function map:put(tables, "id", 1) map:put(tables, "name", "aaa") map:put(kids, 1, "alice") map:put(kids, 2, "tom") map:put(kids, 3, "wendy") map:put(tables, "m_kids", kids) map:put(smap, "test", "test") map:put(smap, "description", "long description") map:put(kids, "m_test", smap) map:put(tables, "m_test_2", smap) map_dump(tables, 0) --map:nested_put(tables, {"m_kids", "m_test", "value"}, 125) --writefln("\n" & map:nested_get(tables, {"m_kids", "m_test", "description"}) & "\n") --map_dump(tables, 0) writefln("\n============\n") object users = map:new() map:nested_put( users, {"m_alice","id"}, "alice" ) map:nested_put( users, {"m_alice","name"}, "Alice" ) map:nested_put( users, {"m_alice","email"}, "alice@example.com" ) map:nested_put( users, {"m_bob","id"}, "bob" ) map:nested_put( users, {"m_bob","name"}, "Bob" ) map:nested_put( users, {"m_bob","email"}, "bob@example.com" ) map:nested_put( users, {"m_carol","id"}, "carol" ) map:nested_put( users, {"m_carol","name"}, "Carol" ) map:nested_put( users, {"m_carol","email"}, "carol@example.com" ) my_nested_put(users, {"carol", "age"}, 25) map_dump(users, 0) writefln("\n" & my_nested_get(users, {"carol", "email"}))
20. Re: "dumping" a map
- Posted by ghaberek (admin) Feb 23, 2023
- 1256 views
FYI you can use <eucode></eucode> tags instead of {{{ }}} to get syntax-colored code blocks.
-Greg
21. Re: "dumping" a map
- Posted by cjnwl Feb 24, 2023
- 1107 views
Thanks, I'd missed that possibility.
22. Re: "dumping" a map
- Posted by cjnwl Mar 02, 2023
- 833 views
So, I've been using my map_dump function for over a week now and it works fine.
I did have to add a to_string call in the two wrapper functions, so that I get e readable string when the submap key is an integer :
temp[i] = m_ & to_string(temp[i])
Note too that when I use want to insert an already-created map as a submap I have to explicitly include the m_ prefix before the final key in the call to my_nested_put:
my_nested_put(main_map, {key_1, m_ & key_2}, sub_map)