Euphoria
Ticket #166:
Unicode get routines
-
Reported by
SPringle
Jun 07, 2010
for Euphoria 4.1: routines matching the pattern get_integer\d\d should return -1 when they encounter the end of the file. The documentation doesn't mention this case either.
Details
1. Comment by jimcbrown
Jun 07, 2010
So the extra 3 bytes that get_integer32() grabs from a stream before seeing EOF should just be lost, thrown away?
2. Comment by DerekParnell
Jun 07, 2010
In my opinion, get_integerXX() are incorrectly named. As integers can be negative and positive these functions don't Get Integers at all. They get XX number of bits and convert that data into a positive integer. Thus using these on text files is kinda weird anyway.
I think we need a get_character() that retrieves the next Unicode character from the file, which depending on the encoding can fetch one of 8, 16, 24, 32, or 40 bits.
3. Comment by jimcbrown
Jun 07, 2010
I like the idea of having a get_utf8_character(), get_utf16_character(), etc, plus a generic get_utf_character() that can autodetect the type of UTF encoding used. But this isn't a big deal.
This still raises the question of what to do if we're a byte short, but, if it isn't a valid UTF character then those bytes are probably meaningless junk or otherwise too scrambled to make sense of anyways, in which case disposing of them and returning -1 probably isn't a big deal at all.
4. Comment by DerekParnell
Aug 05, 2010
All characters are implemented as positive integers, be that for ASCII or Unicode. If we need to indicate a failed get then returning a negative value seems to be reasonable.
For UTF gets, we could use -1 to indicate EOF has been reached and -2 to indicate that a mal-formed UTF character was found. And as for the bytes collected during the detection of a mal-formed character, I'm happy to lose them. They are meaningless in terms of a UTF character and probably means that the data source is corrupted or similar, and that will need to be investigated by other means by the user.
5. Comment by jimcbrown
Aug 05, 2010
This seems reasonable to me. Once you get an invalid character in UTF-8, I'm not sure if it's possible to figure out how to keep going without getting all out of sync with the byte stream, but in the worst case scenario we just returna bunch of -2s until we hit EOF and return -1...
6. Comment by DerekParnell
Aug 06, 2010
Changed this to a request because Unicode will be dealt with after the 4.0 release.
7. Comment by SDPringle
Mar 07, 2011
Is this the right milestone? I don't see unicode.e in the 4.1 trunk.
8. Comment by mattlewis
Mar 07, 2011
Nothing has been done on Unicode, but it's been planned for 4.1, though it might slip to 4.2, especially given the pace (and participation) of development so far.
9. Comment by mattlewis
Mar 07, 2011
As far as renaming...in places like eds.e and serialize.e, we have things like get4(), which follow a convention similar to the way peeks are named.
I agree that these routines are probably not correct, exactly, for unicode purposes. I don't see any obvious intent for these to be unicode centric, except for the interpretation in this ticket. These routines would be useful for any binary read that needed to read numbers larger than a byte.
10. Comment by DerekParnell
Mar 07, 2011
The get_integerXX() routines have nothing whatsoever to do with unicode. Returning -1 at detection of EOF could be done right now without wating for Unicode being supported in a later version.
11. Comment by SDPringle
Jul 06, 2011
See: hg:euphoria/rev/00f63e216512
changeset: 5050:00f63e216512 parent: 5045:8697fbaff8ab user: Shawn Pringle <shawn.pringle@gmail.com> date: Sat Jul 02 14:52:41 2011 -0300 files: docs/release/4.1.0.txt include/std/io.e source/configure source/parser.e description:
12. Comment by SDPringle
Jul 06, 2011
See: hg:euphoria/rev/90471def1f93
changeset: 5051:90471def1f93 parent: 5045:8697fbaff8ab user: Shawn Pringle <shawn.pringle@gmail.com> date: Sat Jul 02 14:52:41 2011 -0300 files: docs/release/4.1.0.txt include/std/io.e description:
GitHub