OpenEuphoria

2. Re: EuGTK and UTF-8

• Posted by EUWX Dec 16, 2012
• 1668 views

You have to understand something more about the syllabic characters and its composition and usage on computers. For a long time we were restricted to 128 character set, which was later expanded to 256 character set.This essentially is based on using ONE BYTE per character. Keyboards do not have 256 keys and various schemes are devised to enter the 256 variations of glyphs for a given language font.

Based on these restrictions and need for using their own languages, most major languages created their own alternative glyphs and methods of joining BUT ALWAYS WITHIN THE 256 GLYPH availability. I mentioned Hindi Saral font, which is one of those simplifications.

Although the chinese have need for many more glyphs, The Hanzi font you mentioned is available as a one byte font
http://www.fontpalace.com/font-download/Hanzi-Kaishu/
with the advent of Unicode, the Indians (via their government) accepted 128 glyphs per Indian language and they reside at about 0980 hex onwards. Although Indian languages are truely syllabic, and there is a need for reserving about 4000 spaces per language, there is no demand from India or India for creating separate glyph pages. The Chinese did ask for and in fact they created an extension from 64K Unicode Glyphs to a 21 bit Unicode to accommodate these.

The Korean Unicode block for Hangul is 256 character block
Unicode Hangul Jamo (U+1100–U+11FF) The Hangul Jamo Unicode Block contains the possible Leading, Middle, and Trailing parts of a Hangul block in the following ranges:
• U+1100–U+1112: the basic 18 lnitial consonants plus 1 silent consonant (NG)
• U+1113–U+1159: ohter initial complex and ancient consonants
• U+115A–U+115E: Reserved
• U+115F: choseong (initial) filler
• U+1160: jungseong (middle) filler
• U+1161–U+1175: the basic 21 vowels and dipthongs
• U+1176–U+11A2: other and ancient vowel combinations
• U+11A3–U+11A7: Reserved
• U+11A8–U+11C2: the basic 27 trailing consonant combinations
• U+11C3–U+11F9: other trailing consonant combinations
• U+11FA–U+FFFF: Reserved
If you notice, you can use it in a software that can access a character in one byte provided the 256 glyphs are attached to one particular page of font.

When I talk about Unicode availability, I mean the ability to put any one of the characters, at least as available form Microsoft in their Arial MS Unicode font, within one word or phrase. That can only happen, if the software allows at least a 16 bit usage per character (and now 21 bit usage). The Microsoft alternative recognised by Microsoft with the introduction of Windows XP was to use the 16 bit character set internally. The European software developers even today are still thinking of code pages of 256 characters, as exemplified by your post regarding Hanzi and Hangul.

In the original thread, I tried to encourage the questioner to post the name of the RTL language he want to use, because I am pretty certain that for the major RTL group, I can find the 256 character ANSI font and and enable him to use any of the current non-Microsoft languages including Euphoria.

EuGTK will work, but only by using the one byte font.
AS IT STANDS WITH EUPHORIA, EUGTK WILL NOT WORK USING THE FULL TWO BYTE STORAGE.
However, Euphoria wrapper can probably be modified for two byte storage of Unicode characters if it is an absolute requirement, such as in a multilingual composition requiring sorting. RTL is a further complication/problem which can be easily tested by him using a byte font in his chosen RightTtoLeft language.

UTF-8 will create problems associated with selection and extraction form withing a phrase, because each "foreign" character can have anything form one to 6 bytes per character!

4. Re: EuGTK and UTF-8

• Posted by EUWX Dec 16, 2012
• 1518 views

1. Kindly look at and provide the Peeks of one of these strings.
2. Try extracting the third and 4th syllable of a string, like extracting "mc" from "jimbrown" in your EuGTK created Chinese text, using Euphoria and then extracting 6th and 7th similar to extracting "ro" from "jimbrown".

There are other aspects of your posts reminiscent of the year 200o-2003, and I will address them later as I get time.

Whenever i come accross Unicode and utf-8, theorizing seems to be the order of day. Outside of simple nice text under utw-8 in the net like this:
"जैसा ये लिखा है",
a major part of it is vapourware.

6. Re: EuGTK and UTF-8

• Posted by EUWX Dec 16, 2012
• 1589 views

The question is not whether GTK or Glib can do something. Can EUGTK do something? - that is the question. When I talked about 2000-2003; that was the time when people were changing over from Win 98 with its codepages to real 16 bit Unicode. There are countless examples of presumption of real Unicode, when actually it was just codepage or specific font related solution.
I can caterogrically state that using EUGTK, as is currently done in Euphoria, you cannot do extraction, rotation of characters cutting down of strings, etc. You need to write more code and more wrappers, or use anotherpiece of software to do that.

Kindly remember that when I talk about programming I am NOT talking about C or C plusplus; I am talking about application development languages. Most application development programmers are vaguely familiar with C plusplus, and cannot use it, but like me, are comfortable with BASIC and EUPHORIA. Even there, we would rather work with fully integrated GUI tools in the language. For world audience we would like fully developed language that can do string manipulation in Unicode, and a search compatible with syllabic languages. That is why, incidentally Hadoop is taking off in database work, because they fundamentally store a data field as a number of bytes in multiples of 4 bytes - which by the way, is the storage method of Euphoria, but not developed enough to reach those levels.

I am concerned that your attitude is always that of defending the existing rather than looking at the weaknesses and wanting to address these weaknesses, or of recognising the need for progress. The fact that I have rejected Freebasic, QB4, Python Schema, etc in favour of Euphoria is because of the strengths of Euphoria, but that does not mean I should ignore or attempt to hide the weaknesses under the hood of a big C

8. Re: EuGTK and UTF-8

• Posted by EUWX Dec 16, 2012
• 1582 views

Thanks for the two pasteys. It is cool, as the younger generation says.
The whole conversion started with the client looking for RTL language support. And in my first response I said this:
"Euphoria has a 4 bytes/per character and has availability of several Peek/Poke operations with memory. Therefore, you should be able to integrate your needs using 16 bit Unicode and Peek/Poke operations. "

In your pasteys you have used these features in the gunicode.e file and seem to have achieved this with UTF-8.

You should let him see these pasteys in the thread he started and ask him to test on RTL. He is also looking for BiDi (bidirectional) language support. He would therefore,need to test your Euphoria code with some bidirectional text.

As far as I am concerned, Microsoft's approach to Unicode, with a 2 byte character is the most rational approach. When memory was not easy to obtain we started with 7 bit ASCII, changing soon to 8 bit ANSI. With the introduction of Unicode, with the whole world's characters using 16 bit per character is logical. Now with 21 bit per character incorporating extended Chinese or whatever it is called, Euphoria using the full 4 bit character WHICH IT ALREADY HAS,is the most logical approach. Some work in C (unfortunately I don't know C and how to wrap.,etc) should enable proper extraction of sub-strings of LE16 and LE32. In fact a proper groups library functions to do conversions between different formats and extractions of substrings is desirable.
The client has brought a new problem. Whether or not, GTK has BiDi, it is doable in Euphoria using C and C language programmers using their own code and/or wrapping existing library. The world extends beyond the Suez Canal, and China is probably the richest country in the world.

10. Re: EuGTK and UTF-8

• Posted by mattlewis (admin) Dec 17, 2012
• 1497 views

The key factor would seem to be which languages you normally use. Personally, I only use English, so UTF-8 works very well for me, and for normal purposes, I don't even need to be aware that I'm using Unicode at all, since it generally looks the same as ASCII. If I used a different language with a different alphabet, then I'd probably prefer UTF-16 or UTF-32.

Soon the GDP of China may surpass that of the US (I don't take this as a given, however, they have a lot of challenges ahead of them), but the people will still be poorer. Their growth is basically them playing catch up with the rest of the developed world, and most of their population has a long way to go before that happens.

Matt