 |
Up |  |
TOC | Index | |||
| << 7 Included Tools | < 8.49 Syntax Coloring | Up: 8 API Reference | 8.51 Unit Testing Framework > | 9 Release Notes >> | |||
8.50 Euphoria Source Tokenizer
8.50.1 tokenize return sequence key
8.50.1.1 ET_TOKENS
include euphoria/tokenize.e namespace tokenize public enum ET_TOKENS
8.50.1.2 ET_ERROR
include euphoria/tokenize.e namespace tokenize public enum ET_ERROR
8.50.1.3 ET_ERR_LINE
include euphoria/tokenize.e namespace tokenize public enum ET_ERR_LINE
8.50.1.4 ET_ERR_COLUMN
include euphoria/tokenize.e namespace tokenize public enum ET_ERR_COLUMN
8.50.2 Tokens
8.50.2.1 T_EOF
include euphoria/tokenize.e namespace tokenize public enum T_EOF
8.50.2.2 T_NULL
include euphoria/tokenize.e namespace tokenize public enum T_NULL
8.50.2.3 T_SHBANG
include euphoria/tokenize.e namespace tokenize public enum T_SHBANG
8.50.2.4 T_NEWLINE
include euphoria/tokenize.e namespace tokenize public enum T_NEWLINE
8.50.2.5 T_COMMENT
include euphoria/tokenize.e namespace tokenize public enum T_COMMENT
8.50.2.6 T_NUMBER
include euphoria/tokenize.e namespace tokenize public enum T_NUMBER
8.50.2.7 T_CHAR
include euphoria/tokenize.e namespace tokenize public enum T_CHAR
quoted character
8.50.2.8 T_STRING
include euphoria/tokenize.e namespace tokenize public enum T_STRING
string
8.50.2.9 T_IDENTIFIER
include euphoria/tokenize.e namespace tokenize public enum T_IDENTIFIER
8.50.2.10 T_KEYWORD
include euphoria/tokenize.e namespace tokenize public enum T_KEYWORD
8.50.2.11 T_DOUBLE_OPS
include euphoria/tokenize.e namespace tokenize public enum T_DOUBLE_OPS
8.50.2.12 T_PLUSEQ
include euphoria/tokenize.e namespace tokenize public enum T_PLUSEQ
8.50.2.13 T_MINUSEQ
include euphoria/tokenize.e namespace tokenize public enum T_MINUSEQ
8.50.2.14 T_MULTIPLYEQ
include euphoria/tokenize.e namespace tokenize public enum T_MULTIPLYEQ
8.50.2.15 T_DIVIDEEQ
include euphoria/tokenize.e namespace tokenize public enum T_DIVIDEEQ
8.50.2.16 T_LTEQ
include euphoria/tokenize.e namespace tokenize public enum T_LTEQ
8.50.2.17 T_GTEQ
include euphoria/tokenize.e namespace tokenize public enum T_GTEQ
8.50.2.18 T_NOTEQ
include euphoria/tokenize.e namespace tokenize public enum T_NOTEQ
8.50.2.19 T_CONCATEQ
include euphoria/tokenize.e namespace tokenize public enum T_CONCATEQ
8.50.2.20 T_DELIMITER
include euphoria/tokenize.e namespace tokenize public enum T_DELIMITER
8.50.2.21 T_PLUS
include euphoria/tokenize.e namespace tokenize public enum T_PLUS
8.50.2.22 T_MINUS
include euphoria/tokenize.e namespace tokenize public enum T_MINUS
8.50.2.23 T_MULTIPLY
include euphoria/tokenize.e namespace tokenize public enum T_MULTIPLY
8.50.2.24 T_DIVIDE
include euphoria/tokenize.e namespace tokenize public enum T_DIVIDE
8.50.2.25 T_LT
include euphoria/tokenize.e namespace tokenize public enum T_LT
8.50.2.26 T_GT
include euphoria/tokenize.e namespace tokenize public enum T_GT
8.50.2.27 T_NOT
include euphoria/tokenize.e namespace tokenize public enum T_NOT
8.50.2.28 T_CONCAT
include euphoria/tokenize.e namespace tokenize public enum T_CONCAT
8.50.2.29 T_SINGLE_OPS
include euphoria/tokenize.e namespace tokenize public enum T_SINGLE_OPS
8.50.2.30 T_EQ
include euphoria/tokenize.e namespace tokenize public enum T_EQ
8.50.2.31 T_LPAREN
include euphoria/tokenize.e namespace tokenize public enum T_LPAREN
8.50.2.32 T_RPAREN
include euphoria/tokenize.e namespace tokenize public enum T_RPAREN
8.50.2.33 T_LBRACE
include euphoria/tokenize.e namespace tokenize public enum T_LBRACE
8.50.2.34 T_RBRACE
include euphoria/tokenize.e namespace tokenize public enum T_RBRACE
8.50.2.35 T_LBRACKET
include euphoria/tokenize.e namespace tokenize public enum T_LBRACKET
8.50.2.36 T_RBRACKET
include euphoria/tokenize.e namespace tokenize public enum T_RBRACKET
8.50.2.37 T_QPRINT
include euphoria/tokenize.e namespace tokenize public enum T_QPRINT
8.50.2.38 T_COMMA
include euphoria/tokenize.e namespace tokenize public enum T_COMMA
8.50.2.39 T_PERIOD
include euphoria/tokenize.e namespace tokenize public enum T_PERIOD
8.50.2.40 T_COLON
include euphoria/tokenize.e namespace tokenize public enum T_COLON
8.50.2.41 T_DOLLAR
include euphoria/tokenize.e namespace tokenize public enum T_DOLLAR
8.50.2.42 T_SLICE
include euphoria/tokenize.e namespace tokenize public enum T_SLICE
8.50.2.43 TF_HEX
include euphoria/tokenize.e namespace tokenize public enum TF_HEX
8.50.3 T_NUMBER formats and T_types
8.50.3.1 TF_INT
include euphoria/tokenize.e namespace tokenize public enum TF_INT
8.50.3.2 TF_ATOM
include euphoria/tokenize.e namespace tokenize public enum TF_ATOM
8.50.3.3 TF_STRING_SINGLE
include euphoria/tokenize.e namespace tokenize public enum TF_STRING_SINGLE
8.50.3.4 TF_STRING_TRIPLE
include euphoria/tokenize.e namespace tokenize public enum TF_STRING_TRIPLE
8.50.3.5 TF_STRING_BACKTICK
include euphoria/tokenize.e namespace tokenize public enum TF_STRING_BACKTICK
8.50.3.6 TF_STRING_HEX
include euphoria/tokenize.e namespace tokenize public enum TF_STRING_HEX
8.50.3.7 TF_COMMENT_SINGLE
include euphoria/tokenize.e namespace tokenize public enum TF_COMMENT_SINGLE
8.50.3.8 TF_COMMENT_MULTIPLE
include euphoria/tokenize.e namespace tokenize public enum TF_COMMENT_MULTIPLE
8.50.3.9 TTYPE
include euphoria/tokenize.e namespace tokenize public enum TTYPE
8.50.3.10 TDATA
include euphoria/tokenize.e namespace tokenize public enum TDATA
8.50.3.11 TLNUM
include euphoria/tokenize.e namespace tokenize public enum TLNUM
8.50.3.12 TLPOS
include euphoria/tokenize.e namespace tokenize public enum TLPOS
8.50.3.13 TFORM
include euphoria/tokenize.e namespace tokenize public enum TFORM
8.50.4 ET error codes
8.50.4.1 ERR_NONE
include euphoria/tokenize.e namespace tokenize public enum ERR_NONE
8.50.4.2 ERR_OPEN
include euphoria/tokenize.e namespace tokenize public enum ERR_OPEN
8.50.4.3 ERR_ESCAPE
include euphoria/tokenize.e namespace tokenize public enum ERR_ESCAPE
8.50.4.4 ERR_EOL_CHAR
include euphoria/tokenize.e namespace tokenize public enum ERR_EOL_CHAR
8.50.4.5 ERR_CLOSE_CHAR
include euphoria/tokenize.e namespace tokenize public enum ERR_CLOSE_CHAR
8.50.4.6 ERR_EOL_STRING
include euphoria/tokenize.e namespace tokenize public enum ERR_EOL_STRING
8.50.4.7 ERR_HEX
include euphoria/tokenize.e namespace tokenize public enum ERR_HEX
8.50.4.8 ERR_DECIMAL
include euphoria/tokenize.e namespace tokenize public enum ERR_DECIMAL
8.50.4.9 ERR_UNKNOWN
include euphoria/tokenize.e namespace tokenize public enum ERR_UNKNOWN
8.50.4.10 ERR_EOF
include euphoria/tokenize.e namespace tokenize public enum ERR_EOF
8.50.4.11 ERR_EOF_STRING
include euphoria/tokenize.e namespace tokenize public enum ERR_EOF_STRING
8.50.4.12 ERR_HEX_STRING
include euphoria/tokenize.e namespace tokenize public enum ERR_HEX_STRING
8.50.4.13 error_string
include euphoria/tokenize.e namespace tokenize public function error_string(integer err)
Get an error message string for a given error code.
8.50.5 get/set options
8.50.5.1 keep_newlines
include euphoria/tokenize.e namespace tokenize public procedure keep_newlines(integer val = 1)
Return new lines as tokens.
default is FALSE
8.50.5.2 keep_comments
include euphoria/tokenize.e namespace tokenize public procedure keep_comments(integer val = 1)
Return comments as tokens
default is FALSE
8.50.5.3 string_numbers
include euphoria/tokenize.e namespace tokenize public procedure string_numbers(integer val = 1)
Return TDATA for all T_NUMBER tokens in "string" format.
Defaults:
- T_NUMBER tokens return atoms
- T_CHAR tokens return single integer chars
- T_EOF tokens return undefined data
- Other tokens return strings
8.50.6 Routines
8.50.6.1 tokenize_string
include euphoria/tokenize.e namespace tokenize public function tokenize_string(sequence code)
8.50.6.2 tokenize_file
include euphoria/tokenize.e namespace tokenize public function tokenize_file(sequence fname)
8.50.7 Debugging
8.50.7.1 token_names
include euphoria/tokenize.e namespace tokenize public constant token_names
Sequence containing token names for debugging
8.50.7.2 token_forms
include euphoria/tokenize.e namespace tokenize public constant token_forms
8.50.7.3 show_tokens
include euphoria/tokenize.e namespace tokenize public procedure show_tokens(integer fh, sequence tokens)
Print token names and data for each token in `tokens` to the file handle `fh`
Parameters:
- fh - file handle to print information to
- tokens - token sequence to print
Comments:
This does not take direct output from tokenize_string or tokenize_file. Instead they take the first element of their return value, the token stream only.
See Also:
tokenize_string, tokenize_file
