OpenEuphoria

Euphoria Ticket #549: tokenize fails to deal with "\n\np[1][2..3]"

• Reported by jeremy Dec 15, 2010
• 1 comments

include std/unittest.e 
include euphoria/tokenize.e 
 
sequence tokens = tokenize_string("\n\np[1][2..3]") 
 
test_equal("tokenize_string ticket #549, 1", 9, length(tokens[1])) 
test_equal("tokenize_string ticket #549, 2", 2, tokens[1][6][TDATA]) 
test_equal("tokenize_string ticket #549, 3", T_SLICE, tokens[1][7][TTYPE]) 
test_equal("tokenize_string ticket #549, 4", 3, tokens[1][8][TDATA]) 
 
test_report() 

Details

1. Comment by jeremy Dec 15, 2010

Added a lookahead to see if the next character after 2. was a period as well, if so, returned right away.

During this test, I found another problem that caused a token to be dropped when a keyword( or keyword[ occured after a double newline. This was caused by calling next_token() and scan_char(). This is also fixed.

Tests were added for both conditions and now pass.

Include: