Euphoria Ticket #902: Tokenize does not deal with nested long comments correctly

The function tokenize_string in the library euphoria/tokenize.e does not deal with nested C-style comments correctly. The function terminates the comment at the first */ no matter how many occurrences of /* there are. Creole uses tokenize_file which uses tokenize_string, so the fault can be demonstrated here.

/* Beginning of a C-style comment. 

/* The comment is now nested to a second level. 

*/ The comment should continue, returning to the first level 
but tokenize has terminated the comment, as can be seen from the colours, 
so this looks like code. 
*/ 

The V4.1.0 interpreter ignores the entire nested comment correctly.

Arthur

Details

Type: Bug Report Severity: Minor Category: Library Routine
Assigned To: unknown Status: New Reported Release: v4.1
Fixed in SVN #: View VCS: none Milestone:

Search



Quick Links

User menu

Not signed in.

Misc Menu