Euphoria Ticket #902: Tokenize does not deal with nested long comments correctly

The function tokenize_string in the library euphoria/tokenize.e does not deal with nested C-style comments correctly. The function terminates the comment at the first */ no matter how many occurrences of /* there are. Creole uses tokenize_file which uses tokenize_string, so the fault can be demonstrated here.

/* Beginning of a C-style comment. 

/* The comment is now nested to a second level. 

*/ The comment should continue, returning to the first level 
but tokenize has terminated the comment, as can be seen from the colours, 
so this looks like code. 

The V4.1.0 interpreter ignores the entire nested comment correctly.



Type: Bug Report Severity: Minor Category: Library Routine
Assigned To: unknown Status: New Reported Release: v4.1
Fixed in SVN #: View VCS: none Milestone:


Quick Links

User menu

Not signed in.

Misc Menu