re2c scanners on C++ iterator source

Question

I am attempting to use re2c with input defined with an iterator pair instead of a null terminated string.

From Manual:

YYCURSOR
[...] 
On entry, YYCURSOR is assumed to point to the first character of the current token.
On exit, YYCURSOR will point to the first character of the following token.

This last point means that it is attempting to iterate past the end.

Is there a trick to make re2c work with iterators ? (Other than not using checked iterators.)

Answer 1

AFAICT, all you need to do is set YYLIMIT to point one character past the end, and have YYFILL() return if no further data exists (which in your case is probably always).

You might want to look at the re2c examples, specifically the strip_comments example, which reads from a file instead of a null terminated string.

Answer 2

Are you using re2c in the pull model where it calls YYFILL(n) or push model by passing -f?

In the pull model, declare YYFILL(n) to call return when your iterator is exhausted.

In the push model, your iterator loop will dictate when input is done and you won't call into the parser again.

Answer 3

Have a look at the examples on re2c website: http://re2c.org/examples/examples.html (they explain in great detail how to use YYFILL correctly or make lexer stop without YYFILL).

If you want to override the default input model (e.g. use re2c with STL iterators and streams), have a look at generic input API: http://re2c.org/manual/features/generic_api/generic_api.html