How to handle end of file in IntelliJ grammar kit

Question

I'm working on IntelliJ support for a DSL using the Grammar-Kit. I have a rule in the bnf file which requires an EOF (end of file) token at the end: rule ::= ( object | (( LCURL EOL* properties EOL* RCURL ) | properties ) ) EOL* EOF

I assumed I can simpy create a lexer rule in the flex file <<EOF>> { return EOF;} and that's it.. Unfortunately somewhere deeper in the IntelliJ code the logic which handles lexer's advance() method does not finish unless it gets a null from the lexer causing an infinite loop as the end of file returns a not-null token.

Is there any elegant (simple) way to handle an End Of File in the Grammar-Kit allowing the 'end of file' to be used in the Parser definition (the bnf)?

Answer 1

In .bnf file you can use <<eof>>. It's an external rule which links to GeneratedParserUtilBase.eof method.

Answer 2

In the end I've simply put an adapter between the generated lexer and its consumer which replaces functionality of the lexer's advance() method. If the EOF has been reached then the next time it will return null. Something like this:

override fun advance(): IElementType? {
    val t: IElementType? = lexer.advance()
    return when {
        eofReached -> null
        t == VUCLTypes.EOF -> {
            eofReached = true
            t
        }
        else -> t
    }
}

The eofReached is a flag which is set to false when initialised and at everty call of lexer's reset() method.

override fun reset(p0: CharSequence, p1: Int, p2: Int, p3: Int) {
    eofReached = false
    lexer.reset(p0, p1, p2, p3)
}

I hope I haven't missed anything..