Rework YANG lexer/parser 57/92157/2
authorRobert Varga <robert.varga@pantheon.tech>
Thu, 13 Aug 2020 18:15:06 +0000 (20:15 +0200)
committerRobert Varga <robert.varga@pantheon.tech>
Fri, 14 Aug 2020 16:24:45 +0000 (18:24 +0200)
commit5e0c9cbfcf3592d0e0deef92257e89d8702679ec
treebcf150f7d3b32b0b5dbfd9102599d0dd8dd30651
parentd502bccd8f1fd9757b3b4cf59784b7cdb21c8dc4
Rework YANG lexer/parser

The definitions in the parser and lexer are quite arcane and are
actually wrong, not allowing for a number of edge cases, which are
completely valid.

Furthermore the definition of IDENTIFIER is wrong, as it allows /
and : to appear in it -- effectively ruining the tokenization in
parser.

Refactor the lexer to perform correct tokenization in every situation.
This makes it more complicated, but also much more obvious as to
what is going on -- especially with regard as to what decisions
end up being parser's responsibility.

Refactor the parser so it recognizes YANG tokenization constructs,
notably quoted and unquoted strings, and assemble them from lexer
tokens.

JIRA: YANGTOOLS-1089
Change-Id: I34472bf0a7e262d4f633ce271952bbcd7639ef2f
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 58de36f365950a14a17b89f4e3ab8e7902090375)
yang/yang-parser-antlr/src/main/antlr4/org/opendaylight/yangtools/yang/parser/antlr/YangStatementLexer.g4
yang/yang-parser-antlr/src/main/antlr4/org/opendaylight/yangtools/yang/parser/antlr/YangStatementParser.g4
yang/yang-parser-rfc7950/src/main/java/org/opendaylight/yangtools/yang/parser/rfc7950/repo/ArgumentContextUtils.java
yang/yang-parser-rfc7950/src/test/java/org/opendaylight/yangtools/yang/stmt/YT1089Test.java [new file with mode: 0644]
yang/yang-parser-rfc7950/src/test/resources/bugs/YT1089/foo.yang [new file with mode: 0644]