Rework YANG lexer/parser 59/92159/2
authorRobert Varga <robert.varga@pantheon.tech>
Thu, 13 Aug 2020 18:15:06 +0000 (20:15 +0200)
committerRobert Varga <robert.varga@pantheon.tech>
Fri, 14 Aug 2020 16:24:26 +0000 (18:24 +0200)
commit52af09aeb286b3c53fc9bd0c84eaf49f05ebc618
treef2064738583d90ecf9d3828d7ea82e32539c845f
parenta33ccd1aec56b0ea227f7b9dc73ac282e179382d
Rework YANG lexer/parser

The definitions in the parser and lexer are quite arcane and are
actually wrong, not allowing for a number of edge cases, which are
completely valid.

Furthermore the definition of IDENTIFIER is wrong, as it allows /
and : to appear in it -- effectively ruining the tokenization in
parser.

Refactor the lexer to perform correct tokenization in every situation.
This makes it more complicated, but also much more obvious as to
what is going on -- especially with regard as to what decisions
end up being parser's responsibility.

Refactor the parser so it recognizes YANG tokenization constructs,
notably quoted and unquoted strings, and assemble them from lexer
tokens.

JIRA: YANGTOOLS-1089
Change-Id: I34472bf0a7e262d4f633ce271952bbcd7639ef2f
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 58de36f365950a14a17b89f4e3ab8e7902090375)
yang/yang-parser-rfc7950/src/main/antlr/org/opendaylight/yangtools/antlrv4/code/gen/YangStatementLexer.g4
yang/yang-parser-rfc7950/src/main/antlr/org/opendaylight/yangtools/antlrv4/code/gen/YangStatementParser.g4
yang/yang-parser-rfc7950/src/main/java/org/opendaylight/yangtools/yang/parser/rfc7950/repo/ArgumentContextUtils.java
yang/yang-parser-rfc7950/src/test/java/org/opendaylight/yangtools/yang/stmt/YT1089Test.java [new file with mode: 0644]
yang/yang-parser-rfc7950/src/test/resources/bugs/YT1089/foo.yang [new file with mode: 0644]