Rework YANG lexer/parser 39/92139/8
authorRobert Varga <robert.varga@pantheon.tech>
Thu, 13 Aug 2020 18:15:06 +0000 (20:15 +0200)
committerRobert Varga <robert.varga@pantheon.tech>
Fri, 14 Aug 2020 12:06:47 +0000 (14:06 +0200)
commit58de36f365950a14a17b89f4e3ab8e7902090375
treeaac52f335f00df3a87df86891d0b865c6ca99fa6
parent0bf84fba6502cb4bc86f9e558a4a928e6397a793
Rework YANG lexer/parser

The definitions in the parser and lexer are quite arcane and are
actually wrong, not allowing for a number of edge cases, which are
completely valid.

Furthermore the definition of IDENTIFIER is wrong, as it allows /
and : to appear in it -- effectively ruining the tokenization in
parser.

Refactor the lexer to perform correct tokenization in every situation.
This makes it more complicated, but also much more obvious as to
what is going on -- especially with regard as to what decisions
end up being parser's responsibility.

Refactor the parser so it recognizes YANG tokenization constructs,
notably quoted and unquoted strings, and assemble them from lexer
tokens.

JIRA: YANGTOOLS-1089
Change-Id: I34472bf0a7e262d4f633ce271952bbcd7639ef2f
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
yang/yang-parser-antlr/src/main/antlr4/org/opendaylight/yangtools/yang/parser/antlr/YangStatementLexer.g4
yang/yang-parser-antlr/src/main/antlr4/org/opendaylight/yangtools/yang/parser/antlr/YangStatementParser.g4
yang/yang-parser-rfc7950/src/main/java/org/opendaylight/yangtools/yang/parser/rfc7950/repo/ArgumentContextUtils.java
yang/yang-parser-rfc7950/src/test/java/org/opendaylight/yangtools/yang/stmt/YT1089Test.java [new file with mode: 0644]
yang/yang-parser-rfc7950/src/test/resources/bugs/YT1089/foo.yang [new file with mode: 0644]