git.opendaylight Code Review - yangtools.git/log

Do not create temporary array for module sorting

ModuleDependencySort uses an internal iterable anyway, so expose
that method and pass our temporary set, skipping an unnecessary
copy operation.

Change-Id: I2253ed4edc3837ebb9182bcb1b0069d94e2f83e7
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit c0f132ca8118358f1e41feb196da4f068f130430)

BUG 8566 Introduce a fallback for ChoiceSchemaNode lookup

With the way parsing callbacks are handled between netconf and yangtools
theres no hook for case nodes, which we are missing in the path when doing
SchemaNode lookup. This introduces a callback that checks Choice children
for a possible match once the normal lookup fails.

Change-Id: Ieefc6771ec3c6892e00103cbab77f6340f197d62
Signed-off-by: Tomas Cere <tcere@cisco.com>

Bug 7844 - Unable to create LeafRefContext for leafref

In order to get correct namespace for a prefix used in a leafref
path we need to find the module which contains definition
of leafref path and then a prefix need to be resolved based on
imports of this module.

Change-Id: Icfb599783e90e44935d5e18eea23b0ac0d86cbf0
Signed-off-by: Peter Kajsa <pkajsa@cisco.com>

BUG-8123: fix URL naming mixup

As it turns out URL.getFile() does the wrong thing, leading to us
taking the whole path as the module name, resulting in wrong comparison.

Change the code to analyze URL.getPath(), stripping it down to the
last component.

Change-Id: Ica3e387caf2924712151f4da5939487872f8e98c
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit c2ca0aa4ddb933bbb4b1f8701c92c4f0d22c08a7)

Bug 7954: Throw an exception when parsing duplicate (sub)modules

In case we add the same YANG (sub)module to the YANG parsing
cycle twice, throw a SourceException which reports the problem
with colliding namespaces/names.

This patch is a manual cherry-pick of change 53283.

Change-Id: If0a0aff23ace2cdcbb3d87b639c9ef74f2302432
Signed-off-by: Igor Foltin <ifoltin@cisco.com>

BUG-8123: be better at guessing identifiers

Given an URI we need to attempt to interpret it to improve our
guess of the module name and revision.

Change-Id: I4d74b06fb9f949c538911d215b6d90a383560f1f
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 8bfdcd0056b08db284a87b55913691c0187c34bb)

BUG-7062: add revision awareness to Yin/YangTextSchemaSource

This adds support for parsing file names according
to https://tools.ietf.org/html/rfc6020#section-5.2.

Change-Id: I497aeba2cd801f618fa6a1075558325332623e31
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit f5d2344a51831445e13facb10db4b9076d8f42d2)

BUG-8039: Enforce binary/string type length

Binary type does not enforce input data length based on the supplied
constraints. Introduce two separate classes to deal with restricted
and unrestricted binary types.

String type fails to do the same, retrofit StringStringCodec with
the appropriate check.

Change-Id: I82516b39d66cbf846ed3eed39ee9e266b882f45d
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 65b95496775cd06d576f59b3ca3948636e5c8f60)

Bumping versions by x.y.(z+1) for next dev cycle

Change-Id: I393e7e43ca1b818ce23e42c944f7f8e8f8dbaa81
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

BUG-7759: return deserialized type

Union codec should not be returning the string representation,
but rather the proper deserialized type.

Change-Id: I4c452ecd59ee72b9b69ead39c3c36868bcd74444
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit b360de75e327e8a416b33b7113ea2bed350f8dce)

Bug 6392: Fix lenient parsing of unkeyed list entries

The old xml parser can be set to ignore nodes
which do not exist in the corresponding YANG model
using a boolean flag. However, this flag is
ignored when parsing unkeyed list entries.

This patch fixes the issue.

Change-Id: I798be8151d575f838e97a8fe44d04f47363491ba
Signed-off-by: Igor Foltin <ifoltin@cisco.com>

Bug 5410 - XSD regular expressions are interpreted as Java regexes (2/2)

Removal of xsd-regex parts from Yangtools.

This patch is a manual cherry-pick of change 53000.

Change-Id: I8bbe225f187c6ae215ceb9e144cac6bdbe1bddd6
Signed-off-by: Peter Kajsa <pkajsa@cisco.com>
Signed-off-by: Igor Foltin <ifoltin@cisco.com>

Bug 5410 - XSD regular expressions are interpreted as Java regexes (1/2)

As both '^' and '$' are special anchor characters in java regular
expressions which are implicitly present in XSD regular expressions,
we need to escape them in case they are not defined as part of
character ranges i.e. inside regular square brackets.

This patch is a manual cherry-pick of change 52999.

Change-Id: Iafbf350f88ebdf96c30e1ccedbd00b90a93d521a
Signed-off-by: Peter Kajsa <pkajsa@cisco.com>
Signed-off-by: Igor Foltin <ifoltin@cisco.com>

Bug 6856: Rpc definition should implicitly define input/output

Empty input and output statements are now automatically
added to every rpc statement that does not declare them.

This patch is a manual cherry-pick of change 48704.

Change-Id: I799137f18fe6ac382c8488684514b746dafbefb5
Signed-off-by: Igor Foltin <ifoltin@cisco.com>
Signed-off-by: Robert Varga <rovarga@cisco.com>
Signed-off-by: Igor Foltin <ifoltin@cisco.com>

Bug 7879 - refine target node was not found

In case of uses-refine used in augment defined in submodule the yang
statment parser was not able to resolve target namespace, because augment
was definied in submodule. In consequence, leaf node retained its original
namespace and therefore refine target node was not found.

This patch fixes the bug described above.

Change-Id: I24668a2efb6853cfd07b515ea5671cd2bfde580c
Signed-off-by: Peter Kajsa <pkajsa@cisco.com>

Bug 7182: YangToSourcesProcessor deletes output directory

Now that a c/52087 has added a lifecycle-mapping-metadata.xml into the
yang-maven-plugin, for this to really properly work in-IDE even on full
builds (we're far from making this work incrementally..) it's important
that the output directory is cleaned.

This is because I've observed in testing that otherwise something else
(I'm not clear what) seems to prevent overwriting already existing
*.java files.  I've verified that with this change, the typical use case
of e.g. git pull master, some changes in *.yang files, you Project >
Clean... (full build, only) - red is gone!  (Finally.)

A full delete of an output directory (on a full rebuild) is, unrelated,
better anyway, as it gets rid of previously generated classes with names
that theoretically possibly don't exist anymore, imagine e.g. after a
renaming in YANG.

This fix is particular to in-IDE code generation, following the
lifecycle mapping.  It typically was never a problem on CLI, cauz
everyone always just does "mvn clean install" anyway - but it won't hurt
there either.  But for in-IDE, we have to clean ourselves; thus this
change.

The two *Test changes fix test failures introduced by this change.
Unless a misunderstood the intent - those tests appear to be wrong - how
can they intend the outputBaseDir to be the SOURCE dir of those YANG
models?!?  (That's what they do as-is, and that's why the test fails,
when the source gets deleted.)

Change-Id: Ie47294511b6f9c57cefdb305344045e112e8d05a
Signed-off-by: Michael Vorburger <vorburger@redhat.com>
(cherry picked from commit 10cfb0891b3b6f7fed03560a2b9aa63a754167a3)

BUG-7182: add M2E lifecycle mapping

Add lifecycle mapping metadata. Execution will probably not work completely,
as the plugin architecture requires BUG-175 to be addressed, i.e. individual
generators should not be touching filesystem directly.

Change-Id: I7995de75c1817fb7bbdbc5e85790ad4e61a39486
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 5a0ce050962d03f87ce0bc2b4d2723141ae78b64)

BUG-7568: silence YangTextSchemaContextResolver

YangTextSchemaContextResolver should understand when we are expanding
a not-present revision and not emit info's then. We still log expansions
on debug and info about mismatches.

This silences the noise on controller startup, but does not completely
fix the issue.

Change-Id: I1b5e4eed9b288e2c6d48b494630eee0dd1fa38c4
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit a5046c41f6344bcba9f718c1639678df615cc4bb)

SourceIdentifier.create() method deprecation explained

- javadoc
- imports optimized

Change-Id: I3dfc20fd23a57051784a691875bce0260af39a5e
Signed-off-by: Martin Ciglan <mciglan@cisco.com>
(cherry picked from commit 3c372c27fe31696e35d03361b6476b0ccd035d77)

Do not confuse statement and argument names

Our extension handling is still flaky, in that it assumes
that implicit extensions take an argument. As we progress
to make these special-cases handled, the confusion is being
flushed out.

This particular error was found when we made argument name
conditional on whether we actually are seeing an argument
being used in the model.

Change-Id: I4b8a0e3aa133343fc65dca38c1dd9e083113fb00
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 33a2fed5d2d2beee742825778be50c8eb133565b)

Fix mandatory statement checking

This fixes a mismatch between the comment and what the method
actually does. For some reason we ended up iterating over the same
collection twice.

Furthermore use .forEach() with a self-referencing lambda instead
of Iterables.concat(), as that can be optimized by the implementation.

Change-Id: I6f74794fd58576b9ce34b6f27217b5e81bef4457
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 09755fe9c958f3b10e49ec533a589fb0ad60545d)

Bug 7440: Fix empty parent in deviate "replace"

Restricted type statement within deviate "replace" statement
now has its SchemaPath properly initialized.

Change-Id: If8a9c6add61ccdda8210f19e5626c925c3c5785e
Signed-off-by: Igor Foltin <ifoltin@cisco.com>
(cherry picked from commit 5797c4f23342d364261e025ac9b1e8b15d197912)

BUG-7267: catch RuntimeExceptions when processing sources

This adds wrapping of all RuntimeExceptions when building
EffectiveSchemaContext. This will allow users to identify
the offending source.

Since raw RuntimeExceptions should not be happening, but
rather should be specialized to SourceException and its
subclasses, also emit a warning guiding users to file
issues to fix codepaths which do not do so.

Also fixes up some instances where we use checkArgument()
and we ca actually use SourceException.throwIf().

Change-Id: Ie1c3d05b39b251996b746c38c5b7a51946b19ff5
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 0048b68fcb198ac3ec97d3efe1e47c5cfa20a619)

BUG-7038: cleanup parser listener

Rework the logic so we throw a SourceException as soon
as possible.

Change-Id: Ie704b87d408f2b5d4f310b6ff2c933607809288a
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 4660562e9a462660111087a76a259b37fd6bccf2)

BUG-7161: Do not tolerate source-level exceptions

We should not be suppressing malformed exceptions,
as that can result in us not providing correct
interpretation of models.

Change-Id: I535f066b329f02489f492e9a2b60f86bd5e21019
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 19d8860becaec36794a0d9450b61e4f4b75f4570)

BUG-7267: catch null path offenders

This provides a guard against invoking QName.create()
with a null base, reporting the original path.

Change-Id: Ic6bc661b94b5214336854a0cb20b6e595e7346e8
Signed-off-by: Robert Varga <rovarga@cisco.com>

Bumping versions by 0.0.1 for next dev cycle

Change-Id: I5bdb8493f2051204c3e5c3747f8cfabdb4ffa614
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

BUG-5968: Mandatory leaf enforcement does not work in some cases

Re-spin of reverted changes. Instead of performing the check at
seal() time, which forces validation of entire subtrees, perform
the check when the merge operation is actually broken down into
writes.

Change-Id: I70999f138ae36c09468ff04db0755e04bc7341c7
Signed-off-by: Robert Varga <rovarga@cisco.com>
Signed-off-by: Peter Kajsa <pkajsa@cisco.com>
(cherry picked from commit c71add8878f29787db7cc898c4982b0da8e92d2d)

Eliminate use of environment variables

Change I1edbed691ac1a01c8bfc1c8178d4836781aba0a1
introduced a back-channel to move effective settings
into the IT test. Use that instead of magic properties,
as it is safer.

Change-Id: If1b2d6e91ca4e9dd0c15f1c8dc10fa64ad7cdcdf
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 3f924bf51bce77949345a75e31b71b656a0e3435)

Fix IT test instantiation

We should be passing maven options for all tests, fix
the two omissions.

Change-Id: I43b5a001997547dc9935f390f046a5d8d839cd9f
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 5e6134e6824f2663516d51a1b5195883d35589a2)

Revert "Bug 5968: Mandatory leaf enforcement does not work in some cases"

This reverts commit 655c13c48277155b02706b75c88554c4a05ce673
and commit 3e4972b4e8edd7387666808014cd33daddef2cb5.

Change-Id: I9b12e58767ad70cb98a44b0a81560725b1a2afa5
Signed-off-by: Peter Kajsa <pkajsa@cisco.com>
(cherry picked from commit 8db9e47c66243774ad4291dcbed9012e70f33503)

BUG-5717: eliminate StmtContext.substatements()

There is a single caller, which really wants to get
the declared belongs-to statement.

Convert the caller and eliminate the method, making
StatementContextBase.substatements a purely-local
collection lending itself to refactoring.

Change-Id: I980c255eb426558a6a535683ccc8179082919c37
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 3c66a20cca00b3bffb9807000400ad88affea6d7)

BUG-6150: add an import statement special-case

This is just a hack to get the bug unblocked, a follow-up
patch is needed to really fix the underlying assumptions
about when a statement should be populated into the map.

Unit tests added.

Change-Id: I80273f1971981d62214e3cabfe687d5fcb61abd3
Signed-off-by: Robert Varga <rovarga@cisco.com>
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
(cherry picked from commit bbce2ddf4f0cfced8ea5ad85d709ce62ffdea41b)

Bug 5968: Mandatory leaf enforcement does not work in some cases

Fix of possible IllegalStateException.

Change-Id: Ide4d1bf060dc7ff8fc6286d5a1de03f8a6bedbfc
Signed-off-by: Peter Kajsa <pkajsa@cisco.com>
(cherry picked from commit 3e4972b4e8edd7387666808014cd33daddef2cb5)

Intern SchemaContext.NAME

This is a constant, we should be interning it.

Change-Id: I403057c9615f5d2ec66173188aa61db664447cc6
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 34c210974fcef3573c92ee3765332aaa218f4434)

Bug 6814: Fix (de)serialization of anyxml in JSON

JSON parser can now deserialize anyxml nodes.
JSON serializer can now serialize anyxml nodes.

Change-Id: I815afb5aac82618f27cd26d747d6637eb4e962f0
Signed-off-by: Igor Foltin <ifoltin@cisco.com>

BUG-7057: introduce UntrustedXML class

This adds the basic newDocumentBuilder() and newSAXParser()
methods. The class is expected to grow more methods in future.

Change-Id: I019891af5f9e8be15eda158bb6e40243dcec0614
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit cd4ec7e735d7db6fcda77c18bcf04b99a923fe1d)

Bug 5968: Mandatory leaf enforcement does not work in some cases

When MapEntry nodes are directly written into datatree, mandatory leaf
enforcement works correctly. However, when a List, Container or any other
parent node containing MapEntry nodes as its children is written into datatree,
datatree does not check presence of mandatory nodes of in this way written
map entry nodes. This patch provides fix which ensures mandatory node validation
in all cases above.

In addition, full validation on unsealed modification was performed after each MERGE
operation performed on this modification, which may lead to undesirable results,
because unsealed modification does not have to contain all elements, which can be
added during next operations on this modification. This patch fixes this behavior of
MERGE operation and performs full validation during seal() method.

Change-Id: If653eeda5378cf1ed1418123eb33713ee80976d9
Signed-off-by: Peter Kajsa <pkajsa@cisco.com>
Signed-off-by: Robert Varga <rovarga@cisco.com>
Signed-off-by: Peter Kajsa <pkajsa@cisco.com>
(cherry picked from commit 5ec45595d85f7c42310707ea058dac8a6be62acb)

Bumping versions by 0.0.1 for next dev cycle

Change-Id: I85fd0feb332b2c75ce6df52bfde65bc1d52042c4
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Hide BuildGlobalContext methods

Non-public class, there is no point in having its methods
declared as public.

Change-Id: Iba52f5c7fb9da4fa5b79ca37308f5af126aaf7d0
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 703f39878f5b52a337584078b9e32538d9ea06df)

BUG-4456: rework leaker integration

Rather than spreading the use of leaker across three
classes, centralize its use in ExtensionEffectiveStatementImpl
by exposing a protected method which can be overridden.

This is a bit cleaner solution and should end up being more
performant, as other statements do not end up touching
the thread-local variable.

Change-Id: I6f26dca90c0e5a907e98c65aa953f52ef87364e0
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit f8cf77d783bd1ada1d83a1e6a58237e15c451391)

BUG-6522: do not re-read models

The source text is ignored, so there is no point in reading it.

Deprecate the current method which makes it look like the text
is needed. And eliminate the read.

Change-Id: Ib5550e6e40ecad1c0883a21e39fada8136c5b820
Signed-off-by: Robert Varga <rovarga@cisco.com>

BUG-6522: grow namespaces lazily

NamespaceStorageSupport is the superclass of StatementContextBase,
hence allocating a default-sized HashMap results in ~75MiB wasted
space. Add lazy expansion to limit the overhead associated with
tracking these.

This patch includes 885bb2cb055b7786b6a44d490499949380750644.

Change-Id: I0825aee92487c94421ef9e505152bbc5fc7383cb
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit af16d2e800fbad3b91655214498f8737f915c7ac)

BUG-6522: Optimize toString() methods

There is no need to use String.format(), just use a simple
string concatenation.

Change-Id: I97383a662a4c7a634ad0dd17da5cc626a891ebc8
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 99710b7bb5d6251ec37648e179f2bb871e283808)

Cleanup SupportedExtensionsMapping

Align members to make them more readable.

Change-Id: I694c87d46e1d7a8fd5b5a28e87d0fe8a9f5a7807
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 47ab5589e16c7f0c965d50375dbc438b4f1d88f7)

BUG-6522: streamline QNameToStatementDefinitionMap API

This really is a Map, but let's not as far as declaring
it as such.

Add putAll() and putIfAbsent(), which make life way easier
for SourceSpecificContext -- which is the sole user. The end
result is better performance at least in the bulk case,
as we end up populating one of the maps via addAll().

Also take time to formulate the class in terms of lambas,
as that takes a lot of verbosity out.

Change-Id: Ie4f70cbc01011eb49b330ea4411d8308c401f76d
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 012fb5b3fac35a0fec1380abbf19d07dcd5e0f28)

BUG-6964: reuse StatementDefinitionNamespace

Instead of performing two lookups, cache the extension's
statement in the source's maps, so it behaves just like
any other statement.

Change-Id: Ie71bedeefb57f5fc1575515cccf56f03ad373117
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit a961d73abd6afcb10b778b04cf8d394a9f4342df)

BUG-6522: intern cached QNames

Heap analysis has shown a huge number duplicate QNameModules
being retained through QNames in this map, wasting about 4MB
of memory. Make sure we run QName.intern() before we store
them. The lookup path is still okay.

Change-Id: I389c16d4c097935dd6ff63da5623b96289a5fe0c
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit aca106473cf5c35c3f6d514fd33f27588e75c9f2)

BUG-6522: Remove an unneeded branch

Instead of maintaining a boolean in an else branch,
examine the resulting collection to see if there is
anything left. Simplifies and speeds up the code.

Change-Id: Ie87c5f845b866e6637a6af0e43cb2153d8e05363
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit e6e603a9fac4c40e09d71ae7db5d0d777fdb38ff)

BUG-7025: make the parser tree immutable

This adds a post-processing phase, which walks the entire tree
and makes sure children are initialized to an ImmutableList.

This is essentially the same operation as running the tree
through org.antlr.v4.runtime.Parser.TrimToSizeListener, except
it also initializes empty lists to immutable collection and
uses a more efficient implementation for singletons.

Change-Id: I2cf484654158ce01bb9f7c2ac51a9866f88ae320
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit acba6c1723f31a8689812cf0bd6f058f688465d3)

BUG-7025: Use ParseTreeWalker.DEFAULT

We do not need to instantiate new walkers, as there
is a utility instance available for reuse.

Change-Id: I3fb20fd333f1f133bc2427f9d21ddddd8721b972
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 8f31398fc04d759e5b7f36d1b2b72a8f3732769d)

BUG-6522: lower map sizing

Useing default-sized HashSet is wasteful, as we typically
have only a single requires/mutates action. This will lower
wastage which was observed to be around 600kB.

Change-Id: I642403858f2cab71c8896b2aa8a8cb0131306236
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 9455adbdd7de21d8da1a3bf6919db7f36dfba165)

BUG-6522: do not accidentally grow lists

If we end up with an empty collection being passed in,
do not do any processing so we do not inflate lists
unnecessarily. In profiling this showed to waste as
much as 23MiB of memory.

Change-Id: Ibb47953894162e6658f473ca763f17b8a98d2cb2
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit ba266efd6a8ebf1a853d7dcc6543991205bcf2db)

BUG-6972: optimize SourceSpecificContext

Statement supports are singletons, do not instantiate
them more than once.

Also make sure we grow importedNamespaces lazily, as there
typically are only a few imports.

Change-Id: Iadfc078fbd4d5e1d6b8b6bfe9e8721ae852ddc07
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit f80cb937cdce794db33e63e0caa4360f485cb8c8)

BUG-6522: Improve parser reactor logging

Add a few logging calls and implement toString() methods
so that the reactor progress can be observed.

Change-Id: Id475ba3aaba75040fe523b1d43c23e893ad25464
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 5b290448af7fb73d73ae22b2647c03b2457728fb)

BUG-6522: grow StatementContext collections lazily

A lot of statements do not have any substatements at all,
so grow collections as needed. For listeners also remove
sub-collections when then are no longer needed.

Change-Id: Ia429bd3761675fb00f742472042b31aa15862e7b
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 4223fda8e0219fe823eca7e2321ef4dd65dc8566)

BUG-6972: refactor copy checking

Original checking code was hard to read, this refactors
it to be more readable and understandable. It also
splits the two sets of copy operation invokers so they
do not share the same set of rules, as they need to be
different.

Change-Id: I9b4f831484c18228b3308709a8a8752bcc8e1b65
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit bd00b5b96a53c7fe6a04c26ba2831615a3220635)

BUG-6972: Remove GroupingUtils.needToCreateNewQName()

This is hard-wired to true and has a single caller,
remove the method and simplify caller.

Change-Id: I955c37b7a9e6b6a4418e1be23b89659a637a1ea0
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit a56a3df7cf9f00eb9af93173cf266941ffb3f847)

BUG-6972: eliminate AugmentUtils

The only callsite is AugmentStatementImpl, so move the code there
so it can be optimized, resolving a long-standing FIXME.

Change-Id: I1052e9c0956a13444252992fa0693d493a07a709
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit fb0d0d9b47ca7f26bc227f9cf3e24f23a6bfc402)

Bug 6669: Mandatory nodes cannot be added to node from another module via augment

Yang parser allowed to add mandatory nodes to target node from another module
via augmentations in some cases. This patch fixes this according to the rules defined
in RFC6020 (https://tools.ietf.org/html/rfc6020#section-3.1,
https://tools.ietf.org/html/rfc6020#section-7.15).

Change-Id: I699fc3c1014583b85ea47504f9a102d06e0b9011
Signed-off-by: Peter Kajsa <pkajsa@cisco.com>
(cherry picked from commit 092e3656c2276fb4559b336264339e8fb6f38ae9)

BUG-6972: Consolidate copy operations

This patch moves common code into a 'copy statement' utility
method, making sure we have do not leak it.

Change-Id: Iba205a6019977c3e6c30762dfb1407b13ec7cf37
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit a597ca7887ffa4b0fc9a5914cb9eeb47263c9251)

Bug 6979 - yang.model.util.EffectiveAugmentationSchema is mutable

fixed leak of mutable set

Change-Id: Iaa0dac93e3521a9a091feb7ea0c78e9c6b0e20e5
Signed-off-by: Filip Gregor <fgregor@cisco.com>
(cherry picked from commit e16dd0836ee7375d8c710f1ae9247f4d258f4760)

Bug 6329: Parser fails when target node of uses-augment is an unknown node

Yang parser fails when target node of uses-augment is an unknown node. It is not
quite clear whether such yang model is valid according to RFC6020 or not, but
yang parser failure causes lots of trouble, because such augment is used widely
in some yang models. So this patch prevents failure of yang parser and rather
introduces a warning and augmentation is not performed, when target node is
an unknown node.

Change-Id: I7f1c5d7b3ef0898ca098466702d7cb1ad7a0f30f
Signed-off-by: Peter Kajsa <pkajsa@cisco.com>

BUG-6522: create a specialized CopyHistory object

Copy histories have a limited number of operations, hence
mutating implementation can express the set of required operations
more efficiently. Also adds some tests to make sure the class
works correctly.

Change-Id: I0b2fd9fffa8a4bc754a9eff3808648f8f5726243
Signed-off-by: Robert Varga <rovarga@cisco.com>

BUG-6522: share instances of Config(Effective)Statement

Config statement has low cardinality, use specialized classes
for its representation.

Change-Id: I7b5dc7e7069e5e7cfa4808d44a95f147f6b7073d
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 2ad7b79d5a61b540afca3ce37ab81738bbb699b7)

BUG-6522: allocate copyHistory lazily

Most of the statements start off with being just originals,
so we can use a shared constant for their copy history
and replace it when it is modified.

Change-Id: Ia4fcd63d2bfb42d8ba5dc29daf74aa330e5b3689
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 8e09689b94fa942be64cdce30a15520383bb177e)

BUG-6522: Adjust collection sizes

StatementContexts are typically leaves or have a low number
of children, leading to wasted space during parsing.

Force collections to be allocated with minimum size and have
them grow as needed, so we limit the run-time overhead.

Change-Id: I30b87e7164f081251bdd44092858be1254d42cba
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 0715b98433da93758203f9cda9ef36d13832fc89)

Improve ContextBuilder error reporting

Instead of throwing an IllegalArgumentException throw
a proper SourceException, which will explain what statement
does not take an argument, pointing to the offending source.

Change-Id: Iee9a984c557f218195f0654bbb54d9e27a9cb9f6
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 8de9752c52b6e398d9651088bbd51795719133aa)

BUG-6965: introduce BuiltinDeclaredStatements

This patch adds squashing of YANG builtin types into
singletons when such types are not otherwise modified.

Change-Id: I3a146b4c7d6d01daf4892b2d41f813fc9bfcc393
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 5fa70757163c82fc46f1f09208a87a9944553401)

BUG-6964: define a namespace to hold model-defined statements

This patch adds a new namespace, StatementDefinitionNamespace,
which holds StatementDefinitions for all extensions defined in
the models being processed.

This namespace is then used to acquire a shared definition instead
of instantiating a definition everytime it is referenced.

Change-Id: I600ad8c6405e688f4fe9da12b58b8e50d3bb1a85
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 7b6b2679e9c45ecbd5b5421cad46aafa64cb0405)

Bug 6491: Some imports are not exposed in Module.getImports

in case of no revision in module and imported module, method
getImports returns empty Set.
added tests, fixed Set with imports in AbstractEffectiveModule

Change-Id: Ie23196d9cbcca98266580d11a85d66c06ac8561c
Signed-off-by: Filip Gregor <fgregor@cisco.com>
(cherry picked from commit a9aa900b8c5d1dbe16011a0c6c0f00cbb2ec105e)

Bug 6961: SchemaContext.getAllModuleIdentifiers() doesnt work for submodules

added submodules to return value from method getAllModuleIdentifiers

Change-Id: I9f328d0050457b7094cbb420f0e0205b8034cf6f
Signed-off-by: Filip Gregor <fgregor@cisco.com>
(cherry picked from commit 097a1cc73aa3ffbf1898b38cba4f6c66956a4c3b)

Reuse cardinality constants

0..1, 0..MAX and 1..1 are common cardinalities. Instead of leaking the MAX constant
create additional builder methods which reuse Cardinality objects.

Change-Id: Ie11e1f4414c484ecd43f0b472f9df8f0d0c18d0e
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit aae3ec264b796274acca53ddb404f0bdecdace70)

Fix inefficient ConstraintEffectiveStatements

Leaking a mutable ArrayList is incorrect, as the result
should be immutable. Furthermore not providing a hint
results in wasted space. Fix this by copying the temporary
ArrayList into an ImmutableList.

Change-Id: Ic6d1f1a72e6c045479a8b2e95b6b36f0b8d3aaaf
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 1f56bcb262d2b5f6b7fb6c879e30143db07945ec)

Fix GroupingEffectiveStatementImpl's unknown nodes

Heap analysis has pointed out a linked list, which is both
a correctness problem (it should be immutable) and efficiency
issue, as most of the time this list should be empty.

Use ImmutableList.Builder to build an immutable list of
unknown statements.

Change-Id: I94a42cb75d8fa433bcae304728268b0873d14683
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 647eddd0c2c6c2a0e2f68f52f65b7a9b6ee0a53b)

BUG-4456: add RecursiveExtensionResolver

This patch adds a hack which allows us to instantiate
recursive extensions. The solution is rather ugly, but
should be workable.

Change-Id: Ib5083804e5b49bd51c3b75c9905be39924e79ae5
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit bf55e6f6cc2a321af90aeacb974a1f7cf26be393)

BUG-6757: revert fix for BUG-4456

The fix has introduced a massive memory leak, which causes
all of temporary build objects to be retained in the final
SchemaContext.

Instead of the leak, add an explicit guard to detect
extensions (transitively) referencing themselves.

Change-Id: If90a4f9420866a6392ce97c71837915fa41ae0c8
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit e5226bf4ddd7a42aa13c392b2fc4d02d27ae2f74)

Fix failing unit test in QueuedNotificationManagerTest

Change-Id: I6a8a467cb2b3965753dccbe9b8bdd8e81a9bce80
Signed-off-by: Igor Foltin <ifoltin@cisco.com>
(cherry picked from commit 8e48344eca0be664e9e4be1480d24fdfdec9f7a5)

Use lambdas instead of anonymous classes

This makes the code more concise.

Change-Id: I0ce9711d0cafcf4660b78d937e5e0c9fa072da5a
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit f6eae2a11c570c1097eb9202debc7e36ce72ef6d)

Use lambdas instead of StmtContextUtils.build{Declared,Effecive}

Using lambdas at actual call sites reduces verbosity and eliminates
the need for unchecked casts and temporary variables. It also makes
call sites for the two target methods a lot more visible.

Change-Id: I4016c9ba447e22a61aeff5ca98d3adb1b80ef81b
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 7e9cdeeef2d18011f43bd848267e13855655f8ad)

BUG-5561: use canonical Bits order

This patch makes BitsTypeBuilder order the bit definitions
according to their position. It also adjusts BitsStringCodec
to normalize user input strings to the canonical ordering.

Change-Id: Ia4123a6275130619f9ef91d24b7340520707a67c
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit b3315a133684c91930b8e2f4004a327e50d48954)

Fix a bunch of warnings

- javadoc order
- unneeded casts
- static methods

Change-Id: I98344b7052eee1a58c709665e92d0cb1a0d72b46
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 7ee0100fadc14a764b1d971013d28f057457f20e)

Intro. new yangtools.testutils artifacts (incl. Mikito)

Seeding new project with proposed Mikito (new), and a
MockitoUnstubbedMethodExceptionAnswer (alternative to
yangtools.mockito-configuration; see the package-info.java in
mockito-configuration about why this is not being proposed in that
project).

Change-Id: I78237936924e0befc0ff17e7604e0fb33262d7cd
Signed-off-by: Michael Vorburger <vorburger@redhat.com>
(cherry picked from commit f07653736cdf6665a0ab8e39fe5427a095c1730b)

Add YangInstanceIdentifier fast paths

Each YangInstanceIdentifier contains() itself and its
.toRelative() is always empty.

Change-Id: I15ef921b661ccd32b9e6308789624798af4d98cf
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 5593e8c66e6a6e2badbcd75aafb0f42a6fb62d0c)

Use YangInstanceIdentifier.EMPTY

Eliminate an internal constant by referring to the well-known
public constant.

Change-Id: I2363714e1f694a352e0163a8c8054c27f22801f7
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 9fe172e110f6eab2f1b0093a2acff3dd27a9d9e4)

Bug 6771: Problem with typedefs nested in augment

When a typedef is nested in a container that is in an augment,
the YANG parser produces an error. This is due to fact, that typedef
statements nested in augments are ignored in statement definition phase,
because AugmentStatement is supported only in full definition phase. The
same issue occurs also for Choice and Case statements. This patch
provides fix of these bugs.

Change-Id: I345fd50e7d9810bbedcc873241338fbdb3186b74
Signed-off-by: Peter Kajsa <pkajsa@cisco.com>
(cherry picked from commit baa1b88aaf83bc2917a3b0859f950c1e8daf40ba)

Allow QueuedNotificationManager to batch notifications

This patch reworks the queueing logic so that notifications
get completely read from the queue and then propagated to the
listener invoker in one go.

Change-Id: I1a4caaa75f7a206abf4ff5b6829918bd25d79551
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit ddc17ff5daf9a58c5cf5f4ede3e49621c0db3d6c)

Add batching to QueuedNotificationManager

Introduce BatchedInvoker, which allows multiple events to be
delivered in one go.

Change-Id: I3d4d054543fa29842d67f6f21f28ff0a47ed135d
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 8aaf251e7eb6616384794e31bcd107ffe8fdba2e)

Cleanup QueuedNotificationManager

Audit code for coding style and unnecessary checks.

Change-Id: Ifce7d4fa4febb7f63ce9014e350e69143508e5c7
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit a0d690e3b716c7574c8fa39b3291220a42a7bdd0)

Bug 6551: Support for third-party Yang extensions implementation

Minor changes in yang statement parser in order to allow implementation
of custom inference pipeline.
- implementation of CustomStatementParserBuilder, which provides
construction of custom statement parser in user-friendly way.
- example and unit test of third-party extension plugin.

Change-Id: I0cdf0e28bd69af4cb41328be6e6a647df58f4fd9
Signed-off-by: Peter Kajsa <pkajsa@cisco.com>

Fix ConstraintDefinition inconsistency

The API contract around min/max elements is unclear, but the intent
is to provide a simple 'no effective constraint' result. This is
represented by a null return, which is in fact what the users expect.

Clarify the API contract and fix the two implementations to follow
it instead of using 0/MAX_INT for constraints.

Change-Id: I84b2e2714ddd017f319be4fc6dd0f7bec02bbf0d
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit edc4d5d22bd1120a47378deb9df56847c8769100)

Bumping versions by 0.0.1 for next dev cycle

Change-Id: I10c423dd908a899ece7f186bf821a070a26770d6
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Yangtools documentation

Added more documentation to the semantic versioning section

Change-Id: I71554017ee11732ef6f6176ca70711c96ee22480
Signed-off-by: Igor Foltin <ifoltin@cisco.com>
(cherry picked from commit dbfc518701f0ed1f74b2b5d55cc8c27064174925)

Yangtools documentation

Added documentation for new features introduced in boron:
- if-feature statement resolution
- semantic version processing

Change-Id: Ie31df99d428ab3eecdd40972a5977cd654c05a6b
Signed-off-by: Igor Foltin <ifoltin@cisco.com>
(cherry picked from commit 5cba929ce67b946502776b9b436dccd32e6c37e2)

BUG-6497: Do not lose augmentation statement order

This is a follow-up patch to keep augmentation order
in uses statement.

Change-Id: Iaef4aa949d225d914af3a04c111d0ec41aab6451
Signed-off-by: Robert Varga <rovarga@cisco.com>

BUG-6497: Do not lose augmentation statement order

Using an interim HashSet can cause unpredictable order
of augmentations, which means the binding spec will not
assign consistent mapping at runtime.

Fix this by using a LinkedHashSet, which retains insertion
order.

Change-Id: I503102d8b61453cc9c8d35b07158cd97138cda42
Signed-off-by: Robert Varga <rovarga@cisco.com>
(cherry picked from commit 3c4bdfac3a8b04ecd1d6eab2cdadbb365b54664e)

BUG-6316: Fix Bit and EnumPair's position/value types

yang.model.api.type defines the effective model of the world,
where these attributes cannot ever be null, which is in contrast
to yang.model.api.stmt, which is the declared model -- where they
can in fact be null.

Fix this discrepancy by forcing the methods to return simple types

Change-Id: I27b9d6d283cd14b2044890d9ccd19bf92647f27e
Signed-off-by: Robert Varga <rovarga@cisco.com>
Signed-off-by: Igor Foltin <ifoltin@cisco.com>
Signed-off-by: Peter Kajsa <pkajsa@cisco.com>

Bug 6420: Fixed SemVer's valueOf method

SemVer's valueOf method now works correctly when called
with a string that contains only major and minor version.

Change-Id: Ia9656fe121d1d73bbe2dc1de87cb7bd7b445961d
Signed-off-by: Igor Foltin <ifoltin@cisco.com>
(cherry picked from commit 9fb2e2367686d5fa1ae56dba0f5209ba2b822e8c)

Bug 6410: Fixed initialization of typedefs in rpc

In the constructor of RpcEffectiveStatementImpl we initialize substatements
of an rpc statment. However, in the for loop we are incorrectly trying to
find type substatements instead of typedef substatements.

Typedef substatements in RpcEffectiveStatementImpl are now initialized correctly.

Change-Id: Ic9a02727ffb4b6e6c798360e2266dc99d77c79e7
Signed-off-by: Igor Foltin <ifoltin@cisco.com>
(cherry picked from commit 8b7928e1337c94936434af5b4d9844804d2720de)