controller.git
5 years agoBump versions by x.y.(z+1) 28/71928/1 stable/nitrogen
jenkins-releng [Thu, 10 May 2018 23:34:38 +0000 (23:34 +0000)]
Bump versions by x.y.(z+1)

Change-Id: I17b58b5e987a6e4846e270b67d7f5bb8994aed85
Signed-off-by: jenkins-releng <jenkins-releng@opendaylight.org>
5 years agoBump to odlparent 2.0.7 04/71504/1
Stephen Kitt [Fri, 27 Apr 2018 10:16:12 +0000 (12:16 +0200)]
Bump to odlparent 2.0.7

Change-Id: Ibf0c01bf22908557a21052255a9077008d06b85a
Signed-off-by: Stephen Kitt <skitt@redhat.com>
6 years agoBump to odlparent 2.0.6 67/71367/1
Stephen Kitt [Thu, 26 Apr 2018 08:34:20 +0000 (10:34 +0200)]
Bump to odlparent 2.0.6

Change-Id: I52a0af9e837ca8c55dd18360128a5559af432b98
Signed-off-by: Stephen Kitt <skitt@redhat.com>
6 years agoRelease permits as transactions are replayed 66/70966/1
Robert Varga [Thu, 12 Apr 2018 11:11:38 +0000 (13:11 +0200)]
Release permits as transactions are replayed

Since we have correct accounting at the cost of needing to finish
the transition to new context in user thread, we need to make sure
we release permits as we push them towards the TransactionContext
if it is not handling it itself -- hence the user threads will not
be charged for operations which have logically been completed.

CONTROLLER-1825
Change-Id: I4bae984b08b99a042766d0a41475110781ef8377
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoFix blueprint event handling 36/70936/1
Tom Pantelis [Mon, 22 Jan 2018 22:15:22 +0000 (17:15 -0500)]
Fix blueprint event handling

The BlueprintBundleTracker registers for blueprint events so it can
log when a bundle finished creating its container, successful or not,
and also has logic to restart the container on dependency failure.
However this was  broken due to changes in Aries. You now have to
register a specific BlueprintListener onstead of a general OGSI
EventHandler.

Also this affected the BlueprintContainerRestartServiceImpl as it
relies on blueprint events to now when all bundles have completed
restart.

Change-Id: Ibb121b3995147a56be7f606ed88e9c1f59e3cd4c
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
(cherry picked from commit 0e0e2378d379e2fbc12d8f93ef41b3267e10f83a)

6 years agoFix TransactionContextWrapper limiter accounting 82/70382/3
Robert Varga [Thu, 5 Apr 2018 14:49:06 +0000 (16:49 +0200)]
Fix TransactionContextWrapper limiter accounting

The fix for CONTROLLER-1814 wrecked the fragile OperationLimiter
accounting between TransactionContextWrapper and
RemoteTransactionContext. The problem is that during initial
connect time TransactionContextWrapper acquires limiter operations
and the state of the OperationLimiter is inherited by
RemoteTransactionContext. This though means that we need to know
which operation actually acquired a permit and which did not.

As it turns out, this needs to make TransactionContext methods take
an additional hint as to whether a limit attempt was made and what
was the result of it.

Furthermore the internal TransactionContextWrapper logic needs to
be changed from 'enqueue and wait' to 'wait and enqueue' strategy,
so when an operation is added to the queue and potentially picked
up by executePriorTransactionOperations() we already know whether
a permit was acquired or not.

JIRA: CONTROLLER-1823
Change-Id: I78d43a1abde8c6da6e3da2f56823bba130499133
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoMove OperationLimiter.acquire() warnings to callers 99/70199/1
Robert Varga [Tue, 23 Jan 2018 16:42:44 +0000 (17:42 +0100)]
Move OperationLimiter.acquire() warnings to callers

We have two distinct call sites from where we are acquiring permits,
where failure to acquire has different root cause:
RemoteTransactionContext's failure to acquire indicates a slow leader,
while TransactionContextWrapper indicates a failure to discover a leader
in time.

Modify acquire() to report success and log failures to acquire from
the two call sites, with different messages so it is immediately clear
what is going on.

Change-Id: Ie828984c6a984690224494af1a9fdd0e4f257fab
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 6a0fab98dc4d16a52922c6594f99a97d4050e800)

6 years agoChange transformation from DOMRpcResult to RpcResult 89/69689/1
Peter Nosal [Mon, 19 Mar 2018 07:57:29 +0000 (08:57 +0100)]
Change transformation from DOMRpcResult to RpcResult

When RpcResult was created, it was always as successful RpcResult,
this is changed so in presence of RpcErrors, RcpResult is build as
failed RpcResult with RpcErrors and binding result.

Change-Id: Ibb74e5f5675e33e0103d1a7d4962176bb320fce3
Signed-off-by: Peter Nosal <peter.nosal@pantheon.tech>
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit fd643adb6b413aac90c4f5a23874beeee486f257)

6 years agoFix RemoteTransactionContext limiter accounting 00/68900/2
Robert Varga [Fri, 9 Feb 2018 15:55:27 +0000 (16:55 +0100)]
Fix RemoteTransactionContext limiter accounting

In case we lose connectivity between the frontend and backend
at the early stages of a big transaction, e.g. after the transaction
is created at the backend and before it is submitted, we can run into
OperationLimiter preventing recovery.

The reason for this is that OperationLimiter itself does not know
how many permits a BatchedModification request contained, hence
on AskTimeoutException it would only decrement permits by one
and the operations would remain throttled. With large transactions
this means the application will suddenly become bogged down
by the OperationLimiter, preventing it from submitting the transaction
or otherwise recovering.

Once any BatchedModifications request fails, the transaction is
doomed anyway, as the message counts on frontend and backend will not
match. Furthermore we must not issue any reads -- the backed does
not have all the modifications, hence it could return an incorrect
result.

Move permit tracking to RemoteTransactionContext, where we can capture
the number of permits in the OnComplete that gets invoked, properly
returning permits which correspond to the BatchedModifications message.
If we have failed to acquire a permit, we also note that and do not
underflow the semaphore.

In case a BatchedModifications message fails, we mark that fact and
turn into a bypass mode: we fail any subsequent reads and do not send
any further BatchedModifications until we see ready being set -- at
which point we coordinate with backend to shoot down the transaction.

An alternative strategy would be to continue transmitting
BatchedModifications, but that would incur an AskTimeout during split,
slowing down the time it takes us to kill flush the doomed transaction
out of the system.

JIRA: CONTROLLER-1814
Change-Id: I919bae0e7173910665e8ec2342d076a710c1c7bf
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 7925d904ffd56c13ddde53e0e7bf6b08b437757d)

6 years agoAdd clustering IT tests to mdsal-artifacts 52/68252/3
Robert Varga [Wed, 14 Feb 2018 15:36:00 +0000 (16:36 +0100)]
Add clustering IT tests to mdsal-artifacts

All artifacts produced should be listed here, fix that.

Change-Id: Ic8c198a69553b471f934e3052742bdc36664ccec
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoFix ReadyLocalTransactionSerializer 12/68412/2
Robert Varga [Mon, 8 Jan 2018 12:22:08 +0000 (13:22 +0100)]
Fix ReadyLocalTransactionSerializer

The following exception was seen in the field:

2017-12-20 19:37:05,507 | ERROR | ult-dispatcher-2 | Remoting                         | 174 - com.typesafe.akka.slf4j - 2.4.7 | java.lang.ClassNotFoundException: org.opendaylight.controller.cluster.datastore.messages.BatchedModifications
org.apache.commons.lang3.SerializationException: java.lang.ClassNotFoundException: org.opendaylight.controller.cluster.datastore.messages.BatchedModifications
        at org.apache.commons.lang3.SerializationUtils.deserialize(SerializationUtils.java:229)
        at org.apache.commons.lang3.SerializationUtils.deserialize(SerializationUtils.java:267)
        at org.opendaylight.controller.cluster.datastore.messages.ReadyLocalTransactionSerializer.fromBinaryJava(ReadyLocalTransactionSerializer.java:49)
        at akka.serialization.JSerializer.fromBinary(Serializer.scala:177)
        at akka.serialization.Serialization$$anonfun$deserialize$2.apply(Serialization.scala:124)
        at scala.util.Try$.apply(Try.scala:192)
        at akka.serialization.Serialization.deserialize(Serialization.scala:114)
        at akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:80)
        at akka.serialization.Serialization$$anonfun$deserialize$2.apply(Serialization.scala:124)
        at scala.util.Try$.apply(Try.scala:192)
        at akka.serialization.Serialization.deserialize(Serialization.scala:114)
        at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:24)
        at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:60)
        at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:60)
        at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:78)
        at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:978)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
        at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:447)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.ClassNotFoundException: org.opendaylight.controller.cluster.datastore.messages.BatchedModifications
        at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:501)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:421)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:412)
        at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:683)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1863)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1746)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2037)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:428)
        at org.apache.commons.lang3.SerializationUtils.deserialize(SerializationUtils.java:223)
        ... 26 more

As it turns out, ReadyLocalTransactionSerializer is not following JSerializer
documentation recommendations of loading classes via ExtendedActorSystem.

Change-Id: Idef62f8c7a50d607ef152083693fac63c7e92447
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit c64b1e26da272928abe57648757d578c2ac33869)

6 years agoBump versions by x.y.(z+1) 66/68566/1
Thanh Ha [Fri, 23 Feb 2018 21:20:35 +0000 (16:20 -0500)]
Bump versions by x.y.(z+1)

Change-Id: Id0514c7b1c020035782dc67727c0afa78c04849a
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
6 years agoGuards iteration against concurrent modification 82/68082/2
Jie Han [Mon, 13 Nov 2017 03:26:39 +0000 (11:26 +0800)]
Guards iteration against concurrent modification

- fix or it would throw a ConcurrentModificationException.
Change-Id: I39664b2238ef62d2add96cb76ac8c2113cfc2265
Signed-off-by: Jie Han <han.jie@zte.com.cn>
(cherry picked from commit a373371d34293ce0e436700ac328a58e9ea37f2e)

6 years agoConcurrentDOMDataBroker LOG debug instead of error 80/68080/2
Michael Vorburger [Mon, 11 Dec 2017 22:24:39 +0000 (23:24 +0100)]
ConcurrentDOMDataBroker LOG debug instead of error

for "Tx: {} Error during phase {}, starting Abort"

see CONTROLLER-1802 and NETVIRT-916 for the full background story to
why this will help reduce confusion when we analyze logs.

Change-Id: I7f791bc92d3c22d96462381d6b966755134647d4
Signed-off-by: Michael Vorburger <vorburger@redhat.com>
(cherry picked from commit 1c717bbf117d3486196a0fdd73ac650721f9c557)

6 years agoRemove stax-utils from features 79/68079/2
Robert Varga [Thu, 25 Jan 2018 14:50:46 +0000 (15:50 +0100)]
Remove stax-utils from features

This dependency should not be needed, removed it.

Change-Id: I712f1ee6a4f525a4053e5082fd3fe8b0fa4aa8a9
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 492bca50c7fdd725c4ba559c5a015efb69817c15)

6 years agoFix infinite loop on cancel transaction 77/68077/1
Jaime Caamaño Ruiz [Thu, 1 Feb 2018 09:02:46 +0000 (10:02 +0100)]
Fix infinite loop on cancel transaction

This patch fixes a problem where you would run into an infinite loop
after cancelling DOMForwardedWriteTransaction following an exception
thrown by the backed transaction ready or submit methods.

Change-Id: I24ce3706dcc52e35890246b4796090cd6b1c99b9
JIRA: CONTROLLER-1812
Signed-off-by: Jaime Caamaño Ruiz <jcaamano@suse.com>
(cherry picked from commit 7ab6f974861e01daa16ff56658eeb1be163cbfec)

6 years agoMDSAL-298: properly handle unkeyed lists 63/67663/3
Robert Varga [Mon, 29 Jan 2018 13:22:35 +0000 (14:22 +0100)]
MDSAL-298: properly handle unkeyed lists

Unkeyed lists are not representable in binding, which means they
cannot be reported as modified children, either. This has implications
for any data change which contains unrepresentable fields as child
modifications.

Previously we would report SUBTREE_MODIFIED for the container node,
but would fail to report any children, which is obviously wrong, as the
user is left guessing as to what exactly happened.

This patch modifies LazyDataObjectModification to report a WRITE event
if modifications to unrepresentable children are found in SUBTREE_MODIFIED
case.

It also reuses BindingStructuralType from md-sal, so that we do not have
two copies of this critical class lying around requiring maintenance.

Change-Id: I30599bc26d3c993c5b6c5e2f9e6f66144a530cb5
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit d6dc62697ecc8fea9098fb2011a666312c1f0ff7)

6 years agoFixup test referring to description statement 75/67975/1
Robert Varga [Tue, 6 Feb 2018 18:25:32 +0000 (19:25 +0100)]
Fixup test referring to description statement

This was broken by yangtools properly processing whitespace,
which retains the newline.

Change-Id: I5796275a80bc1989061fd745270d51a4a37f97bd
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoFix ModificationType.APPEARED mapping 66/67666/2
Robert Varga [Mon, 29 Jan 2018 11:02:41 +0000 (12:02 +0100)]
Fix ModificationType.APPEARED mapping

When a node appears, it is an event equivalent to a WRITE,
not SUBTREE_MODIFIED, otherwise we are logically crossing
a non-existent node.

Change-Id: I0876a18ec4af799db30c384fe4a7e38b9b2833c7
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoFix intermittent RemoteRpcRegistryMXBeanImplTest failures 50/67650/2
Tom Pantelis [Fri, 26 Jan 2018 05:48:41 +0000 (00:48 -0500)]
Fix intermittent RemoteRpcRegistryMXBeanImplTest failures

testFindRpcByRoute(org.opendaylight.controller.remote.rpc.registry.mbeans.RemoteRpcRegistryMXBeanImplTest)  Time elapsed: 0.98 sec  <<< ERROR!
java.lang.IllegalStateException: Attempted to access local bucket before recovery completed
at com.google.common.base.Preconditions.checkState(Preconditions.java:501)
at org.opendaylight.controller.remote.rpc.registry.gossip.BucketStoreActor.getLocalBucket(BucketStoreActor.java:384)
at org.opendaylight.controller.remote.rpc.registry.gossip.BucketStoreActor.getLocalData(BucketStoreActor.java:110)
at org.opendaylight.controller.remote.rpc.registry.mbeans.RemoteRpcRegistryMXBeanImpl.findRpcByRoute(RemoteRpcRegistryMXBeanImpl.java:91)
at org.opendaylight.controller.remote.rpc.registry.mbeans.RemoteRpcRegistryMXBeanImplTest.testFindRpcByRoute(RemoteRpcRegistryMXBeanImplTest.java:142)

The problem is that the RemoteRpcRegistryMXBeanImpl access the enclosing
RpcRegistry Actor instance directly and violates actor encapsulation.
RemoteRpcRegistryMXBeanImpl should access the RpcRegistry via messages
sent to its ActorRef.

Change-Id: Icfd67c38e5d1bc3de283949207009d7aa34ab855
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
(cherry picked from commit 10427641dc0a75ec62e78ecfc4a7a0a7d438d462)

6 years agoregister RemoteRpcRegistryMXBean 49/67649/1
wusandi [Wed, 20 Dec 2017 07:09:12 +0000 (15:09 +0800)]
register RemoteRpcRegistryMXBean

The RemoteRpcRegistryMXBean had not registered because of code refactoring.
I'm sure that it had been removed from one of history commits,
But I think it should be added back to support jmx for query rpcs
when one of rpcs breaks down.

Change-Id: I506ed5c25c7615b8bb7ac9c0102bf671ff40bb78
Signed-off-by: wusandi <wusandi@163.com>
(cherry picked from commit 105587c7c4068aa8a0721669cff6aae7f28f6492)

6 years agoDo not inline javax.annotation 61/67161/1
Robert Varga [Mon, 15 Jan 2018 13:58:55 +0000 (14:58 +0100)]
Do not inline javax.annotation

We should not be including a private copy of javax.annotation,
as we do not need it.

Change-Id: If36ced7ad72b19d1240f49c3224ff65c29ee22a0
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoPrevent non-voting member from becoming entity owner 82/66982/1
Tom Pantelis [Tue, 2 Jan 2018 14:59:07 +0000 (09:59 -0500)]
Prevent non-voting member from becoming entity owner

The use case for a non-voting member is as a backup node so
it makes sense not to allow a non-voting member to become an
entity owner. Also, when a voting state change occurs,
ownership is re-evauated for all entities.

Change-Id: Id8d75c4ae2f91be10aa20e97bdc4448776ae4445
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
(cherry picked from commit 29c8203015aa8f2891c305e82f0cf70c3de3f281)

6 years agoFix io.dropwizard.metrics version 01/66201/2
Michael Vorburger [Thu, 9 Nov 2017 18:32:59 +0000 (19:32 +0100)]
Fix io.dropwizard.metrics version

so that it can be removed from odlparent dependencyManagement
in I0ce0e4b7ad54e171c061e1bb91d568b25e9d1145
for https://jira.opendaylight.org/browse/INFRAUTILS-19

see https://lists.opendaylight.org/pipermail/odlparent-dev/2017-November/001460.html

also minor clean up and remove the io.dropwizard.metrics
dependency in sal-distributed-datastore, because that
actually inherits it from sal-clustering-commons anyway.

Change-Id: I74f41742fbcb18bbb9b798b9a4fe73ce5add87eb
Signed-off-by: Michael Vorburger <vorburger@redhat.com>
(cherry picked from commit 8562ca124ecb4d93f35098cfb20f3cbb569d545b)

6 years agoBug 9060: Fix odl-mdsal-trace's missing mdsaltrace_config.xml 81/66281/2
Michael Vorburger [Tue, 29 Aug 2017 20:06:00 +0000 (22:06 +0200)]
Bug 9060: Fix odl-mdsal-trace's missing mdsaltrace_config.xml

This seems to have gotten lost in the Karaf 4 migration.

see
https://wiki.opendaylight.org/view/Karaf_4_migration#.3Cconfigfile.3E

Change-Id: Id7c20c1daaaeb0844ef2278fe4931b24e7ef5b5d
Signed-off-by: Michael Vorburger <vorburger@redhat.com>
(cherry picked from commit 5efdda583a57c5f4585bd15c3be6d5c188cb75f6)

6 years agoBug 9060: Minor mdsaltrace_config.xml /this/will/never/exist 82/66282/1
Michael Vorburger [Mon, 4 Sep 2017 13:24:42 +0000 (15:24 +0200)]
Bug 9060: Minor mdsaltrace_config.xml /this/will/never/exist

Found during use, and avoids having to do this workaround:

log:set ERROR org.opendaylight.controller.md.sal.trace.dom.impl

Change-Id: Iff5fb5eee8d938f1ec6dcb33d5d8a6ec58f2a2b9
Signed-off-by: Michael Vorburger <vorburger@redhat.com>
6 years agoToaster is shardless 57/65757/2
Robert Varga [Thu, 24 Aug 2017 23:33:54 +0000 (01:33 +0200)]
Toaster is shardless

It's not like we broke it into shards. Nothing like that, our toaster
is fully working. Nevertheless it is a sample and has no place
in production code nor its configuration.

Change-Id: Ie14c698c1ea45a5fe201d1b6227eeb4f2d9790a5
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 9b5705d5d4b396e4076d2bc46d554644ac3f150b)

6 years agoCorrect logging in FrontendClientMetadataBuilder 57/66057/1
Martin Dindoffer [Tue, 21 Nov 2017 17:43:01 +0000 (18:43 +0100)]
Correct logging in FrontendClientMetadataBuilder

Change-Id: I7851d72119607dbb354e213e82091354792b063b
Signed-off-by: Martin Dindoffer <martin.dindoffer@pantheon.tech>
(cherry picked from commit e5d4949c74b1d6cf50c16eaabf5600d255a743f4)

6 years agoForwardingDataBroker 05/65605/2
Michael Vorburger [Thu, 14 Sep 2017 12:25:57 +0000 (14:25 +0200)]
ForwardingDataBroker

as discussed on https://git.opendaylight.org/gerrit/#/c/63120/

Change-Id: I374d2a9f6cbad89f33e453eca270e5cc5f75a224
Signed-off-by: Michael Vorburger <vorburger@redhat.com>
(cherry picked from commit bc13f52e618510599b0dfcd5cd439f1a9b9bf1b5)

6 years agoForwardingRead[Only]/WriteTransaction implementations 03/65603/2
Michael Vorburger [Fri, 22 Sep 2017 22:27:29 +0000 (00:27 +0200)]
ForwardingRead[Only]/WriteTransaction implementations

this came up during https://git.opendaylight.org/gerrit/#/c/63372/

a bit similar (but otherwise unrelated) to my earlier
https://git.opendaylight.org/gerrit/#/c/63135/

Change-Id: If669f94b47bf41f49e54c66c6024aeff9805f8d6
Signed-off-by: Michael Vorburger <vorburger@redhat.com>
(cherry picked from commit 702a44e462672d6e4f7e5bcd94def61b70be8009)

6 years agoBump versions by x.y.(z+1) 60/65960/1
jenkins-releng [Tue, 28 Nov 2017 03:49:03 +0000 (03:49 +0000)]
Bump versions by x.y.(z+1)

Change-Id: I411256032d871df7ba310cd1bcb2604b00686b6f
Signed-off-by: jenkins-releng <jenkins-releng@opendaylight.org>
6 years agoBug 9165: Log config subsystem readiness as INFO 37/63137/3
Vratko Polak [Thu, 14 Sep 2017 13:10:11 +0000 (15:10 +0200)]
Bug 9165: Log config subsystem readiness as INFO

Change-Id: I487760e19ac317f7246ac9b9b47f2a65df100e6b
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
6 years agoAdd debug to pinpoint lastApplied movement 54/62254/3
Robert Varga [Mon, 21 Aug 2017 16:35:20 +0000 (18:35 +0200)]
Add debug to pinpoint lastApplied movement

This method is called from multiple call sites, only one of which
is actually logging the change. Make sure we catch all transitions
by adding a LOG.debug() into the setter.

Change-Id: Ie777f8047a0893f9450fb132faa8adea235fbc5f
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 866cdf141262dd7b57e95b3e3aa75709b7d0c586)

6 years agoLower verbosity in SimpletxDomRead 03/63903/1
Vratko Polak [Mon, 14 Aug 2017 15:45:22 +0000 (17:45 +0200)]
Lower verbosity in SimpletxDomRead

Info on such big objects really makes the log hard to navigate.

Change-Id: I794e06766377ddd6e09e3f2f4142719d6049ac84
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
(cherry picked from commit db81f42c68508645f3a7d81f8041693b208d9a03)

6 years agoBUG-8639: always invalidate primary info cache 94/62994/2
Robert Varga [Mon, 11 Sep 2017 14:03:58 +0000 (16:03 +0200)]
BUG-8639: always invalidate primary info cache

When we remove local shard, make sure we invalidate the associated
cache entry.

Change-Id: I83d6320e7308fe9bdf9c66c928fa91198674eae1
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 39b7a263d64559bc4f593726f56aa38ab9cc0b1c)

6 years agoBump odlparent 2.0.4 to 2.0.5 83/63683/1
Stephen Kitt [Wed, 27 Sep 2017 13:35:13 +0000 (15:35 +0200)]
Bump odlparent 2.0.4 to 2.0.5

Change-Id: Ic7eb7d8fd27f5f3e5ae0f0aea9ab642a9b1b6fac
Signed-off-by: Stephen Kitt <skitt@redhat.com>
(cherry picked from commit b62376fbd434e9e236c841d95384da4ea1572978)

6 years agoBug 9008: Fix the error of the persisted journal data format 30/62530/2
HeYunBo [Fri, 18 Aug 2017 12:31:39 +0000 (20:31 +0800)]
Bug 9008: Fix the error of the persisted journal data format

We have to clear the lastLeafSetQName while processing the end event for node
in NormalizedNodeInputStreamReader and AbstractNormalizedNodeDataOutput.

Otherwise while processing the leaf-list node, the leaf-list entry node
may use the other LeafSetQName as it's node identifier incorrectly.
The DataTree reconstructed from the persisted journal after the controller
restart will be not equal to the DataTree before restart under certain
circumstances.

Change-Id: I4ee823f59fe477d08f982ae73e3850433dfea8ee
Signed-off-by: HeYunBo <he.yunbo@zte.com.cn>
(cherry picked from commit 0077859d16ed922af1449f075033069f4d9dbffe)

6 years agoAdd an explicit null data check 70/62670/2
Robert Varga [Mon, 4 Sep 2017 20:38:11 +0000 (22:38 +0200)]
Add an explicit null data check

We have observed restconf attempting to put null data into a path,
leading to an implicit NPE without any explanation. Add an explicit
guard to catch such attempts.

Change-Id: I2e2125b04782d4f1c050d0aeb72161158da28c9f
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit ee013d68ef799361ba9cdb0dfa5d8af5ece3d48a)

6 years agoFix testLeaderAndFollowerEntityOwnersReassignedAfterShutdown failure 31/63231/2
Tom Pantelis [Thu, 14 Sep 2017 18:50:00 +0000 (14:50 -0400)]
Fix testLeaderAndFollowerEntityOwnersReassignedAfterShutdown failure

testLeaderAndFollowerEntityOwnersReassignedAfterShutdown(org.opendaylight.controller.cluster.datastore.entityownership.DistributedEntityOwnershipIntegrationTest)  Time elapsed: 10.834 sec  <<< FAILURE!
00:35:56 java.lang.AssertionError: Last index expected:<18> but was:<19>
00:35:56  at org.junit.Assert.fail(Assert.java:88)
00:35:56  at org.junit.Assert.failNotEquals(Assert.java:743)
00:35:56  at org.junit.Assert.assertEquals(Assert.java:118)
00:35:56  at org.junit.Assert.assertEquals(Assert.java:555)
00:35:56  at org.opendaylight.controller.cluster.datastore.entityownership.DistributedEntityOwnershipIntegrationTest.lambda$testLeaderAndFollowerEntityOwnersReassignedAfterShutdown$7(DistributedEntityOwnershipIntegrationTest.java:456)
00:35:56  at org.opendaylight.controller.cluster.datastore.MemberNode.verifyRaftState(MemberNode.java:147)
00:35:56  at org.opendaylight.controller.cluster.datastore.entityownership.DistributedEntityOwnershipIntegrationTest.testLeaderAndFollowerEntityOwnersReassignedAfterShutdown(DistributedEntityOwnershipIntegrationTest.java:455)

The problem in this run is that lastIndex on the leader updated after
it was captured. It's difficult to know when all PurgeTransaction payloads
have been committed. The test tries to force follower1 to become leader
after the current leader is shut down but it doesn't really matter which
of the remaining followers becomes leader - we only care that the expected
entities get re-assigned. So I changed the test to enable elections on all
remaining followers and not expect a particular one to grab leadership.

Change-Id: I56a457db9ca6ba07f3a2a6378b745291c54ac99e
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
(cherry picked from commit f899094ab7c5ac2a8bfe141b27ba11c5b9e34b02)

6 years agoFix intermitent testFollowerResyncWith*LeaderRestart failure 17/62417/2
Tom Pantelis [Mon, 28 Aug 2017 19:44:31 +0000 (15:44 -0400)]
Fix intermitent testFollowerResyncWith*LeaderRestart failure

NonVotingFollowerIntegrationTest#testFollowerResyncWithOneMoreLeaderLogEntryAfterNonPersistentLeaderRestart fails intermittently:

NonVotingFollowerIntegrationTest.testFollowerResyncWithOneMoreLeaderLogEntryAfterNonPersistentLeaderRestart:233 Did not receive message of type class org.opendaylight.controller.cluster.raft.base.messages.SnapshotComplete

This seems to be a side-effect of https://git.opendaylight.org/gerrit/#/c/62255/
which changes the timing a bit such that an install snapshot doesn't occur on the
follower which should happen in order to completely re-sycnc it with the leader -
instead it ends up removing the stale out-of-sync entries and appending the new ones
from the leader which gets the journal up-to-date but the stale entries had already
been applied to the state which leaves the state out-of-sync with journal. I added
an additional check in the follower to force the leader to install a snapshot
if the first out-of-sync log entry index <= the lastAppliedIndex which means the
entries to be removed have already been applied to the state.

Change-Id: Ic3815a694a8531d9f7f42f19ad8978d52fc902b3
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
(cherry picked from commit 88e2974b8d391d6e91a6338b0a1b8dbf966a8a71)

6 years agoBump versions by x.y.(z+1) 46/63546/1
Thanh Ha [Tue, 26 Sep 2017 21:13:05 +0000 (17:13 -0400)]
Bump versions by x.y.(z+1)

Change-Id: I635fc9abf3855d948cf957c753b4db7ea91fe6dd
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
6 years agoBug 8741: Fix broken Maven archetype 78/62978/2
Michael Vorburger [Thu, 26 May 2016 13:34:57 +0000 (15:34 +0200)]
Bug 8741: Fix broken Maven archetype

Includes migration to Karaf 4 features, as per
https://wiki.opendaylight.org/view/Karaf_4_migration

The main goal here was just to get what I've found to be a completely
badly broken archetype; it even had some other other problems unrelated
to Karaf 4 migration; this looks like it had been haywire for quite a
while even before the Karaf 4 migration already?

This also contributes a basis for future automated self testing; see also
http://blog2.vorburger.ch/2016/05/how-to-make-maven-archetype-plugin.html,
but this is skipped in this change, because while it works great locally,
it fails for some strange reason related to SFT on (only) Jenkins.

I'm hoping that this just a very first step to ongoing active
maintenance of the archetype (by myself, but hopefully others too),
without ever breaking it anymore - thanks to the new self test. Here are
some possible future TBDs:

* remove archetype.test.skip and fix weird SFT problem (@vorburger)

* re-activate commented out IT which breaks during build self test

* odl-X* feature directory names with ${artifactId} instead of X
   (dunno how to get archetype to create folders named by variables)

* karaf.localFeature which breaks during build self test

* some clean-up e.g. remove useless site generation?

* real test of the produced Karaf distro? Using Pax Exam, but not *IT..

Change-Id: I013c305fb9b63cc965cbcc112cbe6017ae47de21
Signed-off-by: Michael Vorburger <vorburger@redhat.com>
6 years agoBug 9057: Revert "Take advantage of default" ... 51/62851/2
Vratko Polak [Thu, 7 Sep 2017 17:53:31 +0000 (19:53 +0200)]
Bug 9057: Revert "Take advantage of default" ...

... " methods in DOMRpcProviderService"
This reverts commit adf49155eced15c9f654d7bed7ee45cd95686e4f.

This is perhaps only a workaround,
depending on why test app expects class in the first place.

Change-Id: I68840f79aec25b128ddb78edc1a1e5514de16631
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
6 years agoFix intermittent testOwnerChangesOnPeerAvailabilityChanges failure 41/62441/2
Tom Pantelis [Tue, 29 Aug 2017 19:42:12 +0000 (15:42 -0400)]
Fix intermittent testOwnerChangesOnPeerAvailabilityChanges failure

EntityOwnershipShardTest.testOwnerChangesOnPeerAvailabilityChanges:647->AbstractEntityOwnershipTest.verifyRaftState:280->lambda$testOwnerChangesOnPeerAvailabilityChanges$2:648 getRaftState expected:<[]Leader> but was:<[Pre]Leader>

It seems this was indirectly introduced by the addition of the
PurgeTransactionPayload - changes the timing of things a bit. I added
code to ensure peer2's lastAppliedIndex is up-to-date with the leader's
prior to stopping the leader to make it deterministic (ie peer2 should
be able to go straight to Leader).

Change-Id: I9abb950c7dc67b2d481d07b9b421ae46421b6510
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
(cherry picked from commit 1b0f84c4957e464bad6f7cb7350a8171c3d1621b)

6 years agoBUG-9054: do not use BatchedModifications needlessly 23/62523/1
Robert Varga [Wed, 30 Aug 2017 14:28:28 +0000 (16:28 +0200)]
BUG-9054: do not use BatchedModifications needlessly

Transaction identifier, which is a required parameter for
BatchedModifications is a resource tracked on the backend and is
assumed to be allocated contiguously. Using BatchedModifications
to transport only a list of modifications means we are allocating
transactions IDs which we then never use.

This patch reworks the logic so it tracks modifications in a list
and allocates BatchedModifications only when we are ready to actually
commit something.

Change-Id: I3f71511cfd68e96e80790e69d28d083f195e5e12
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 71a4b6377ba598b18c64b89b6b16538751d2d116)

6 years agoBUG-9028: make NonPersistentDataProvider schedule invocation 56/62256/1
Robert Varga [Tue, 22 Aug 2017 17:59:21 +0000 (19:59 +0200)]
BUG-9028: make NonPersistentDataProvider schedule invocation

We need to make NonPersistentDataProvider behave in a fashion
similar to what PersistentDataProvider does for asynchronous
persistence calls, which is schedule execution of the provided
procedure rather than direct execution (which is fair for synchronous
execution).

In order to make that work we introduce ExecuteInSelfActor, which
has an executeInSelf() method, which uses internal mechanics to
schedule the call at a later point.

Change-Id: I116708d98154c8244ea80b4a1a1aa615abc3075d
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit b66d6180f06097e3501a88aac9fb684336addd58)

6 years agoBump karaf-empty dependency version to 2.0.4 57/61757/1
Vratko Polak [Tue, 15 Aug 2017 11:08:24 +0000 (13:08 +0200)]
Bump karaf-empty dependency version to 2.0.4

Change-Id: I15329e8d207fdeaceee4b75a731391f2924bae03
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
6 years agoMake ShardTest.testCommitWhenTransactionHasModifications() wait a bit 42/61742/2
Robert Varga [Mon, 14 Aug 2017 16:03:51 +0000 (18:03 +0200)]
Make ShardTest.testCommitWhenTransactionHasModifications() wait a bit

Committed transactions involve also a purge payload, which is persisted
asynchronously, hence it may or may not be visible in the journal just
after the transaction is reported as committed. Wait for two heartbeat
intervals before looking at the stats.

Change-Id: Ibe699edced12d006bf5ea8cd99aa821ab56d115d
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 3e86a2e5c83a86daba1afa8eeabc14c060c7f454)

6 years agoMake testTransactionForwardedToLeaderAfterRetry purge-aware 43/61743/1
Robert Varga [Mon, 14 Aug 2017 20:53:15 +0000 (22:53 +0200)]
Make testTransactionForwardedToLeaderAfterRetry purge-aware

At the point where we are waiting for transaction replication
to fully propagate, we need to account for the purge request,
as otherwise the configuration could interfere with index
sequencing.

Change-Id: I13f93e306e5b77304916e4c05f39dc28fb9cc049
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit d1de9c55e280fc6a972b0cd408189057446c45a0)

6 years agoUpdate .gitreview to stable/nitrogen 35/61635/1
Thanh Ha [Mon, 14 Aug 2017 16:23:57 +0000 (12:23 -0400)]
Update .gitreview to stable/nitrogen

Change-Id: I9ba7fdbec152043154ead9d827d3b263c05f946f
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
6 years agoAdd MXBean to report shard registered DTCL/DCL info 87/59587/5
Tom Pantelis [Tue, 27 Jun 2017 20:09:32 +0000 (16:09 -0400)]
Add MXBean to report shard registered DTCL/DCL info

It's useful to see what listeners (DTCL/DCL) are registered for each shard.
Added a new message, GetInfo, sent to the DTCL/DCL actors that returns
DataTreeListenerInfo, including the stringified user listener instance
and the stringified YangInstanceIdentifier path. A new
ShardDataTreeListenerInfoMXBean is instantiated for each shard which
reports the DTCL and DCL info.

Change-Id: I312bc5d03fe836bc208ea442ebc2af0ef103120f
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
6 years agoBUG-8941: enqueue purges once ask-based transactions resolve 84/61284/5
Robert Varga [Mon, 7 Aug 2017 16:16:23 +0000 (18:16 +0200)]
BUG-8941: enqueue purges once ask-based transactions resolve

Backend state tracking relies on the transaction log to propagate
transaction state from the leader to followers. This includes purging
of transactions, i.e. the information that the frontend will not need
the state (and the final resolution of the transaction).

Tell-based protocol handles this on the frontend, ask-based needs to
do this on the backend (as it has no notion of transaction continuation).

Change-Id: I49e787b38998ef67b4a9ef504a70822263e1a340
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoEliminate protocol-framework 03/60903/3
Robert Varga [Sat, 29 Jul 2017 09:53:35 +0000 (11:53 +0200)]
Eliminate protocol-framework

This piece of code has been moved to netconf, eliminate it from
controller.

Change-Id: I1a04ed800d88ab49ef6e1d0782ca722f18e16581
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoBump odlparent 2.0.2 to 2.0.4 67/61367/2
Stephen Kitt [Tue, 8 Aug 2017 16:21:03 +0000 (18:21 +0200)]
Bump odlparent 2.0.2 to 2.0.4

Change-Id: Iea7270b110536c10878d130db33409ed08dde987
Signed-off-by: Stephen Kitt <skitt@redhat.com>
6 years agoFixup static method warnings 99/61099/2
Robert Varga [Thu, 3 Aug 2017 09:34:07 +0000 (11:34 +0200)]
Fixup static method warnings

- a method can be made static
- invocation of static methods should not go through an instance

Change-Id: I9380a17432340c75fd94bd01c9dc5bb5cdbd8156
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoBug 8885: Fix DistributedShardedDOMDataTree initialization 80/60780/4
Tom Pantelis [Wed, 26 Jul 2017 16:16:48 +0000 (12:16 -0400)]
Bug 8885: Fix DistributedShardedDOMDataTree initialization

DistributedShardedDOMDataTree initialization expects the prefix
configuration shard to be present and ready with leader however
the latter isn't the case when the static module-shards is
bootstrapped without the local member so it can be dynamically
joined into an existing cluster. So I modified the ConfigShardLookupTask
to elide the ConfigShardReadinessTask.

Once past that, creation of the prefix-based default shard is attempted
as there isn't a local module-based shard however this fails b/c the
local prefix configuration shard is not connected to a leader. To alleviate
this I just commented out the code to create the shard. Since the default
shard configuration is present in the out-of-box modules.conf and is
expected to be present, we can assume at this point that the local member
isn't in the replica list with the intention of dynamically joining it to
an existing cluster, at which time the shard will be created.

These changes at least fix the regression with the boostrapping scenario.
We can revisit this iniialization later w.r.t. prefix-based shards.

Change-Id: I1faf531f4c79914d45203ee132dd4e65ad2f18ba
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
6 years agoDeprecate org.opendaylight.controller.md.sal.dom.spi.AbstractRegistrationTree 78/61078/2
Robert Varga [Wed, 2 Aug 2017 21:32:28 +0000 (23:32 +0200)]
Deprecate org.opendaylight.controller.md.sal.dom.spi.AbstractRegistrationTree

This is a utility class, which has a conterpart in mdsal. Deprecate it
and related classes, migrating users.

Change-Id: I8206350ddb60bb19aed93ff3840e0e68e288d55a
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoDeprecated controller fine-grained sharding APIs 77/61077/1
Robert Varga [Wed, 2 Aug 2017 21:18:57 +0000 (23:18 +0200)]
Deprecated controller fine-grained sharding APIs

These are not maintained nor were they ever implemented. Deprecate
them and point to their mdsal counterparts to reduce confusion.

Change-Id: Idd1908c65b0737df0a3731e6e81a7d1c71f272d0
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoRemove logback related stuff 33/60833/3
Michael Vorburger [Thu, 27 Jul 2017 18:55:27 +0000 (00:25 +0530)]
Remove logback related stuff

as discussed on https://lists.opendaylight.org/pipermail/odlparent-dev/2017-July/001262.html '

Change-Id: I09146cd363d1ab706143bc12c8b1e37aa96c8723
Signed-off-by: Michael Vorburger <vorburger@redhat.com>
6 years agoOptimize use of YangInstanceIdentifier.getPathArguments() 07/60907/6
Robert Varga [Sat, 29 Jul 2017 12:05:36 +0000 (14:05 +0200)]
Optimize use of YangInstanceIdentifier.getPathArguments()

This method returns a list, hence we can lookup the first item
without iterating and also can use Lists.transform().

Change-Id: Ie26bfcc225c74154d65ef963e3444ac5ec10bafb
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoBUG-8898: prioritize InternalCommand 81/60981/1
Robert Varga [Mon, 31 Jul 2017 14:08:22 +0000 (16:08 +0200)]
BUG-8898: prioritize InternalCommand

InternalCommand requests should be processed as soon as possible,
and since we are already using ControlAwareMailbox, this is as simple
as marking InternalCommand as a ControlMessage.

Change-Id: Ic6025f4254da47801676c0c474d03e18abbf8f50
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 2ac32ea2c4f57993a1dc49ef8ce380cb03acc822)

6 years agoBUG-8898: do not invoke timeouts directly 80/60980/1
Robert Varga [Mon, 31 Jul 2017 13:54:12 +0000 (15:54 +0200)]
BUG-8898: do not invoke timeouts directly

Request timeouts are occuring with the connection lock held,
at which point the connection can be at the tail of a successor
chain:

oldestConnection -> olderConnection -> connection

If the callback being invoked attempts to transmit an entry,
we will end up attempting to lock the entire chain. This would not
be a problem except that if there is a concurrent attempt to lock
the entire chain it ends up holding the lock of oldestConnection
and it is waiting for the lock on connection -- which will only be
released once the callback finishes executing, but that in turn
waits for oldestConnection to be unlocked -- a classic AB/BA deadlock.

This patch alleviates the problem by deferring callback execution
via executeInActor, i.e. the timeout will be delivered at as part
of normal message processing.

Change-Id: I237908cf214bcdfd477fe0212d09b207a0c2cdbf
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 4367f456f3c7a30c8ee9c7bca738b3e120a4e1d1)

6 years agoBug 8879: Migrate controller to the new XML parser 13/60913/9
Igor Foltin [Sun, 30 Jul 2017 08:41:42 +0000 (10:41 +0200)]
Bug 8879: Migrate controller to the new XML parser

Migrate blueprint to the new XML parser from YANG tools.

Change-Id: Ib82da2f4b2b49dde7df78f425c700b9d3f473a26
Signed-off-by: Igor Foltin <igor.foltin@pantheon.tech>
6 years agoTake advantage of default methods in DOMRpcProviderService 35/51035/11
Robert Varga [Wed, 25 Jan 2017 22:05:02 +0000 (23:05 +0100)]
Take advantage of default methods in DOMRpcProviderService

We can make one of the methods default, as all implementations
are exactly the same (codifying contract).

Change-Id: I64f0b62fd3a0987ed1ed01ec14b2c4d3b77560ac
Signed-off-by: Robert Varga <rovarga@cisco.com>
6 years agoMake -it- parents use current odlparent version 76/60876/2
Vratko Polak [Mon, 31 Jul 2017 11:34:32 +0000 (13:34 +0200)]
Make -it- parents use current odlparent version

Change-Id: I4fd29a6323eb9181f9629126a51ab578db1f5df2
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
6 years agoconfig-persister-impl: use lambdas 70/57170/4
Stephen Kitt [Tue, 16 May 2017 15:48:58 +0000 (17:48 +0200)]
config-persister-impl: use lambdas

This series of patches uses lambdas instead of anonymous classes for
functional interfaces when possible. Lambdas are replaced with method
references when appropriate.

Change-Id: I20e8b07b839c168d0944c44a57602c3b9a96ce6a
Signed-off-by: Stephen Kitt <skitt@redhat.com>
6 years agoSlice front-end request messages 74/60574/7
Tom Pantelis [Wed, 19 Jul 2017 20:47:28 +0000 (16:47 -0400)]
Slice front-end request messages

Added infrastructure to use the MessageSlicer to slice SliceableMessages in
the TransmitQueue.Transmitting class on the front-end. A MessageAssembler is
used on the Shard side to re-assemble. Currently only the
front-end ModifyTransactionRequest is a SliceableMessage as it contains a
NormalizedNode which can be arbitrarily large - the others are small and
don't require slicing.

Change-Id: I7b09e4864e19d3fdb215c2b9dbcb64c14b6a143c
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
6 years agoblueprint: final parameters 08/56908/4
Stephen Kitt [Thu, 11 May 2017 15:42:46 +0000 (17:42 +0200)]
blueprint: final parameters

This automatically-generated patch flags all appropriate parameters as
final (including caught exceptions).

Change-Id: I565047abcceb31a3da2ef8b2ebdee857e6623196
Signed-off-by: Stephen Kitt <skitt@redhat.com>
6 years agoBug 8494: Separate writing and completion threads 03/60703/1
Vratko Polak [Fri, 21 Jul 2017 10:24:49 +0000 (12:24 +0200)]
Bug 8494: Separate writing and completion threads

If AbstractTransactionHandler uses only one executor thread,
future completion callbacks are delayed by throttling on writes.
CSIT aims to detect RequestTimeoutException within a narrow window,
so a separate executor for callbacks is used now.

The delay would not be that critical, but the problem is the timing
between a scheduled execution which exceeds scheduling gaps. These
seem to hold up normally-submitted tasks, leading to futures never
completing.

Therefore we use two Executors and synchronize state modification
call sites. Hence the two tasks (throttled producer) and future
completions can run concurrently (aside from state synchronization).

Change-Id: I642c5295ab6188b2d7e1b5feae62ab7ef52d41eb
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 8744119235b90d89021567e5f12361d98b823b8f)

6 years agoBUG-8618: refresh transaction access when isolated 02/60702/1
Robert Varga [Mon, 17 Jul 2017 15:11:48 +0000 (17:11 +0200)]
BUG-8618: refresh transaction access when isolated

When we are isolated leader we stop accepting messages from
the frontend. If we remain in this state for more than 15 seconds
this can result in a timeout -- which is obvious, but it really
is our fault.

Since we cannot make forward progress anyway, there is no point
in purging the transaction. Update its access time with whatever
the last mark for that frontend was.

Change-Id: I9ff56c91e4fda4b68cd34c05609dc88d6d65fd32
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 1529bb8bdd4c30a782cf1574b0127833da5831b7)

6 years agoBUG-8792: allow transactions to not time out after reconnect 01/60701/1
Robert Varga [Mon, 17 Jul 2017 12:41:47 +0000 (14:41 +0200)]
BUG-8792: allow transactions to not time out after reconnect

During reconnect churn, the frontend may be catching up with previous
transactions, hence we should hold off timing it out until it does.

When we arrive at a timed out transaction, we allow the access time to
be updated to connect time -- effectively saying the transaction was
touched at the time of reconnect.

Change-Id: I3930b5782579f50931b204d8579c2aee51e2bc55
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 55661ed801e178812f16ac990c93b51a3d68c00e)

6 years agoBUG-8619: do not touch forward path during purge enqueue 00/60700/1
Robert Varga [Thu, 20 Jul 2017 15:51:52 +0000 (17:51 +0200)]
BUG-8619: do not touch forward path during purge enqueue

In case of a purge request, the request is sent from the head
of a connection chain (i.e. the original connection which created
the transaction) and propagated via forwarders. This path needs
to make sure it does not go via throttling, as it is an internal
detail.

Separate the transmit paths a bit more, so that TransmitQueue
can push messages to forwarders' replay path.

Change-Id: I5e146b8d11e8654b4beae3959207efb9c2f18315
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit b83c7f5e5cdaee5f250988182dccb749ac7432c2)

6 years agoBUG-8618: record LeaderFrontendState time 99/60699/2
Robert Varga [Sat, 15 Jul 2017 21:33:25 +0000 (23:33 +0200)]
BUG-8618: record LeaderFrontendState time

In order to deal with IsolatedLeader state and transaction timeouts,
we need to maintain an accurate view of when we have seen the frontend
even if we are not accepting messages from it.

Add correspoding field and maintain it whenever we interact with
LeaderFrontend state. Also record last connect ticks for the same use.

Change-Id: I8e49037507fcd01470a03be8c0d611efca55dabf
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 7633a2a50144dad7cf987b29959dc06509575c05)

6 years agoBug 3401: Remove/cleanup Import-Package in maven-bundle-plugin config 55/59955/5
Tom Pantelis [Wed, 5 Jul 2017 03:41:06 +0000 (23:41 -0400)]
Bug 3401: Remove/cleanup Import-Package in maven-bundle-plugin config

Some of the pom files don't need to explicitly specify Import-Package
in the maven-bundle-plugin configuration. Others were cleaned up to remove
unnecessary entries.

Change-Id: I6b9a741d1a110f17d371497e04e2ab2187aff6b6
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
6 years agoBUG-7464: use yangtools.triemap 61/60561/1
Robert Varga [Wed, 19 Jul 2017 12:08:04 +0000 (14:08 +0200)]
BUG-7464: use yangtools.triemap

Yangtools is moving away from using upstreap Triemap to its
internal fork of that codebase. Switch this code, too.

Change-Id: I0d60ccc8927505a83a35631333203817484da9e0
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoBug 8619: Introduce inheritance of progress trackers 38/60538/1
Vratko Polak [Fri, 14 Jul 2017 15:50:32 +0000 (17:50 +0200)]
Bug 8619: Introduce inheritance of progress trackers

+ Introduce cancelDebt method.
+ Use the newly introduced functionality in client code.
+ Delete unused copy constructors (including unit test).

Change-Id: Ib976343ed5f50c649ea08206c897cb70dead8b86
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 12b4928ef66a82f4a128a11701663ac23143c1d7)

6 years agoSimplify QuarantinedMonitorActor 84/60484/3
Robert Varga [Mon, 17 Jul 2017 12:57:08 +0000 (14:57 +0200)]
Simplify QuarantinedMonitorActor

As per the comments, upstream has provided a dedicated event, hence
use that instead of digging inside akka internals.

Change-Id: I4731dfbbdd228d562ddd32ec5fd3d0e9af0855d0
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoBUG-8143: issue a JVM restart 85/60485/1
Robert Varga [Mon, 17 Jul 2017 13:03:50 +0000 (15:03 +0200)]
BUG-8143: issue a JVM restart

Instead of just restarting the OSGi framework, instruct karaf to
re-execute the JVM.

Change-Id: I10709f61b71d578e4677a5948c23e38f9871c6a1
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoProgressTracker: Decrease delay due nearestAllowed 38/60438/2
Vratko Polak [Fri, 7 Jul 2017 11:14:57 +0000 (13:14 +0200)]
ProgressTracker: Decrease delay due nearestAllowed

If nearestAllowed is in past, that means we have
a temporary interval of relatively small demand for tasks.
We can reduce delay, as if the time since nearestAllowed
was a "delay in advance".

This way the queue stays closer to the intended capacity.

Change-Id: I40f95ea9cb25ea62d8c65ee78cafc79e9b56cc11
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
(cherry picked from commit 80e6514d56cd4dc6aa40997dea2b460723148341)

6 years agoBUG-8618: fix test driver 37/60437/1
Robert Varga [Mon, 10 Jul 2017 14:08:57 +0000 (16:08 +0200)]
BUG-8618: fix test driver

Since the test can produce bursts of completions, which in turn can
get slowed down by writout of new messages, offload future completion
to the executor we have internally. This in turn simplifies things,
as we can rely on state being manipulated (mostly) from a single thread.

Also change ArrayDeque to a HashSet to ensure removal of tasks completes
quickly even in face of misordered responses.

Change-Id: Ia5341633af2dbe3e26e7208436405daf7632a876
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 2be77b3bcef31ad8b6dbdce073471561d2cf76d6)

6 years agoBUG-8618: add pause/unpause mechanics for tell-based protocol 36/60436/1
Robert Varga [Thu, 6 Jul 2017 16:26:44 +0000 (18:26 +0200)]
BUG-8618: add pause/unpause mechanics for tell-based protocol

When we are transitioning to/from paused state, we need to remove
all frontend-related state, including pending transactions, to ensure
ShardDataTree does not track them.

When we change to unpaused leader, we can reconstruct the state
from the journal -- the rest will be forwarded from the frontend anyway.

Change-Id: I28d486d1a6695e21dd7e6518609680d54e5a15eb
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 40d27d44d6f0b0358505b2e8ac5abbad25f47d4b)

6 years agoBUG-8618: introduce RaftActor.unpauseLeader() 19/60419/2
Robert Varga [Thu, 6 Jul 2017 15:15:21 +0000 (17:15 +0200)]
BUG-8618: introduce RaftActor.unpauseLeader()

This is a preparatory patch, which notifies RaftActor when
the operation hooked to pauseLeader() fails to complete and the
leader should resume its normal operation.

This is needed to correctly resume operations of tell-based protocol
after a pauseLeader() completes without actually changing the leader.

Change-Id: Ia00e52ebb327575a484af62bf0c31131a33303b3
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 3a10a45e0f78337435c8bc84015c4724a9fa7741)

6 years agoBUG-8618: eliminate SimpleShardDataTreeCohort subclasses 18/60418/2
Robert Varga [Mon, 3 Jul 2017 17:57:38 +0000 (19:57 +0200)]
BUG-8618: eliminate SimpleShardDataTreeCohort subclasses

Now that we handle pre-cancommit failures useing reportFailure(),
there is no need to have specialized subclasses for cohorts, as
the initial failure can cleanly be handled via nextFailure.

This also places a guard in reportFailure() so we do not override
a failure once it is set -- which should only happen in the case
of a dead-on-arrival transaction and it timing out in READY state.

Change-Id: I057c5b36006843f51d60034d30af83bac4e02cd7
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 2783c9dffdd91dae87d3351f4ebffbd8679e3133)

6 years agoBUG-8618: rework AbstractProxyTransaction.flushState() 17/60417/2
Robert Varga [Thu, 6 Jul 2017 07:04:10 +0000 (09:04 +0200)]
BUG-8618: rework AbstractProxyTransaction.flushState()

Instead of directly forwarding state use ModifyTransactionRequest
to encapsulate state and forward it separately to the successor.

This eliminates sendRequest() from replay path, ensuring the replay
thread is not blocked.

Change-Id: Ice86791d417b7487b9d3b1df06341dd028cde7f8
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit c525e5f25b951daa28d0cbde237ba3040b68f99f)

6 years agoBUG-8618: reconnect connections more aggressively 16/60416/2
Robert Varga [Mon, 3 Jul 2017 17:10:22 +0000 (19:10 +0200)]
BUG-8618: reconnect connections more aggressively

Given that the timeout period on backend for an existing transaction
is 15 seconds, sleeping for 5 seconds between reconnect attempts seems
excessive. Lower the timer to 1 second, which should give us a slightly
better chance to avoid timeouts.

Change-Id: Ib74480f5630865cb7a11ca7027e0495443d1d14e
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 70f287502823bab284555b52b91043c3204b829b)

6 years agoBUG-8618: turn timeouts in READY state into canCommit failures 15/60415/2
Robert Varga [Mon, 3 Jul 2017 16:55:18 +0000 (18:55 +0200)]
BUG-8618: turn timeouts in READY state into canCommit failures

This patch adds more details to the TimeoutException reported when
we prune a transaction while it is in the queue. It also peels the
READY case from the defaults and makes sure we send an authoritative
reply back to the frontend when it requests the transaction to be
committed.

Change-Id: I21364ff7e7103af8be6988b8483adc112c3c1d25
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 0d5408c4babc902d270d9f81ed53c6af93bb2867)

6 years agoBUG-8618: improve logging 14/60414/2
Robert Varga [Mon, 3 Jul 2017 15:30:43 +0000 (17:30 +0200)]
BUG-8618: improve logging

While target sequence is important, we also need to log transmit
sequence, too.

Since this issue involves a state mismatch on the backend, improve
ShardDataTreeCohort logging to include transaction identifier
and state.

Change-Id: I21735870a9ae7983dc14a8f8f4d7464d3448ca60
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit d2d9179e52a0d87aae2b9014b4c36384e24692e3)

6 years agoBug 7449: Slice ReadTransactionSuccess response 07/59407/11
Tom Pantelis [Thu, 22 Jun 2017 14:20:11 +0000 (10:20 -0400)]
Bug 7449: Slice ReadTransactionSuccess response

Added slicing of the ReadTransactionSuccess message. The slicing is
initiated by the Shard usung a MessageSlicer and re-assembly is done
by the ClientActorBehavior on the FE. Introduced a SliceableMessage
interface implemented by ReadTransactionSuccess which Shard checks for
to determine if the response message should be sliced.

Change-Id: Ie55e35aa82a9d2bc21f7a8f24396cb4df467252e
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoBug 8163: getDataTreeChangeListenerExecutor() & DataBrokerTestModule 12/60312/1
Michael Vorburger [Thu, 13 Jul 2017 21:17:24 +0000 (02:47 +0530)]
Bug 8163: getDataTreeChangeListenerExecutor() & DataBrokerTestModule

Adjust AbstractDataBrokerTestCustomizer with a
getDataTreeChangeListenerExecutor() instead of a
setDataTreeChangeListenerExecutor(), just for more consistency with the
existing getCommitCoordinatorExecutor() method.  Also less confusing (to
me) than seeing the private Executor set by default which may get
changed by the setter later.

Adjust ConcurrentDataBrokerTestCustomizer with the
useMTDataTreeChangeListenerExecutor as a constructor argument, instead
of an useMTDataTreeChangeListenerExecutor() method, just for consistency
for how you already have it in AbstractConcurrentDataBrokerTest.

Extend ConstantSchemaAbstractDataBrokerTest and DataBrokerTestModule to
allow passing through this new opt-in tweak flag, so that tests in
downstream projects such as genius and netvirt can staring exploring
enabling this.

Change-Id: I4ad85ac48163d2f4bac865f46a3b047d5b7d333a
Signed-off-by: Michael Vorburger <vorburger@redhat.com>
6 years agoBug 8163: Use MT DTCL executor in AbstractConcurrentDataBrokerTest 22/60022/5
Tom Pantelis [Thu, 6 Jul 2017 12:30:28 +0000 (08:30 -0400)]
Bug 8163: Use MT DTCL executor in AbstractConcurrentDataBrokerTest

Using a direct executor can cause deadlocks so the DTCL executor was
made configurable to use a threadpool as an opt-in. Direct executor
is still the default as many existing tests would break.

Change-Id: I41e14f1e6d3b77a44e61dfc75abff29d11a777dc
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
6 years agoRefactor Follower#handleAppendEntries 85/59085/7
Tom Pantelis [Fri, 16 Jun 2017 14:51:19 +0000 (10:51 -0400)]
Refactor Follower#handleAppendEntries

This method is large - refactor it a bit.

Change-Id: Idae1883accdd7c73b57471501e66398306cf6e91
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
6 years agoRemove <version> from org.apache.aries.blueprint.core dep. 66/58366/2
Michael Vorburger [Tue, 6 Jun 2017 18:24:02 +0000 (20:24 +0200)]
Remove <version> from org.apache.aries.blueprint.core dep.

Because in c/58365a the latest version was added to oldparent
dependencyManagement, and it seems wiser to declare this in only a
single place, to avoid possible version discrepancy problems re. this
artifact in the future.

Change-Id: If4370f3c7e80123fe6c225a11c1224145b3ad2b9
Signed-off-by: Michael Vorburger <vorburger@redhat.com>
6 years agoBUG-8676: add UnsignedLongRangeSet 03/59703/6
Robert Varga [Thu, 29 Jun 2017 16:47:14 +0000 (18:47 +0200)]
BUG-8676: add UnsignedLongRangeSet

This patch adds the wrapper class and updates users to use it directly.
The implementation itself is not changed, that will be done in a follow-up
patch.

Change-Id: Ie240ca5c3c9fc1448629bb5db6ecfa1029f66b8f
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoCleanup warnings 89/59589/11
Robert Varga [Tue, 27 Jun 2017 21:43:40 +0000 (23:43 +0200)]
Cleanup warnings

- pom.xml groupId duplicate
- Futures.addCallback()
- Throwables.propagate*()
- pontentially-static methods
- remove 'throws Exception' where it is not really needed

Change-Id: Ib47e6255e0f510ab7dd0dcd08f71f2dd124df7b7
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
6 years agoFix Verify/Preconditions string format 66/59866/3
Robert Varga [Mon, 3 Jul 2017 08:17:54 +0000 (10:17 +0200)]
Fix Verify/Preconditions string format

These methods take a String.format() string, not a logging one, hence
we are not getting the information we want.

Change-Id: I46de0d64c85594e3d7b8be97951f1cf5249bca8f
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 3e9ac68fea1aef0c7fedec346e50882efdde8acc)

6 years agoBug 6794: Remove threadpool config modules 85/59985/5
Tom Pantelis [Wed, 5 Jul 2017 18:35:41 +0000 (14:35 -0400)]
Bug 6794: Remove threadpool config modules

With the removal of the CSS netconf connector, the threadpool config modules
are no longer used so remove them. The *Wrapper classes are used via blueprint
so they remain. Also removed the eventbus config modules which were deprecated
in Carbon.

Change-Id: Ic528e5817a9f5ccdb67ef41987128ead4db51cbd
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
6 years agoUpgrade to odlparent 2.0.2 90/60090/2
Stephen Kitt [Fri, 7 Jul 2017 15:47:55 +0000 (17:47 +0200)]
Upgrade to odlparent 2.0.2

Change-Id: I748830e39c108056ecd81809a0556e8c43d251f4
Signed-off-by: Stephen Kitt <skitt@redhat.com>
6 years agoSpeed up slow tell-based Distributed*IntegrationTest cases 01/60101/4
Tom Pantelis [Fri, 7 Jul 2017 21:13:59 +0000 (17:13 -0400)]
Speed up slow tell-based Distributed*IntegrationTest cases

Some test cases for tell-based take up to 2 minutes due to the default
2 minute request timeout and 30 sec backend aliveness timer. Speed up
the tests by setting much lower values.

Change-Id: If8dba80d625ded8753178e937f14b435675ef0e4
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
6 years agoMake AbstractClientConnection timeouts configurable 87/60087/4
Tom Pantelis [Fri, 7 Jul 2017 14:59:40 +0000 (10:59 -0400)]
Make AbstractClientConnection timeouts configurable

So we can tweak them in production and unit tests.

Change-Id: I39ce8cdf3cd5397a71f52c42357943dfe5eccb7c
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
6 years agoFix incorrect spelling of fileBackedStreamFactory 64/60064/1
Tom Pantelis [Fri, 7 Jul 2017 08:53:14 +0000 (04:53 -0400)]
Fix incorrect spelling of fileBackedStreamFactory

filedBackedStreamFactory should be fileBackedStreamFactory.

Change-Id: Ib0b65d68d37c5b0ded4f1739d4ddc578973fe6ec
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>