Remove legacy NormalizedNode serialization classes Removed the pre-Lithium protobuff-based NormalizedNode classes and related classes as they are no longer used. Change-Id: I6ae34c9f3778f31bfa26cb4b6d30f3f3eb1f6fc8 Signed-off-by: Tom Pantelis <tpanteli@brocade.com>
Change InstallSnapshot and reply to use Externalizable Proxy This makes InstallSnapshot cleaner with no public no-arg constructor. I also removed the InstallSnapshot protobuff message. In addition, SerializableUtils is no longer needed as there's no more protobuff messages. Change-Id: I17aa4f7195cf09b798daee5587bbf50ccbc4bff0 Signed-off-by: Tom Pantelis <tpanteli@brocade.com>
Remove deprecated Helium Payload methods Removed the deprecated encode/decode methods from Payload and its derived classes. Change-Id: I52beb2d3991efe9f3a7c2631b8682dc1f324e271 Signed-off-by: Tom Pantelis <tpanteli@brocade.com>
Remove unused protobuff messages Removed the following unused protobuff messages: ShardManager.proto MockPayload.proto KeyValueMessages.proto SimpleNormalizedNode.proto Change-Id: I245068fe7c9ead9ea70d9983e73984067c7655da Signed-off-by: Tom Pantelis <tpanteli@brocade.com>
Remove ListenerRegistration protobuff messages The ListenerRegistration protobuff messages are used to serialize the corresponding CDS messages but we don't actually send these messages over the wire so they don't need serialization. So the protobuff messages were removed. If we do need to serialize these messages in he future we won't use protobuff. Change-Id: I3818a965e0fd4e1876364022f2de09b1bac216d5 Signed-off-by: Tom Pantelis <tpanteli@brocade.com>
Remove DataChangeListener protobuff messages The DataChangeListener protobuff messages are used to serialize the corresponding CDS messages but we don't actually send these messages over the wire so they don't need serialization. So the protobuff messages were removed. If we do need to serialize these messages in the future we won't use protobuff. Change-Id: Ia72f3da2edbc6c1eeebfb022916894e2b1713840 Signed-off-by: Tom Pantelis <tpanteli@brocade.com>
Remove Helium ReadDataReply protobuff message The ReadDataReply protobuff message was kept for backwards compatibility with Helium so remove it. Change-Id: I3a30f113903f6d908bea698c0bb9bc773046237e Signed-off-by: Tom Pantelis <tpanteli@brocade.com>
Remove Helium Tx modify operation messages Removed the following deprecated Helium Tx messages and related code: DeleteData[Reply] MergeData[Reply] WriteData[Reply] ReadyTransaction Also removed the associated protobuf message classes. Change-Id: I6f5db0096a60b50e51bf94b12f38941a4be9ca20 Signed-off-by: Tom Pantelis <tpanteli@brocade.com>
Enabling Data Change Notifications for all nodes in cluster. Two new interfaces are introduced ClusteredDataChangeListener and ClusteredDOMDataChangeListener and external applications will have to implement any of that interface, if those applications want to listen to remote data change notifications. Datastore registers listeners, which are instance of that interface, even on followers. Change-Id: I0e29cdf2a08a2051de5fc8ce73b9ec8ac408e45b Signed-off-by: Harman Singh <harmasin@cisco.com>
Bug 2265: Use new NormalizedNode streaming in messages Utilized the new NormalizedNode streaming classes for the WriteData, MergeData, ReadDataReply and DataChanged messages. One solution was to add a bytes field to the protobuff messages and make the previous fields optional. For backwards compatibility, in the wrapper message's fromSerializable method, check whether or not the protobuff message has the bytes field and act accordingly. While this works, it results in an undesirable inefficiency. Protobuff translates the bytes field to a ByteString type. So when streaming, we need to create a ByteArrayOutputStream and pass that to the NormalizedNode stream writer. Then call ByteString.copyFrom(bos.toByteArray) to get the resulting ByteString. However this results in 2 copies of the serialized byte[]. The byte[] cannot be passed to ByteString as is as it always copies it to maintain immutability. I looked into subclassing ByteString and lazily streaming the data on demand but the ByteString ctor is package scoped so they don’t allow subclassing. So I went with an approach to make each message Externalizable instead of using protobuff. The classes writes the version number to enable us to handle compatibility in the future. So in this manner, we can get the efficient direct streaming we want and easily handle backwards and forwards compatibility. I added a VersionedSerializableMessage interface whose toSerializable method takes a version number. The version # is passed from the remote version # obtained from the CreateTransactionReply. This allows the message classes to handle backwards compatibility. So if the version is Helium-1 or less, send the protobuff message otherwise send the Externalizable instance. In the fromSerializable method, it checks if the passed Object is an instance of the Externalizable class or the protbuff message type. Change-Id: I5ebb968e70ac8ff92c29183c52e6c3fe5362ae34 Signed-off-by: tpantelis <tpanteli@brocade.com>
BUG 2371 : Leader should reset it's snapshot tracking when follower is restarted This patch adds a new protocol to InstallSnapshot. It the InstallSnapshotReply returns a failure and the chunkIndex is -1 then the Leader will reset the FollowerSnapshot so that when the next heartbeat occurs the Leader would start sending chunks from the beginning. Change-Id: I0d5f0a4230209856ecf9bcef46220ae348f52b5d Signed-off-by: Moiz Raja <moraja@cisco.com>
BUG 2353 : Handle binary, bits and instanceidentifier types in NodeIdentifiers This fix properly serializes and deserializes NodeIdentifier attributes of the types - binary (byte[]) - yanginstanceidentifier - bits (set) Change-Id: I612d40f9730c939be0594496c26370db96ea6449 Signed-off-by: Moiz Raja <moraja@cisco.com>
BUG 2325 : Value type of byte[] not recognized by the NormalizedNodeSerializer Change-Id: I16eab6cdbf7712624f2c1fafb5bf41107b6ae379 Signed-off-by: Moiz Raja <moraja@cisco.com> (cherry picked from commit 819b04091a3d4d96612a1036228638a3b4e85d09)
BUG 2296 : TransactionProxy should support the ability to accept a local TPC actor path In Helium when the ShardTransaction processes a Ready message it sends back a the path of a ThreePhaseCommitCohort actor in the ReadyReply. This path is actually a local actor path, this local actor path is then converted into a remote path by the TransactionProxy. The fix for Bug 1607 breaks this capability which is required to support forward compatibility in a cluster where a transaction request originates in a node that has been upgraded to Helium-1 and the actual transaction is happening on a node which has not yet been upgraded to Helium-1. Change-Id: I857384bdd61b3492ea270dcf04d14883811c37c2 Signed-off-by: Moiz Raja <moraja@cisco.com>
Bug 2294: Handle Shard backwards compatibility Implemented as outlined in Bug 2294. Change-Id: I14aa8ef5f320f9d165492396ece4ea63cce9b0c3 Signed-off-by: tpantelis <tpanteli@brocade.com>
Bug 2038: Ensure only one concurrent 3-phase commit in Shard Added a ShardCommitCoordinator class that ensures there's only one concurrent 3-phase commit. The following outlines the new commit workflow: - On ready, the ShardTransaction creates the dom store cohort and forwards a new ForwardedReadyTransaction message to the shard. - The shard calls its ShardCommitCoordinator to add the cohort and modificaton to a cached keyed by transaction ID. - On CanCommitTransaction message, the ShardCommitCoordinator looks up and removes the cohort entry from the cache corresponding to the transaction ID passed via the CanCommit message. The ShardCommitCoordinator also caches the cohort entry for the current transaction in progress. If there's no transaction in progress, the committing transaction becomes the current transaction and canCommit is called on the cohort. Otherwise, the cohort entry is queued to be processed after the current tranaction completes. - On CommitTransaction message, if the transaction ID passed via the Commit message matches the currently cached cohort entry, the preCommit and commit phases are performed. When complete, the ShardCommitCoordinator dequeues the next waiting transaction cohort entry, if any, and process it. If a Tx is aborted and it is the current transaction, the ShardCommitCoordinator handles it as a completed Tx. Implemented a timeout mechanism using the akka scheduler such that if the commit message isn't received after a period of time (default 30 s) after the canCommit message, the transaction is aborted so that the next transaction can proceed. This is to handle remote node or network failures during a 3-phrase commit. The ThreePhaseCommitCohort actor was removed along with the ForwardedCommitTransaction. Change-Id: Iaa5692ca45cd7635d1a06a609f4bf98bec50df14 Signed-off-by: tpantelis <tpanteli@brocade.com>
Bug 2003: CDS serialization improvements In NormalizedNodeToNodeCodec#encode, significant time was spent serializing the YangInstanceIdentifier path via PathUtils even though it wasn't actually needed - the decode method didn't decode it. This might have been used by WriteModification and MergeModification originally however they currently serialized/deserialize their YangInstanceIdentifier path separately from the NormalizedNode via InstanceIdentifierUtils. It turns out this takes significant time as well as it's implemented similarly as PathUtils. So I ended up using NormalizedNodeToNodeCodec to encode/decode the YangInstanceIdentifier along with the NormalizedNode but changed InstanceIdentifierUtils to utilize the new PathArgumentSerializer and the NormalizedNodeSerializer's special QName encoding. With serializing a 5K batch of WriteModifications with flow data, the time went down from ~350 ms to ~150 ms. Deserialization was also improved. Other smaller optimizations in NormalizedNodeSerializer, NormalizedNodeType, PathArgumentSerializer and PathArgumentType chopped another 20-30 ms off the time. I also changed InstanceIdentifierUtils to serialize/deserialize via the new PathArgumentSerializer and the NormalizedNodeSerializer's special QName encoding by default, even when the ID isn't encoded as part of a NormalizeNode. This seems reasonable to me as a standalone IID will likely have repeated namespaces and revisions plus we get savings by not serializing each path arg class name. Removed the deprecated InstanceIdentifierUtils class in sal-distributed-datastore bundle. Change-Id: Iaa29daeaececf4b93065f4d46d0c2796c4d8188f Signed-off-by: tpantelis <tpanteli@brocade.com>
BUG 1712 - Distributed DataStore does not work properly with Transaction Chains The fix is as follows, 1. When Creating a trasaction chain create a unique identifier for the transaction chain using the member name and the current timestamp 2. When a transaction is created using the transaction chain pass the transaction chain id along to the remote shard 3. If the remote shard receives a transaction with a valid transaction chain (one which is not empty) then it creates a new transaction chain if one does not exist. If one does exist then the Shard just creates a new transaction on the existing transaction chain. This way if a single transaction chai was used to create transactions on multiple different shards the a transaction chain would be created on each one of those shards. 4. When a transaction chain is closed a Close Transaction Chain message is broadcast to all the Shards in the system. If those shards had a transaction chain with the specified id then the transaction chain would be closed. The sender does not care about receiving a response 5. When a state change occurs on a Shard we check if the Shard is not a leader. If that is the case we automatically close all existing transaction chains on that shard and clear the map which tracks the transaction chains for that shard Change-Id: I6bcfb9de3d0ec666e4152afb69c702dda4f38171 Signed-off-by: Moiz Raja <moraja@cisco.com>
BUG 1595 - Clustering : NPE on startup This seemed to be happening because of recovery where the topology manager was trying to add something into the datastore which was already present in the journal. The fix was to add a timestamp to each modification so that they do not appear to be the same even if their content was exactly the same. Change-Id: I2d5c98bd87ded7c8b64b8cebf757bfa073327061 Signed-off-by: Moiz Raja <moraja@cisco.com>