Remove shard-snapshot-chunk-size Remove shard-snapshot-chunk-size and replace its occurrences with maximum-message-slice-size. JIRA: CONTROLLER-1980 Change-Id: Iab862d0789b75a89b6476b4ec048d7344362adf2 Signed-off-by: Robert Varga <robert.varga@pantheon.tech> Signed-off-by: Matej Sramcik <matej.sramcik@pantheon.tech>
Remove ask-based protocol client Remove the use-tell-based-protocol knob, effectively making it always true. This means DistributedDataStore cannot be instantiated, which leads to its removal. That in turn makes all of DatastoreContext, OperationLimiter and similar classes superfluous, so we remove those as well. A few classes are used to drive the shard backend in integration tests, and hence those are moved to test sources. JIRA: CONTROLLER-2054 Change-Id: Ie20b1c898576d3c89b70b34121310e58faddbf8e Signed-off-by: bentom-binoy <bentom.binoy@infosys.com> Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
Obsolete ask-based protocol Mark the switch to use ask-based protocol obsolete and deprecate all classes implementing it. JIRA: CONTROLLER-2053 Change-Id: Ib0f5d6a946090addde255423d51746a52e785b2a Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
Cap shard-snapshot-chunk-size to 480KiB We have a few places where we are using a default of 2MiB for chunk size. Bring them down to 480KiB to match datastore.cfg. JIRA: CONTROLLER-2037 Change-Id: I4dfc4cf3f41acd8c4b782e6450d17f5d5e8bf281 Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
Remove obsolete datastore.cfg properties We have a number of leftovers here, which have been no-ops. Remove them to reduce code clutter. JIRA: CONTROLLER-1984 Change-Id: I490188fb7ebc83c344997861d637852f40fce7a6 Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
Snapshot and journal export on recovery Added ability to export snapshot and journal content into json file during recovery. JIRA: CONTROLLER-1955 Change-Id: Ic2d6181ab56d7b413f06ed91cf5f9d37e3aa2029 Signed-off-by: tadei.bilan <tadei.bilan@pantheon.tech> Signed-off-by: Oleksii Mozghovyi <oleksii.mozghovyi@pantheon.tech> Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
Add direct in-memory journal threshold Some deployments benefit from placing an absolute numeric limit on the retained memory. Introduce a new tunable, which overrides the usual percentange limit. JIRA: CONTROLLER-1956 Change-Id: I688e226b173386765bea74931b6aaf617bda30a8 Signed-off-by: tadei.bilan <tadei.bilan@pantheon.tech> Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
Add optional lz4 compression for snapshots Added ability to use lz4 compression both for snapshots sent to followers and snapshots in storage. JIRA: CONTROLLER-1936 Change-Id: I073120efddde869b10999450057b91e75f0ffe07 Signed-off-by: tadei.bilan <tadei.bilan@pantheon.tech> Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
Add an option to trigger snapshot creation on root overwrites In some cases (such as DAEXIM import), it does not necessarily make sense to retain previous data in the journal, as all of it has been superseded. JIRA: CONTROLLER-1913 Change-Id: I5d634faac06e6764a417c23e88c728373b900924 Signed-off-by: Tibor Král <tibor.kral@pantheon.tech> Signed-off-by: Tomas Cere <tomas.cere@pantheon.tech>
Allow incremental recovery Expose configuration knob in DatastoreContext to specify the amount of recovered journal entries after which a Snapshot should be taken and the journal purged. JIRA: CONTROLLER-1915 Change-Id: I4b20a0abe0329965ca5ac1ab5df7d9ca8480cfb2 Signed-off-by: Tibor Král <tibor.kral@pantheon.tech>
Allow shard settle timeout to be tuned When we are starting the datastore, we wait up to a fixed number of election timeout intervals (3) for shards to finish electing a leader. This is not always enough, especially if recovery takes a long time, hence introduce shardLeaderElectionTimeoutMultiplier to make this tunable. JIRA: CONTROLLER-1914 Change-Id: Iba1d116d0248fc6046aeeae3ec30ecac50f373c9 Signed-off-by: Tibor Král <tibor.kral@pantheon.tech> Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
Specify initial serialization buffer capacity for Payloads JFR shows a lot of re-allocations of the backing byte [] when serializing CommitTransactionPayload. Specify a reasonable initial buffer capacity (the default in ByteArrayOutputStream is 32) to reduce re-allocations. This is also configurable via the cfg file (default is 512). Also did the same for the other smaller Payload classes like PurgeTransactionPayload. JIRA: CONTROLLER-1870 Change-Id: I7ebced56812bfc102409b5b2a8b7f4b1b54359fc Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
Remove CSS modules The CSS features are still installed until we can work out the ModuleInfoBackedContext/ClassLoaderStrategy situation that is currently provided by the ConfigManagerActivator. Change-Id: Id3b2cc287d17532482a2dd893c41e2dcd8d29ff0 Signed-off-by: Tom Pantelis <tompantelis@gmail.com> Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
Eliminate DatastoreContext CSS dependency The CSS-generated Config/OperatonalProperties classes were used to obtain the default values from the yang. That is now done by creating an empty container NormalizedNode and running it thru BindingNormalizedNodeSerializer#fromNormalizedNode to fill in the default values which are then injected into the DatastoreContext builder via reflection. Change-Id: I0963dfcf7850b705d048030056f4b233cc7e4566 Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
Make AbstractClientConnection timeouts configurable So we can tweak them in production and unit tests. Change-Id: I39ce8cdf3cd5397a71f52c42357943dfe5eccb7c Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
Bug 7449: Add maximum-message-slice-size config param Added a new maximum-message-slice-size config param that will be used when fragmenting messages thru the akka remoting framework. This is a generalized version of the shard-snapshot-chunk-size param and replaces it. Change-Id: I4dc4cc0de92d6f876e5587cd8cb3ade2abb59285 Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
BUG-8618: make sync threshold tuneable We are observing quite a few of these transitions, which may be coming from batching scenarios. Introduce sync-index-threshold config knob to expose control over it. Change-Id: Ief4c89c2fe5b95cebaf3fb83cbcdda37cac126b6 Signed-off-by: Robert Varga <robert.varga@pantheon.tech> (cherry picked from commit 890e4bbf40aee318a2174bd4130cf34437e5617b)
Bug 7521: Add FileBackedOutputStream and use for snapshot chunking Added a FileBackedOutputStream class, similar to the one in guava except: - allows for the temp dir to be confgured - keeps track of the count of bytes written for efficiency - uses a PhantomReference to delete the temp file instead of using finalize. FileBackedOutputStream is now used in chunking the snapshop on the leader side and re-assembling on the follower side. Change-Id: Ieea4bc1388ffe18d6803783d2bb714089716b7b1 Signed-off-by: Tom Pantelis <tpanteli@brocade.com>
BUG-5280: split DistributedDataStore Split the DistributedDataStore into two components into an abstract base class and concretization running with TransactionProxies. Add another concretization, which uses DataStoreClient to instantiate requests. Change-Id: I454eec76d54c2fd4e4ea1e5cd16d12398eec81f0 Signed-off-by: Robert Varga <rovarga@cisco.com>
Use configured maximum memory to determine limits Use of Runtime.totalMemory() leads to different behavior as memory grows, meaning we will perform different snapshots based on initial memory sizing as well as the memory pressures from other parts of the system (which mean the total memory will grow differently). Use Runtime.maxMemory() instead, which will cause snapshotting to work consistently with a fully-used system. Change-Id: If6e4a135dc0b9a9d4795d1e5d75f7cbc60550c0d Signed-off-by: Robert Varga <rovarga@cisco.com>