BUG 2437 - Enable snapshotting based on size of data in the in-memory journal 48/13148/4
authorMoiz Raja <moraja@cisco.com>
Wed, 26 Nov 2014 17:07:40 +0000 (09:07 -0800)
committerMoiz Raja <moraja@cisco.com>
Mon, 1 Dec 2014 18:38:53 +0000 (10:38 -0800)
commit82be26d1da9096cb86f2f36e142854003415f4ae
tree1c1fbdb8de8a955c99b8bb390bf29b6a0b44d4a5
parent3e3f54ed3ff69a24f663cf2d3b10e61c49c58ccd
BUG 2437 - Enable snapshotting based on size of data in the in-memory journal

- Changed RaftActor to snapshot based on the transaction count or the size of the
  data in the in-memory journal whichever comes earlier
- The size of data that is used is the rough (not-accurate) size of the Payload
  in the Replicated log entry
- In ShardStats exposed another property which is the data size of the in-memory
  journal
- The snapshot data threshold percentage is configurable using the config sub-system and is
  set to a default of 12%. The reason for setting it at 12% by default is because we have
  a total of 8 default shards out of the box. I could have set this to 16% as toaster is not
  a "real" data shard.
- The snapshot data threshold is calculated as a percentage of the Runtime.totalMemory()
  which is the total memory the jvm considers available for object allocation. From testing
  it appears that the total memory is what would appear in jconsole as the committed memory.

I have not added any unit testing for this - but tested this using the scenario described in
bug 2437 and it seems to work pretty well. The deployment used only 2G of memory and worked
fine for a 7 switch topology and I observed that it had not run out of memory after more than
2 hours.

Change-Id: I09ec0827c0411c42a9224bb6d159d5590c22e20b
Signed-off-by: Moiz Raja <moraja@cisco.com>
19 files changed:
opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/example/messages/KeyValue.java
opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/AbstractReplicatedLogImpl.java
opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/ConfigParams.java
opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/DefaultConfigParamsImpl.java
opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/RaftActor.java
opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/ReplicatedLog.java
opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/ReplicatedLogEntry.java
opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/ReplicatedLogImplEntry.java
opendaylight/md-sal/sal-akka-raft/src/test/java/org/opendaylight/controller/cluster/raft/AbstractReplicatedLogImplTest.java
opendaylight/md-sal/sal-akka-raft/src/test/java/org/opendaylight/controller/cluster/raft/MockRaftActorContext.java
opendaylight/md-sal/sal-clustering-commons/src/main/java/org/opendaylight/controller/cluster/raft/protobuff/client/messages/CompositeModificationPayload.java
opendaylight/md-sal/sal-clustering-commons/src/main/java/org/opendaylight/controller/cluster/raft/protobuff/client/messages/Payload.java
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/DatastoreContext.java
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/Shard.java
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/jmx/mbeans/shard/ShardStats.java
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/jmx/mbeans/shard/ShardStatsMXBean.java
opendaylight/md-sal/sal-distributed-datastore/src/main/yang/distributed-datastore-provider.yang
opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/CompositeModificationPayloadTest.java
opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/programs/appendentries/Client.java