CDS: Real snapshot log trimming changes

CDS: Real snapshot log trimming changes

Bug 2692 changed real snapshotting to trim the in-memory log based on
the current replicatedToAllIndex for the normal case when all follower's
are up and "fake" snapshotting is advancing replicatedToAllIndex.

When a follower is down, real snapshots don't trim the log b/c
replicatedToAllIndex isn't advancing unless the memory threshold is
exceeded. So it will let the in-memory log keep growing past the
snapshot batch count. This sort of defeats the purpose of snapshotting,
ie to keep the journal size in check both on disk and in memory. It's
also a bit dangerous - it can chew up a lot of memory and starve the
rest of the system and cause large STW GC's once the follower comes
back up and the log is cleared. This can also cause multiple snapshots
in the follower once it comes back and catches up - eg, if it's behind
60K entries, it will snapshot after each 20K batch in quick succession.

To alleviate the potential excessive memory growth, in addition to
trimming the log from the captured lastAppliedIndex if the log memory
size threshold is exceeded, I changed the code to do the same if the log
size exceeds the snapshot batch count. So if a follower is down long
enough to exceed the snapshot batch count, the leader will install a
single snapshot to catch up the follower. Otherwise, the follower will
be caught up via AppendEntries.

I also noticed that if snapshot tries happen in quick succession, a
second attempt may be tried prematurely while the previous one is in
the PERSISTING state. This is b/c isCapturing() returns false in the
PERSISTING state. The state machine prevents another capture from
actually initiating b/c only the IDLE state implements capture but I
think to be clean we should disallow it by returning true from
isCapturing() in the PERSISTING state. This avoids state violations
during valid workflows (it's not invalid to attempt a capture while
another is in progress). Therefore, since every state now returns true
for isCapturing() except IDLE, I made true the default in
AbstractSnapshotState so only IDLE overrides it.

I added more end-to-end test cases to
ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest to cover
leader snapshots where the log size exceeds the snapshotBatchCount,
where the memory threshold is exceeded and where neither of the first
2 conditions are met and the log is effectively not trimmed. The former
2 cases trim the log to last applied and result in a snapshot installed on
the follower once it's resumed. The latter case results in the leader
catching up the follower via AppendEntries.

Change-Id: Iaec9ba94232a17d6fa7b192c31c431b328e3d22e
Signed-off-by: Tom Pantelis <tpanteli@brocade.com>