+Failure handling
+----------------
+
+Overview
+--------
+
+A fundamental problem in distributed systems is that network
+partitions (split brain scenarios) and machine crashes are indistinguishable
+for the observer, i.e. a node can observe that there is a problem with another
+node, but it cannot tell if it has crashed and will never be available again,
+if there is a network issue that might or might not heal again after a while or
+if process is unresponsive because of overload, CPU starvation or long garbage
+collection pauses.
+
+When there is a crash, we would like to remove the affected node immediately
+from the cluster membership. When there is a network partition or unresponsive
+process we would like to wait for a while in the hope that it is a transient
+problem that will heal again, but at some point, we must give up and continue
+with the nodes on one side of the partition and shut down nodes on the other
+side. Also, certain features are not fully available during partitions so it
+might not matter that the partition is transient or not if it just takes too
+long. Those two goals are in conflict with each other and there is a trade-off
+between how quickly we can remove a crashed node and premature action on
+transient network partitions.
+
+Split Brain Resolver
+--------------------
+
+You need to enable the Split Brain Resolver by configuring it as downing
+provider in the configuration::
+
+ akka.cluster.downing-provider-class = "akka.cluster.sbr.SplitBrainResolverProvider"
+
+You should also consider different downing strategies, described below.
+
+.. note:: If no downing provider is specified, NoDowning provider is used.
+
+All strategies are inactive until the cluster membership and the information about
+unreachable nodes have been stable for a certain time period. Continuously adding
+more nodes while there is a network partition does not influence this timeout, since
+the status of those nodes will not be changed to Up while there are unreachable nodes.
+Joining nodes are not counted in the logic of the strategies.
+
+Setting ``akka.cluster.split-brain-resolver.stable-after`` to a shorter duration for having
+quicker removal of crashed nodes can be done at the price of risking a too early action on
+transient network partitions that otherwise would have healed. Do not set this to a shorter
+duration than the membership dissemination time in the cluster, which depends on the cluster size.
+Recommended minimum duration for different cluster sizes:
+
+============ ============
+Cluster size stable-after
+============ ============
+5 7 s
+10 10 s
+20 13 s
+50 17 s
+100 20 s
+1000 30 s
+============ ============
+
+.. note:: It is important that you use the same configuration on all nodes.
+
+When reachability observations by the failure detector are changed, the SBR
+decisions are deferred until there are no changes within the stable-after
+duration. If this continues for too long it might be an indication of an
+unstable system/network and it could result in delayed or conflicting
+decisions on separate sides of a network partition.
+
+As a precaution for that scenario all nodes are downed if no decision is
+made within stable-after + down-all-when-unstable from the first unreachability
+event. The measurement is reset if all unreachable have been healed, downed or
+removed, or if there are no changes within stable-after * 2.
+
+Configuration::
+
+ akka.cluster.split-brain-resolver {
+ # Time margin after which shards or singletons that belonged to a downed/removed
+ # partition are created in surviving partition. The purpose of this margin is that
+ # in case of a network partition the persistent actors in the non-surviving partitions
+ # must be stopped before corresponding persistent actors are started somewhere else.
+ # This is useful if you implement downing strategies that handle network partitions,
+ # e.g. by keeping the larger side of the partition and shutting down the smaller side.
+ # Decision is taken by the strategy when there has been no membership or
+ # reachability changes for this duration, i.e. the cluster state is stable.
+ stable-after = 20s
+
+ # When reachability observations by the failure detector are changed the SBR decisions
+ # are deferred until there are no changes within the 'stable-after' duration.
+ # If this continues for too long it might be an indication of an unstable system/network
+ # and it could result in delayed or conflicting decisions on separate sides of a network
+ # partition.
+ # As a precaution for that scenario all nodes are downed if no decision is made within
+ # `stable-after + down-all-when-unstable` from the first unreachability event.
+ # The measurement is reset if all unreachable have been healed, downed or removed, or
+ # if there are no changes within `stable-after * 2`.
+ # The value can be on, off, or a duration.
+ # By default it is 'on' and then it is derived to be 3/4 of stable-after, but not less than
+ # 4 seconds.
+ down-all-when-unstable = on
+ }
+
+
+Keep majority
+^^^^^^^^^^^^^
+
+This strategy is used by default, because it works well for most systems.
+It will down the unreachable nodes if the current node is in the majority part
+based on the last known membership information. Otherwise down the reachable
+nodes, i.e. the own part. If the parts are of equal size the part containing the
+node with the lowest address is kept.
+
+This strategy is a good choice when the number of nodes in the cluster change
+dynamically and you can therefore not use static-quorum.
+
+* If there are membership changes at the same time as the network partition
+ occurs, for example, the status of two members are changed to Up on one side
+ but that information is not disseminated to the other side before the
+ connection is broken, it will down all nodes on the side that could be in
+ minority if the joining nodes were changed to Up on the other side.
+ Note that if the joining nodes were not changed to Up and becoming a majority
+ on the other side then each part will shut down itself, terminating the whole
+ cluster.
+
+* If there are more than two partitions and none is in majority each part will
+ shut down itself, terminating the whole cluster.
+
+* If more than half of the nodes crash at the same time the other running nodes
+ will down themselves because they think that they are not in majority, and
+ thereby the whole cluster is terminated.
+
+The decision can be based on nodes with a configured role instead of all nodes
+in the cluster. This can be useful when some types of nodes are more valuable
+than others.
+
+Configuration::
+
+ akka.cluster.split-brain-resolver.active-strategy=keep-majority
+
+::
+
+ akka.cluster.split-brain-resolver.keep-majority {
+ # if the 'role' is defined the decision is based only on members with that 'role'
+ role = ""
+ }
+
+Static quorum
+^^^^^^^^^^^^^
+
+The strategy named static-quorum will down the unreachable nodes if the number
+of remaining nodes are greater than or equal to a configured quorum-size.
+Otherwise, it will down the reachable nodes, i.e. it will shut down that side
+of the partition.
+
+This strategy is a good choice when you have a fixed number of nodes in the
+cluster, or when you can define a fixed number of nodes with a certain role.
+
+* If there are unreachable nodes when starting up the cluster, before reaching
+ this limit, the cluster may shut itself down immediately.
+ This is not an issue if you start all nodes at approximately the same time or
+ use the ``akka.cluster.min-nr-of-members`` to define required number of
+ members before the leader changes member status of ‘Joining’ members to ‘Up’.
+ You can tune the timeout after which downing decisions are made using the
+ stable-after setting.
+
+* You should not add more members to the cluster than quorum-size * 2 - 1.
+ If the exceeded cluster size remains when a SBR decision is needed it will
+ down all nodes because otherwise there is a risk that both sides may down each
+ other and thereby form two separate clusters.
+
+* If the cluster is split into 3 (or more) parts each part that is smaller than
+ then configured quorum-size will down itself and possibly shutdown the whole
+ cluster.
+
+* If more nodes than the configured quorum-size crash at the same time the other
+ running nodes will down themselves because they think that they are not in the
+ majority, and thereby the whole cluster is terminated.
+
+The decision can be based on nodes with a configured role instead of all nodes
+in the cluster. This can be useful when some types of nodes are more valuable
+than others.
+
+By defining a role for a few stable nodes in the cluster and using that in the
+configuration of static-quorum you will be able to dynamically add and remove
+other nodes without this role and still have good decisions of what nodes to
+keep running and what nodes to shut down in the case of network partitions.
+The advantage of this approach compared to keep-majority is that you do not risk
+splitting the cluster into two separate clusters, i.e. a split brain.
+
+Configuration::
+
+ akka.cluster.split-brain-resolver.active-strategy=static-quorum
+
+::
+
+ akka.cluster.split-brain-resolver.static-quorum {
+ # minimum number of nodes that the cluster must have
+ quorum-size = undefined
+
+ # if the 'role' is defined the decision is based only on members with that 'role'
+ role = ""
+ }
+
+Keep oldest
+^^^^^^^^^^^
+
+The strategy named keep-oldest will down the part that does not contain the oldest
+member. The oldest member is interesting because the active Cluster Singleton
+instance is running on the oldest member.
+
+This strategy is good to use if you use Cluster Singleton and do not want to shut
+down the node where the singleton instance runs. If the oldest node crashes a new
+singleton instance will be started on the next oldest node.
+
+* If down-if-alone is configured to on, then if the oldest node has partitioned
+ from all other nodes the oldest will down itself and keep all other nodes running.
+ The strategy will not down the single oldest node when it is the only remaining
+ node in the cluster.
+
+* If there are membership changes at the same time as the network partition occurs,
+ for example, the status of the oldest member is changed to Exiting on one side but
+ that information is not disseminated to the other side before the connection is
+ broken, it will detect this situation and make the safe decision to down all nodes
+ on the side that sees the oldest as Leaving. Note that this has the drawback that
+ if the oldest was Leaving and not changed to Exiting then each part will shut down
+ itself, terminating the whole cluster.
+
+The decision can be based on nodes with a configured role instead of all nodes
+in the cluster.
+
+Configuration::
+
+ akka.cluster.split-brain-resolver.active-strategy=keep-oldest
+