Split up transaction chunks 16/85016/1
authorRobert Varga <robert.varga@pantheon.tech>
Thu, 19 Sep 2019 23:50:56 +0000 (01:50 +0200)
committerRobert Varga <robert.varga@pantheon.tech>
Wed, 9 Oct 2019 12:27:28 +0000 (14:27 +0200)
commitb04a1cc2d691429e5d1bd92c4f4465d36d13f2dd
tree42b321b6241dff367bf8ce84d71e3d3966d0ed79
parentf6b88c1391cca1bd72f3f92e1928c5b354617cbf
Split up transaction chunks

This adds the infrastructure to prevent allocation of large byte[]s,
as when those exceed 0.5-16MiB, under G1GC they end up in humongous
object region. Not only that, but after some cut-off point, the
copying of arrays starts to dominate performance.

What we do here is ensure we always receive up to a configurable
number of bytes, defaults to 256KiB, and keep those chunks in a list.
This way we may end up with larger overhead, but that really is
neglibeble -- even a 2GiB payload would end up using only about
8K arrays.

While the input/output streams are similar to
org.apache.commons.io.output.ByteArrayOutputStream, the design here
is geared towards having the intermediate representation available
as well as devolving to a single byte[] for memory efficiency
reasons.

JIRA: CONTROLLER-1920
Change-Id: I2b79a633ebf4fdf8d68d2accc644326e30b41f22
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit d6875afc779570935fb6ff4df03f769cf89a7ae1)
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/persisted/ChunkedByteArray.java [new file with mode: 0644]
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/persisted/ChunkedInputStream.java [new file with mode: 0644]
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/persisted/ChunkedOutputStream.java [new file with mode: 0644]
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/persisted/CommitTransactionPayload.java
opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/persisted/ChunkedOutputStreamTest.java [new file with mode: 0644]