Split up transaction chunks 13/85013/1
authorRobert Varga <robert.varga@pantheon.tech>
Thu, 19 Sep 2019 23:50:56 +0000 (01:50 +0200)
committerRobert Varga <nite@hq.sk>
Wed, 9 Oct 2019 10:06:11 +0000 (10:06 +0000)
commitd6875afc779570935fb6ff4df03f769cf89a7ae1
treecfa66b453823b88eb20e46d6e192cf14705f52ec
parentbc6c86ebabd6a651a7312a76c22886f49f7a0d20
Split up transaction chunks

This adds the infrastructure to prevent allocation of large byte[]s,
as when those exceed 0.5-16MiB, under G1GC they end up in humongous
object region. Not only that, but after some cut-off point, the
copying of arrays starts to dominate performance.

What we do here is ensure we always receive up to a configurable
number of bytes, defaults to 256KiB, and keep those chunks in a list.
This way we may end up with larger overhead, but that really is
neglibeble -- even a 2GiB payload would end up using only about
8K arrays.

While the input/output streams are similar to
org.apache.commons.io.output.ByteArrayOutputStream, the design here
is geared towards having the intermediate representation available
as well as devolving to a single byte[] for memory efficiency
reasons.

JIRA: CONTROLLER-1920
Change-Id: I2b79a633ebf4fdf8d68d2accc644326e30b41f22
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/persisted/ChunkedByteArray.java [new file with mode: 0644]
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/persisted/ChunkedInputStream.java [new file with mode: 0644]
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/persisted/ChunkedOutputStream.java [new file with mode: 0644]
opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/persisted/CommitTransactionPayload.java
opendaylight/md-sal/sal-distributed-datastore/src/test/java/org/opendaylight/controller/cluster/datastore/persisted/ChunkedOutputStreamTest.java [new file with mode: 0644]