CARVIEW |
Navigation Menu
-
Notifications
You must be signed in to change notification settings - Fork 124
Description
This is a bit of a doozy.
We're in the process of moving an application to lmdbjava. We observe that when running the tests the following occurs:
- If the tests are all run in the same JVM process then we consistently get very strange errors related to ByteBuf validation constraints:
For example:
Caused by: java.lang.IndexOutOfBoundsException: readerIndex: 1297, writerIndex: 1095 (expected: 0 <= readerIndex <= writerIndex <= capacity(1095))
at io.netty.buffer.AbstractByteBuf.checkIndexBounds(AbstractByteBuf.java:112)
at io.netty.buffer.AbstractByteBuf.writerIndex(AbstractByteBuf.java:135)
at org.lmdbjava.ByteBufProxy.out(ByteBufProxy.java:166)
at org.lmdbjava.ByteBufProxy.out(ByteBufProxy.java:41)
at org.lmdbjava.KeyVal.valOut(KeyVal.java:134)
at org.lmdbjava.Cursor.seek(Cursor.java:377)
at org.lmdbjava.Cursor.first(Cursor.java:126)
at org.lmdbjava.CursorIterable.executeCursorOp(CursorIterable.java:125)
at org.lmdbjava.CursorIterable.update(CursorIterable.java:172)
at org.lmdbjava.CursorIterable.access$100(CursorIterable.java:52)
at org.lmdbjava.CursorIterable$1.hasNext(CursorIterable.java:99)
This occurs using the very standard code to iterate over all the keys in a table:
try (final CursorIterable<ByteBuf> cursor = table.iterate(txn)) {
for (final KeyVal<ByteBuf> kv : cursor) {
final ByteBuf keyBuffer = kv.key();
final ByteBuf valueBuffer = kv.val();
- If the maven build is run using -DforkCount=1C -DreuseForks=false then all the tests run perfectly fine. These system properties tell maven to run each test class in its own JVM and it eliminates all errors in the build at the expense of a much slower build.
Additionally if we run a single test using the -Dtest=$TestName syntax the test passes fine.
Clearly there is some erorr related to static state that is leading to inter-test dependencies. If we eliminate static state by running each Test in its own JVM all the tests pass. If we run the tests run the tests in the same JVM fails.
I've tried to dig into this and get some idea of what's happening and I doi believe the problem is related to the ByteBufProxy class. There are some red flags in this class:
-
This class maintains significant static state in a threadlocal BUFFERS cache. There seems to be no way to reset or clear this BUFFERS. As far I can tell the cache persists even if you call Env.close() and it might explain why we're getting ByteBufs whose readIndex is larger than their writeIndex.
-
The ByteBufProxy class is also hardcoded to use PooledByteBufAllocator.DEFAULT. We've had problems in the past with multiple components (netty, netty extensions) all mucking around with PooledByteBufAllocator.DEFAULT. We tend to avoid using PooledByteBufAllocator.DEFAULT and instead having each component use its own explicit PooledByteBufAllocator. Unfortunately there's no way to configure ByteBufProxy to use a different ByteBufAllocator.
-
The ByteBufProxy class makes several questionable assumptions. ByteBufs are allocated and not released, and no attempt is made to clear() ByteBufs returned from the pool.