Chronicle Queue

How Chronicle Works

Chronicle is a Java project focused on building a persisted low latency messaging framework for high performance and critical applications.

What is Different?

This project can be seen as yet another queue implementation but it has major design choices that should be emphasized.

Using non-heap storage options (RandomAccessFile) Chronicle provides a processing environment where applications do not suffer from garbage collection (GC).

While implementing high performance, and memory-intensive, applications in Java, one of the biggest problems is garbage collection. Garbage collection may slow down your critical operations non-deterministically at any time. In order to avoid non-determinism, and escape from GC delays off-heap memory solutions are addressed. The main idea is to manage your memory manually so it does not suffer from GC. Chronicle behaves like a management interface over off-heap memory, so that you can build your own solutions over it.

Chronicle uses RandomAccessFiles while managing memory, and this choice brings a great deal of possibilities. Random access files permit non-sequential, or random, access to a files contents. To access a file randomly, you open the file, seek a particular location, and then read from, or write to, that file. RandomAccessFiles can be seen as large C-type byte arrays that allows you to access any random index directly using pointers. File portions can be used as ByteBuffers if the portion is mapped into memory.

What is the effect of page faults when we have a huge Chronicle and not enough RAM ?

Pages are swapped in and out by the operating system on demand. Writes are performed asynchronously, and under moderate loads do not impact the latency of writes. Sequential reads are also read using look-ahead, that is, before you ask for them. Random reads perform best when the data accessed is already in memory

The size of data stored can exceed the amount of memory you have. Provided the amount you use is less than the main memory size, you see little impact. If you exceed main memory size, you will see measurable performance degradation, but it is dramatically more graceful than using too much heap. Using the same amount of heap can cause the machine to fail.

Building Blocks

Chronicle Queue has two main conceptual components:

Tailer - used for both sequential reads and random reads
Appender - sequential writes

Chronicle Queue is the main interface for management and can be seen as the collection class of the Chronicle environment.

Chronicle Queue allows you to put huge numbers of objects of different sizes onto a single queue. For each data record, Chronicle builds it indexes ( used for random access reads ) automatically when ever messages are appended to the queue.

An appender is used to write/add data to the queue.

Getting Started

In the following example we will:

Create a Chronicle Queue
Put a record to the queue
Read the record from queue.

import net.openhft.chronicle.queue.impl.single.SingleChronicleQueue;
import net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder;

// will write the .cq4 file to working directory
try (SingleChronicleQueue queue = SingleChronicleQueueBuilder.builder().path(".").build();
     ExcerptAppender appender = queue.createAppender();
     ExcerptTailer tailer = queue.createTailer()) {

  // write 'hello world' to the queue
  appender.writeText("hello word");

  // read 'hello world' from the queue
  System.out.println(tailer.readText());
}

Create a tmp directory for the Chronicle Queue giving the name starting Java_temp_directory/SimpleChronicle as the base folder.

File tmpPath = Files.createTempDirectory("SimpleChronicle").toFile();
SingleChronicleQueue queue = SingleChronicleQueueBuilder.builder().path(tmpPath).build();

Create appender and reader:

ExcerptAppender appender = queue.acquireAppender();
ExcerptTailer tailer = queue.createTailer();

The write and read an object

package net.openhft.chronicle.queue;

import net.openhft.chronicle.queue.impl.single.SingleChronicleQueue;
import net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder;
import net.openhft.chronicle.wire.Marshallable;

import java.io.IOException;
import java.nio.file.Files;

public class Example {

    static class MyObject implements Marshallable {
        String name;
        int age;

        @Override
        public String toString() {
            return Marshallable.$toString(this);
        }
    }

    public static void main(String[] args) throws IOException {
        // will write the .cq4 file to working directory
        try (SingleChronicleQueue queue = SingleChronicleQueueBuilder.builder()
                .path(Files.createTempDirectory("queue").toFile()).build();
             ExcerptAppender appender = queue.createAppender();
             ExcerptTailer tailer = queue.createTailer()) {

            MyObject me = new MyObject();
            me.name = "rob";
            me.age = 40;

            // write 'MyObject' to the queue
            appender.writeDocument(me);

            // read 'MyObject' from the queue
            MyObject result = new MyObject();
            tailer.readDocument(result);

           // System.out.println(result);
        }
    }
}

outputs the following:

!net.openhft.chronicle.queue.Example$MyObject {
  name: rob,
  age: 40
}

Tiered Indexing

Chronicle Queue uses tiered multi-level indexing to provide a fast and efficient method for searching for messages in a large queue.

Primary index

# position: 385, header: -1 #  (1)
--- !!meta-data #binary
index2index: [ #  (2)
  # length: 32768, used: 1
  262568, #  (3)
  0, 0, 0, 0, 0,
]

position specifies the starting address of the data that follows, within the queue (*.cq4) file.
index2Index defines a pointer to the next lower level of index.
specifies the point to the next level index.

Note	In this way, indexes can be tiered (primary, secondary, tertiary, etc.) as required to enhance indexing speed and efficiency.

Final level index

# position: 262568, header: -1 #  (1)
--- !!meta-data #binary
index: [ #  (2)
  # length: 32768, used: 1
  524744, #  (3)
  0, 0, 0, 0, 0, 0, 0, 0,
]

position specifies the point in the queue.
index defines a pointer to the data.
specifies the point in the queue where the data begins.

Data location

# position: 524744, header: 0 #  (1)
--- !!data #binary #  (2)
"": some more text
...
# 785952 bytes remaining

position specifies the point in the queue where the data begins.
data defines the information that follows is data (--- !!data #binary #), rather than than meta data (--- !!meta-data #binary).

File rolling

A Chronicle Queue is a logical view of a directory on the file-system. The queue data itself is split across multiple files, each of which contains data belonging to a single cycle. The length of the cycle is determined by the rollCycle parameter passed to the queue builder.

Example configuration of RollCycle:

RollCycles.DAILY events stored in the queue will be grouped into 24-hour periods
RollCycles.HOURLY every hour, a new queue file will be created for written events

As new files are created to accommodate events being written to the queue, a persisted data-structure (directory-listing.cq4t) is updated with the lowest and highest cycle numbers present in the directory.

Maintaining this table allows an ExcerptTailer to busy-spin waiting for new data to be appended to the queue, without the need to make costly calls to the file-system to check for the existence of new queue files.

Read-only mode

When opening a queue in read-only mode, it is not possible to utilise the structure described above (since the structure needs to be written to). In this case, Chronicle Queue will fallback to inspecting the file-system to determine when new queue files are created.

For this reason, significant garbage will be generated when using an ExcerptTailer in a busy-spin loop in read-only mode.

Queue File Reference Counting (Enterprise Edition Feature)

As a queue is written to, it will create a new file for each roll-cycle.

Over time, it may become necessary to automatically delete or archive old files. An automated process needs to ensure that there are not active file-handles open on a queue file before attempting to delete.

To facilitate this operation, Enterprise Chronicle Queue tracks references to its roll-cycle files internally.

The suggested approach is to perform the maintenance operation from a separate JVM to the application, in the following manner:

public void removeOldQueueFiles() throws IOException {
    final Path queuePath = Paths.get("/path/to/queue");
    try (final SingleChronicleQueue queue = SingleChronicleQueueBuilder.
            binary(queuePath).build()) {

        try (final Stream<Path> queueFiles = Files.list(queuePath).
                filter(p -> p.toString().endsWith(SingleChronicleQueue.SUFFIX))) {

            queueFiles.filter(p -> isReadyForDelete(p)).map(Path::toFile).
                    filter(f -> queue.numberOfReferences(f) == 0).
                    forEach(File::delete);

        }
    }
}

Use the ChronicleQueue.numberOfReferences() method to ensure that there are zero references to a given file before attempting to delete it.

Configuration

Chronicle Queue (CQ) can be configured via a number of methods on the SingleChronicleQueueBuilder class.

RollCycle

One such piece of configuration is the RollCycle that determines the rate at which CQ will roll the underlying queue files. For instance, using the following code snippet will result in the queue files being rolled (i.e. a new file created) every hour:

ChronicleQueue.singleBuilder(queuePath).rollCycle(RollCycles.HOURLY).build()

Once a queue’s roll-cycle has been set, it cannot be changed at a later date. More formally, after the first append has been made to a Chronicle Queue, any further instances of SingleChronicleQueue configured to use the same path must be configured to use the same roll-cycle.

This check is enforced by SingleChronicleQueueBuilder, so the following code causes an exception to be thrown:

final Path queueDir = Paths.get("/data/queue/my-queue");
try (ChronicleQueue queue = ChronicleQueue.singleBuilder(queueDir).rollCycle(SECONDLY).build()) {
    // this append call will 'lock' the queue to use the SECONDLY roll-cycle
    try (DocumentContext documentContext = queue.acquireAppender().writingDocument()) {
        documentContext.wire().write("somekey").text("somevalue");
    }
}

// this call will fail since we are trying to create a new queue,
// at the same location, with a different roll-cycle
try (ChronicleQueue recreated = ChronicleQueue.singleBuilder(queueDir).rollCycle(HOURLY).build()) {

}

In the case where a Chronicle Queue instance is created before any appends have been made, and there is a subsequent append operation with a different roll-cycle, then the roll-cycle will be updated to match the persisted roll-cycle. In this case, a warning log message will be printed in order to notify the library user of the situation:

// creates a queue with roll-cycle MINUTELY
try (ChronicleQueue minuteRollCycleQueue = ChronicleQueue.singleBuilder(queueDir).rollCycle(MINUTELY).build()) {
    // creates a queue with roll-cycle HOURLY - valid since no appends have yet been made
    try (ChronicleQueue hourlyRollCycleQueue = ChronicleQueue.singleBuilder(queueDir).rollCycle(HOURLY).build()) {
        // append using the HOURLY roll-cycle
        try (DocumentContext documentContext = hourlyRollCycleQueue.acquireAppender().writingDocument()) {
            documentContext.wire().write("somekey").text("somevalue");
        }
    }
    // now try to append using the queue configured with roll-cycle MINUTELY
    try (DocumentContext documentContext2 = minuteRollCycleQueue.acquireAppender().writingDocument()) {
        documentContext2.wire().write("otherkey").text("othervalue");
    }
}

console output:

[main] WARN SingleChronicleQueueBuilder - Overriding roll cycle from HOURLY to MINUTELY.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!