FlowG is a log management platform that lets you ingest, transform, and query logs using a visual pipeline builder. It handles structured logs without requiring predefined schemas and relies on BadgerDB as its storage backend.

For more informations, consult the documentation.

You wish to contribute? Please read our guidelines first.

• Pipelines: Build and manage log processing pipelines visually, with support for transormations with VRL scripts and conditional routing to dedicated streams
• Dynamic Logs: Manage logs with varying structures without needing predefined schemas
• Interoperability: Configuration and log ingestion can entirely be done via the REST API
• Observability: Query and visualize logs in real-time in the integrated Web interface

FlowG's primary goal is to make log refinement and routing as easy as possible. As such, it relies on React Flow to help you build such pipelines with as little code as possible. For the actual code part, we use the Vector Remap Language, which gives a solid base for log refinement.

It aims to replace tools like Logstash by integrating the feature right in the solution.

It also leverages BadgerDB which is a battle tested Key/Value database, with the right feature set to easily support indexing dynamically structured logs, as well as log compression.

Run with (see Build section bellow):

export BENCHMARK_ITERATIONS="1000000"
export BENCHMARK_JOBS="24"  # usually, number of CPUs
task docker:build test:bench

System:

• CPU: 13th Gen Intel(R) Core(TM) i7-13700K
• RAM: 64GB

Result:

Requests sent: 1000000
Total time:    266.75s
Rate:          3748.76 req/s

Requirements:

• Go 1.22+
• C Toolchain
• Rust and Cargo (edition 2021)
• NodeJS

go install github.com/go-task/task/v3/cmd/task@latest
task build:all

Then, start the server with:

./bin/flowg-server \
  --auth-dir ./data/auth \
  --log-dir ./data/logs \
  --config-dir ./data/config \
  --http-bind 127.0.0.1:5080 \
  --mgmt-bind 127.0.0.1:9113 \
  --syslog-bind 127.0.0.1:5514

Now, you can access:

• the WebUI at https://localhost:5080
• the API documentation at https://localhost:5080/api/docs
• the Syslog Server at udp://localhost:5514
• the Management API at https://localhost:9113:
  • /health: health check, always return 200 OK
  • /metrics: Prometheus Exporter

A default user root (password: root) and a default pipeline are bootsrapped if no configuration exists during startup.

To build the Docker image linksociety/flowg:latest locally:

task docker:build

Using Docker:

docker run \
  -p 5080:5080/tcp \
  -p 9113:9113/tcp \
  -p 5514:5514/udp \
  -v flowg-data:/data \
  linksociety/flowg:latest

⚠️ EXPERIMENTAL ⚠️ Using Kubernetes (and Helm):

helm install flowg ./k8s/charts/flowg -n flowg-system --create-namespace

This software is released under the terms of the MIT License