PIXL Driver + Command line interface

The PIXL CLI driver provides functionality to populate a queue with messages containing information required to run electronic health queries against the VNA image system. Once a set of queues are populated the consumers can be started, updated and the system extractions stopped cleanly.

Prerequisites

Python version 3.11 (matching python versions in pixl-ci and dev).
Docker with version >=27.0.3
Docker Compose with version >=v2.28.1-desktop.1
uv to install PIXL in a virtual environment See detailed instructions here

Installation

Activate your python virtual environment and install PIXL project in editable mode by running

uv sync

Usage

Note: The rabbitmq, export-api and imaging-api services must be started prior to using the CLI This is done by spinning up the necessary Docker containers through docker compose.

See general pixl commands and subcommands with:

pixl --help

For convenience, we provide the pixl dc command, which acts as a wrapper for docker compose, but takes care of some of the configuration for you.

For example,

pixl dc up

will run docker compose --project pixl_{pixl_env} up --wait --build --remove-orphans, where pixl_env is determined by the ENV environment variable.

Note, if your .env file is in a different directory from the docker-compose.yml file, you can pass in the path to you .env file using the --env-file argument.

Configuration

Rabbit MQ and PostgreSQL

The rabbitmq and PIXL DB postgres services are configured by setting the following environment variables (default values shown):

RABBITMQ_HOST=localhost
RABBITMQ_PORT=7008
RABBITMQ_USERNAME=rabbitmq_username
RABBITMQ_PASSWORD=rabbitmq_password

CLI_PIXL_DB_HOST=localhost
CLI_PIXL_DB_PORT=7001
PIXL_DB_USER=pixl_db_username
PIXL_DB_PASSWORD=pixl_db_password
PIXL_DB_NAME=pixl

The rabbitmq queues for the imaging API is configured by setting:

PIXL_IMAGING_API_HOST=localhost
PIXL_IMAGING_API_PORT=7007
PIXL_IMAGING_API_RATE=1

where the *_RATE variables set the default querying rate for the message queues.

Host directories

The PIXL root and export directories on the host machine can be configured using the following environment variables:

PIXL_ROOT=../
HOST_EXPORT_ROOT_DIR=../projects/exports
HOST_EXPORT_ROOT_DIR_MOUNT=./projects/exports

The PIXL_ROOT directory must contain the docker-compose.yml file and projects/configs folders from the top-level directory of this repository. PIXL_ROOT must also contain the .sample.env file if you would like to use the pixl check_env command. This path can be absolute or relative to your .env file (which must be in your current working directory). This variable is used by the PIXL CLI when running PIXL.

The HOST_EXPORT_ROOT_DIR is the directory on your host machine that data will be exported to. This path can be absolute or relative to your .env file. This variable is used by the PIXL CLI when running PIXL.

The HOST_EXPORT_ROOT_DIR_MOUNT is the directory on your host machine that will be mounted for exporting data to. The path can be absolute or relative to the PIXL_ROOT. This variable is used by docker-compose to mount the export directory when starting PIXL.

Running the pipeline

Populate queue for Imaging using parquet files:

pixl populate </path/to/parquet_dir>

where parquet_dir contains at least the following files:

parquet_dir
├── extract_summary.json
├── private
│   ├── PERSON_LINKS.parquet
│   └── PROCEDURE_OCCURRENCE_LINKS.parquet
└── public
    └── PROCEDURE_OCCURRENCE.parquet

Alternatively, the queue can be populated based on records in CSV files:

pixl populate <path/to/file.csv>

One advantage of using a CSV file is that multiple projects can be listed for export in the file. Using the parquet format, in contrast, only supports exporting a single project per call to pixl populate.

Extraction will start automatically after populating the queues. If granular customisation of the rate per queue is required or a queue should not be started then supply the argument --no-start and use pixl start... to launch processing.

Once the messages have been processed, the OMOP extracts (including radiology reports) can be exported to a parquet file using

pixl export-patient-data </path/to/parquet_dir>

Stop Imaging extraction

pixl stop

High-priority messages

By default, messages will be sent to the queue with the lowest priority (1).

To send to the queue with a different priority, you can use the --priority argument to populate:

pixl populate --priority 5 <path/to/file.csv>

priority must be an integer between 1 and 5, with 5 being the highest priority.

Development

Help commands

The CLI is created using click. To see which commands are currently available, you can use the pixl --help command:

Local installation

Activate your python environment and install project locally in editable mode with the development and testing dependencies by running

uv sync

Running tests

The CLI tests require a running instance of the rabbitmq service, for which we provide a docker-compose file. The service is automatically started by the run_containers pytest fixture. So to run the tests, run

pytest -vs tests #for all tests
pytest -vs tests/test_docker_commands.py #e.g., for particular tests

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!