PQClean has an extensive automated test framework which performs a variety of quality, consistency, and correctness tests. These tests are run as part of our continuous integration checks when code is committed to Github. You can also run the tests locally following the instructions below.

The test framework is written in Python and relies on the pytest testing framework.

Before running the tests, please ensure all required Python packages are installed (pip3 install -r requirements.txt).

You do not need these exact versions to run the tests, but we run tests with:

• GCC 9
• Clang 9

As available on Debian Buster. The powerpc tests are run on an image based on Debian Sid, so they may run different versions.

Different compiler versions or versions of tools, like clang-tidy, may give different warnings.

You can run a test individually, for example:

cd test
python3 test_functest.py 

This will run that test for all schemes. To filter tests by scheme, see the filtering section below.

Use the following command to run all tests:

cd test
pytest
# or
python3 -m pytest

You can use pytest's own filtering functionality using expressions as the following:

cd test
pytest -k "kyber512 and avx2"

By default, the test framework will run all tests for all schemes. The following environment variables can be used to control which tests and schemes are tested:

• PQCLEAN_ONLY_TESTS: Run only the tests specified. Format: comma-separated list of test names, omitting the test_ prefix.
• PQCLEAN_SKIP_TESTS: Run all tests except the ones specified. Format: comma-separated list of test names, omitting the test_ prefix.
• PQCLEAN_ONLY_TYPES: Run tests only on schemes of the types provided. Format: comma-separated list of scheme types, either kem or sign.
• PQCLEAN_SKIP_TYPES: Run tests on all schemes except ones of the types provided. Format: comma-separated list of scheme types, either kem or sign.
• PQCLEAN_ONLY_SCHEMES: Run tests only on the schemes specified. Format: comma-separated list of scheme names.
• PQCLEAN_SKIP_SCHEMES: Run tests on all schemes except the ones specified. Format: comma-separated list of scheme names.

For example, the following configuration runs only two tests on all schemes except one:

cd test
PQCLEAN_ONLY_TESTS=compile_lib,functest \
    PQCLEAN_SKIP_SCHEMES=kyber768 \
    pytest

The environment variable PQCLEAN_ONLY_DIFF has the following effect:

• if PQCLEAN_ONLY_DIFF is set
• and the current directory is a git checkout on a branch other than master
• and the diff of the current directory against the master branch shows changes only against individual schemes, but not the general framework (i.e., inside scheme directories like crypto_kem/scheme1234 or crypto_sign/scheme1234 or in a test/duplicate_consistency file)

then all tests will be run but only on the affected schemes.

• common: Runs functional tests for common crypto functions (AES, SHA-2, SHA-3).
• compile_lib: Checks that each implementation can be successfully built into a static library.
• functest: Checks that each implementation passes basic functional tests, including:
  • successful encapsulation and decapsulation (for KEMs);
  • successful signature verification (for signature schemes);
  • does not overwrite memory outside of its allocated buffers;
  • fails when secret keys or ciphertexts are modified (for KEMs);
  • fails when public keys are modified (for signature schemes).
• functest_santizers: Checks that each implementation passes the above basic functional tests even when clang's address sanitizer is enabled (-g -fsanitize=address,undefined).
• nistkat: Checks that each KEM implementation produces the same (hash of the) known answer test values as specified in the NIST submission. See test_nistkat.py for details on generating the hash value.
• testvectors: Checks that each signature scheme implementation produces the same (hash of the) test vectors as specified in the META.yml file. This value is generated by building and running bin/testvectors_scheme_implementation | sha256sum when the scheme is first imported to PQClean; the test harness then checks that the implementation continues to generate the same values.
• valgrind: Uses valgrind to check that no memory errors are encountered when running the above basic functional tests.

• api_h: Checks that the api.h file for each implementation is self-contained and does not include any dependencies via #include. Rationale: make it simple to include api.h in other C code without a chain of header files.
• char: Checks that each implementation does not make unqualified use of the char type. Rationale: signed and unsigned use is fine, but char on its own can be ambiguous.
• dynamic_memory: Checks that each implementation does not use dynamic memory functions (malloc, free, realloc, calloc). Rationale: dynamic memory functions not available on all platforms.
• format: Uses astyle to check that every .c and .h file in each implementation meets the project's C formatting requirements as specified in the .astylerc file. Rationale: consistent formatting makes code easier to read.
• linter: Uses clang-tidy to check that each implementation passes the C linter requirements specified in the .clang-tidy file. Rationale: can catch certain classes of errors.
• preprocessor: Checks that every .c and .h file in each implementation does not use preprocessor conditionals (#if, #ifdef, #ifndef). Note that header guards (#ifndef on the first line of a .h file) are permitted. Rationale: "clean" code can be harder to read if there are complex preprocessor macros.
• symbol_namespace: Checks that the symbols exported in the library generated for each implementation are all properly namespaced (i.e., start with PQCLEAN_SCHEME_IMPLEMENTATION_ for the current scheme and implementation). Rationale: PQClean implementations should be in their own namespace to avoid colliding with other programs' symbols.

• duplicate_consistency: Uses the test/duplicate_consistency YAML files to check that any files that are meant to be similar between implementations (e.g., same file between two parameterizations of the same scheme) are kept consistent, up to namespacing.
• license: Checks that a LICENSE or LICENSE.txt file is present for each implementation.
• makefile_dependencies: Checks that every .c and .h file in each implementation is a dependency of the implementation's Makefile and that changing any one of them triggers a rebuild.
• metadata: Checks that the META.yml file for each scheme contains all relevant fields.
• metadata_sizes: Checks that, for each implementation, the parameter sizes (public key, secret key, etc.) in the implementation's api.h file match the values in the scheme's META.yml file.
• no_symlinks: Checks that each implementation does not contain symlinks to any other files.