Available Containerized Benchmarks

A list of all available benchmarks

For more details, we refer to the code of each benchmark in hpobench.benchmark.<family>.

Family	Name	Type	Note
pybnn(x)	BNNOn*	raw	There are 4 benchmark in total (`ToyFunction`, `BostonHousing`, `ProteinStructure`, `YearPrediction`)
lr_benchmark	LRBenchmark	raw	Works on OpenML Dataset IDs
nn_benchmark	NNBenchmark	raw	Works on OpenML Dataset IDs
rf_benchmark	RandomForestBenchmark	raw	Works on OpenML Dataset IDs
svm_benchmark	SVMBenchmark	raw	Works on OpenML Dataset IDs
tabular_benchmark(x)	TabularBenchmark	tabular	Tabular benchmarks for the benchmarks above ('lr', 'svm', 'xgb', 'rf', 'nn')
histgb_benchmark	HistGBBenchmark	raw	Works on OpenML Dataset IDs
xgboost_old	XGBoost*Benchmark	raw	Works with OpenML task ids. `extended` contains Additional Parameter `Booster
svm_benchmark_old	SupportVectorMachine	raw	Works with OpenML task ids.

Family	Name	Type	Note
cartpole	CartpoleFull	raw	Not deterministic.
cartpole(x)	CartpoleReduced	raw	Not deterministic. Uses a smaller searchspace than `full`
learna_benchmark	LeaRNA	Not deterministic.
learna_benchmark	MetaLearna	Not deterministic.

Family	Name	Type	Note
tabular_benchmarks(+)	*Benchmark	tabular	There are 4 benchmarks (`SlizeLocalization`, `ProteinStructure`, `NavalPropulsion`, `ParkinsonsTelemonitoring`)
tabular_benchmarks(+,x)	*BenchmarkOriginal	tabular	Same as above, but with a higher lowest fidelity of `3`
nasbench_101(+)	NASCifar10*Benchmark	tabular	There are 3 benchmark in total (A, B, C)
nasbench_201(+)	*NasBench201Benchmark	tabular	There are 3 benchmarks in total (Cifar10Valid, Cifar100, ImageNet). Note: We assume that there are 3 seeds per configuration (777,888,999). For configurations with less seeds we re-use of the existing ones (See #138).
nasbench_201(+,x)	*NasBench201BenchmarkOriginal	tabular	Same as above, but with a higher lowest fidelity of `12`
nasbench_1shot1(+)	NASBench1shot1SearchSpace*Benchmark	tabular	There are 3 benchmarks in total (1,2,3)

Family	Name	Type	Note
paramnet_benchmark	ParamNet*OnStepsBenchmark	paramnet	surrogate
paramnet_benchmark	ParamNetReduced*OnStepsBenchmark	paramnet	surrogate
paramnet_benchmark	ParamNet*OnTimeBenchmark	paramnet	surrogate
paramnet_benchmark(x)	ParamNetReduced*OnTimeBenchmark	paramnet	surrogate
svm_benchmark	SurrogateSVMBenchmark	surrogate_svm	surrogate

Family	Name	Type	Note
od_ae	ODAutoencoder	raw	Works on 15 datasets (`annthyroid`, `arrhythmia`, `breastw`, `cardio`, `ionosphere`, `mammography`, `musk`, `optdigits`, `pendigits`, `pima`, `satellite`, `satimage-2`, `thyroid`, `vowels`, `wbc`)
od_kde	ODKernelDensityEstimation	raw	Works on 15 datasets (see above). No fidelities available
od_ocsvm	ODOneClassSupportVectorMachine	raw	Works on 15 datasets (see above). No fidelities available

Note:

(x) benchmarks that have been used in the HPOBench paper, other benchmarks are considered experimental
(+) Tabular benchmarks might take a few minutes to load.

Note: All containers are uploaded here