Uncertainty quantification benchmarks are identical to models regarding implementation of both server and client. They only differ in that they fully define a UQ problem rather than just a forward model.
The existing benchmarks provide Bayesian inverse problems by defining the Bayesian posterior density evaluation. They are based on forward models that are themselves available as stand-alone containers.
When defining your own benchmarks, it is recommended to separate forward model and UQ problem in the implementation. The result is a general-purpose forward model and a benchmark only adding minimal code overhead. This can easily be achieved by having the benchmark server (defining the UQ problem) in turn connect to the forward model as a client. The entire benchmark (forward model and UQ problem) can then easily be provided in a single container by building the benchmark container on top of the forward model base image. Communication between benchmark and forward model then happens inside the container, while the benchmark itself is exposed to the outside.
Refer to benchmarks defined in this repository for working examples.
A simple Gaussian posterior to sample from is available from the UM-Bridge repository. Its main purpose is demonstrating how to define UQ benchmarks.
docker run -p 4243:4243 linusseelinger/benchmark-testbenchmark:latest
We welcome any new benchmarks added to the collection. To add a new benchmark you will need a Dockerfile defining the setup of your software and a README.md. Place these into a folder within benchmarks. You can access all existing benchmarks definitions in this repository for reference.
The Dockerfile defining how your model or benchmark Docker image should be built is fairly simple. You can start from any base image (e.g. Debian Linux), proceed with instructions for installing possible dependencies, copy in your code and (if needed) compile it.
The only assumption on the resulting image is that, when a container is spun up, your application supporting the UM-Bridge interface is run (this can be achieved with an appropriate RUN command).
The README is automatically incorporated in this documentation. It must follow the structure prescribed by the template README located in the docs folder. Ensure that all appropriate sections are filled.
Some components of the README can be auto-generated by using the
generate_benchmark_readme_info.py in the docs folder. This script connects to your running model server and extracts basic information from the UM-Bridge protocol.
To include images in the readme either link to an external website or place the image in your benchmark’s directory and provide the full path to the raw image.