ado-vllm-performance

This repository contains the vLLM ado actuator for benchmarking LLM inference performance with vLLM. (For more about Actuators, what they represent, how to create them etc., see the ado docs).

The actuator implements a set of functionalities to deploy and run serving benchmarks for different LLMs for vLLM. This actuator deploys vLLM on to an OpenShift cluster and runs benchmarks using either vLLM's built-in benchmarking tool (vllm bench serve) or GuideLLM. The actuator is named vllm_performance and provides thirteen experiments spanning standard LLM benchmarking, agentic tool-calling, and geospatial model benchmarking.

Getting Started

This guide has two parts:

• Getting Started
  • Installing and configuring the vLLM actuator
    • Installation
    • Configuring the actuator
  • A Simple Benchmarking Exercise
    • Creating a Discovery Space to describe the vLLM configurations to test
      • Activating the local context
      • Defining a Discovery Space of vLLM configurations
      • Querying the Discovery Space
    • Exploring the vLLM workload configuration space
• Exploring Further
  • vLLM testing approach
  • Geospatial Model Experiments
  • The Actuator Package: Key Files
    • Customising Actuator Configurations
    • Customising Experiment Protocol
    • Notes on the Random walk operation
  • A few ideas for further exploration

After running the exercise, please feel free to explore further and try a larger experiment.

[!NOTE]

These prerequisites must be fulfilled before you start with this actuator

1. Access to an OpenShift cluster with at least 1 node with 1 available NVIDIA GPU. You will need access to a namespace with permissions for GPU-based deployments
2. You will need to have downloaded and installed ado according to this guide.

Installing and configuring the vLLM actuator

Installation

Ensure the virtual environment you installed ado into is active. Then, run:

pip install ado-vllm-performance

This will automatically install both vLLM and GuideLLM benchmarking tools, enabling all experiments:

• vllm-bench-deployment and vllm-bench-endpoint (vLLM benchmarks)
• guidellm-bench-deployment and guidellm-bench-endpoint (GuideLLM benchmarks)
• geospatial-vllm-bench-deployment and geospatial-vllm-bench-endpoint (Geospatial model benchmarks with vLLM)
• geospatial-guidellm-bench-deployment and geospatial-guidellm-bench-endpoint (Geospatial model benchmarks with GuideLLM)
• geospatial-vllm-bench-deployment-custom-dataset and geospatial-vllm-bench-endpoint-custom-dataset (Geospatial with custom datasets using vLLM)
• geospatial-guidellm-bench-deployment-custom-dataset and geospatial-guidellm-bench-endpoint-custom-dataset (Geospatial with custom datasets using GuideLLM)

For development from source:

pip install -e plugins/actuators/vllm_performance

from the root of the ado source repository. You can clone the repository with:

git clone https://github.com/IBM/ado.git

Confirm that the actuator is installed:

ado get actuators --details

You should see vllm_performance in the list. To confirm all experiments are registered, run:

ado get experiments --details

The vllm_performance experiments you should see are:

• vllm-bench-deployment
• vllm-bench-endpoint
• guidellm-bench-deployment
• guidellm-bench-endpoint
• test-agentic-tool-calling
• geospatial-vllm-bench-deployment
• geospatial-vllm-bench-endpoint
• geospatial-guidellm-bench-deployment
• geospatial-guidellm-bench-endpoint
• geospatial-vllm-bench-deployment-custom-dataset
• geospatial-vllm-bench-endpoint-custom-dataset
• geospatial-guidellm-bench-deployment-custom-dataset
• geospatial-guidellm-bench-endpoint-custom-dataset

You can understand the constitutive properties required for the experiment and the target and observed properties measured by an experiment by running:

ado describe experiment vllm-bench-deployment

The experiment protocols for the vLLM actuator are defined in ado_actuators/vllm_performance/experiments/performance_testing.yaml and ado_actuators/vllm_performance/experiments/performance_testing_geospatial.yaml. You will need to update these if you want to modify the values that can be accepted as valid for the input properties.

Configuring the actuator

Before using the vLLM actuator to execute experiments, you must configure its parameters. First, get the template for the configuration:

ado template actuatorconfiguration --actuator-identifier vllm_performance \
                                   -o actuatorconfiguration.yaml

This will create the actuatorconfiguration.yaml file, which will look like:

actuatorIdentifier: vllm_performance
metadata:
  description: null
  labels: null
  name: null
parameters:
  benchmark_retries: 3
  deployment_template: null
  developer_mode: false
  free_environment_ttl: 300
  gc_force_delete: false
  gc_force_delete_threshold: 3
  hf_token: ''
  image_pull_secret_name: ''
  in_cluster: false
  max_environments: 1
  namespace: null
  node_selector: {}
  otlp_traces_endpoint: null
  pvc_name: null
  pvc_template: null
  retries_timeout: 5
  service_template: null
  verify_ssl: false

The three key parameters we have to set here are hf_token, namespace, and node_selector.

• hf_token: Access token from HuggingFace.

• namespace: The namespace you have access to in your OpenShift cluster.

• node_selector: JSON dictionary representing a Kubernetes selector for a node with available GPUs. Make sure it is formatted correctly, for example:

  node_selector: {"kubernetes.io/hostname":"cpu16"}
  

We will discuss the other parameters later. Once you have put in the parameters, create the actuator configuration with:

ado create actuatorconfiguration -f vllm_performance_actuatorconfiguration.yaml

Note: You can have multiple configurations for an actuator.

A Simple Benchmarking Exercise

To get started, we have provided an exercise to run a benchmarking experiment for a single vLLM deployment configuration. The instructions for this exercise assume you are running ado from a machine outside of the target Kubernetes/OpenShift cluster.

Creating a Discovery Space to describe the vLLM configurations to test

[!NOTE]

Since this is an example exercise, we will use the local context and the default sample store.

Activating the local context

To ensure the local context is active, run:

ado context local

Defining a Discovery Space of vLLM configurations

ado uses the concept of Discovery Spaces to describe what to test (in this case vLLM workload configurations) and how to test them (the vLLM benchmark(s) to run).

The set of configurations to test is defined by the entity space, and the set of experiments to perform by the measurement space.

An example discoveryspace for vLLM inference benchmarking can be found in yamls/discoveryspace_override_defaults_small.yaml. This defines a simple discovery space with four entities.

Our sample space will benchmark vLLM serving the LLM specified by model_name, on a node (determined through node_selector) with a specific GPU (NVIDIA-A100-80GB-PCIe) specified in gpu_type.

[!NOTE]

Ensure that the GPU specified in gpu_type is present on the node. To find out the gpu model of your selected node, try the following command:

oc describe node <node name> | grep "nvidia.com/gpu.product"

If this returns a different GPU model, then you must update the experiment protocol.

Create the discoveryspace:

ado create space -f yamls/discoveryspace_override_defaults_small.yaml \
                 --use-default-sample-store

Querying the Discovery Space

Before we run any experiment, we can see that the discoveryspace is empty:

ado show measurements space --use-latest

Will output:

Nothing was returned for entity type matching and property format observed in space space-c81773-default.

To see all the entities (parameter combinations) that are waiting to be measured, try executing:

ado show measurements space --include missing --use-latest

The output will look like:

┌───────┬────────────────┬────────────────┬────────┬────────┬───────┬─────────────┬──────────────┬────────────────┬────────────────┬─────────────┬─────────────────┬─────────────┬────────┬────────────────┐
│ INDEX │ model          │ image          │ n_cpus │ memory │ dtype │ num_prompts │ request_rate │ max_concurren… │ gpu_memory_ut… │ cpu_offload │ max_batch_toke… │ max_num_seq │ n_gpus │ gpu_type       │
├───────┼────────────────┼────────────────┼────────┼────────┼───────┼─────────────┼──────────────┼────────────────┼────────────────┼─────────────┼─────────────────┼─────────────┼────────┼────────────────┤
│ 0     │ meta-llama/Ll… │ quay.io/datap… │ 8      │ 128Gi  │ auto  │ 100         │ 100          │ 100            │ 0.9            │ 0           │ 16384           │ 256         │ 1      │ NVIDIA-A100-8… │
│ 1     │ meta-llama/Ll… │ quay.io/datap… │ 8      │ 128Gi  │ auto  │ 100         │ 100          │ 100            │ 0.9            │ 0           │ 32768           │ 256         │ 1      │ NVIDIA-A100-8… │
│ 2     │ meta-llama/Ll… │ quay.io/datap… │ 8      │ 128Gi  │ auto  │ 250         │ 100          │ 100            │ 0.9            │ 0           │ 16384           │ 256         │ 1      │ NVIDIA-A100-8… │
│ 3     │ meta-llama/Ll… │ quay.io/datap… │ 8      │ 128Gi  │ auto  │ 250         │ 100          │ 100            │ 0.9            │ 0           │ 32768           │ 256         │ 1      │ NVIDIA-A100-8… │
└───────┴────────────────┴────────────────┴────────┴────────┴───────┴─────────────┴──────────────┴────────────────┴────────────────┴─────────────┴─────────────────┴─────────────┴────────┴────────────────┘

Which is the entity we want to measure.

Exploring the vLLM workload configuration space

First, log in to your OpenShift cluster and select your assigned namespace

oc login <your OpenShift API endpoint>
oc project <your assigned namespace>

Next, we'll set up the operation to measure our entity defined above.

In ado parlance, measurements are executed through operations which represent the executions of experiments on entities.

An example of an operation can be found in yamls/random_walk_operation.yaml. You can run the operation using the actuator configuration and space that we have created earlier with:

ado create operation -f yamls/random_walk_operation.yaml \
                     --use-latest space --use-latest actuatorconfiguration

ado will initialise a local Ray cluster and starts the measurement at the point where these lines appear:

...
=========== Starting Discovery Operation ===========

(RandomWalk pid=79429) 'all' specified for number of entities to sample. This is 4 entities - the size of the entity space
...

The actuator uses the entity to create a vLLM deployment, followed by execution of the benchmark script. This process will take some time as it involves downloading the container image from Quay and the model from HuggingFace, both of which are network-intensive. You can monitor if the deployment is ready by executing the following in another shell:

oc get deployments --watch

The experiment is successfully completed if the ado output is similar to the following:

(RandomWalk pid=46852) Continuous Batching: EXPERIMENT COMPLETION. Received finished notification for experiment in measurement request in group 0: request-4332aa-experiment-performance-testing-entities-model.ibm-granite/granite-3.3-8b-instruct-image.quay.io/dataprep1/data-prep-kit/vllm_image:0.1-n_cpus.8-memory.128Gi-dtype.auto-num_prompts.500-request_rate.-1-max_concurrency.-1-gpu_memory_utilization.0.9-cpu_offload.0-max_batch_tokens.16384-max_num_seq.256-n_gpus.1-gpu_type.NVIDIA-A100-80GB-PCIe (explicit_grid_sample_generator)-requester-randomwalk-0.9.7.dev10+b7a010dd.dirty-42ad60-time-2025-08-11 15:53:54.137571+01:00
(RandomWalk pid=46852) Continuous batching: GET EXPERIMENT. No new experiments in queue. Requests made: 1. Experiments Completed: 1

If the output contains EXPERIMENT FAILURE, then something has gone wrong.

Verify that the entity has been measured by running:

ado show measurements space --use-latest -o csv --output-file entities.csv

The csv file will have one line representing the entity featuring values for all its measured properties (performance-testing-output_throughput,performance-testing-total_token_throughput,performance-testing-mean_ttft_ms, etc.)

Congratulations! You have successfully executed the vLLM benchmark on a vLLM workload configuration using ado!

Exploring Further

vLLM testing approach

vLLM testing implementation is based on this guide which is using benchmark_serving.py to implement the actual benchmarking. The benchmarking is done using HTTP requests using vLLM OpenAI API server.

To use this approach it is necessary to:

• Create a docker image: Existing docker images for VLLM project are not directly suitable for this purpose, as they are hard to use on Openshift clusters and not directly extensible. We have provided a Docker image to get started but if you want to customize it for your installation, then you will need to rebuild it. We provide a slightly different build, described here
• Create automation for vLLM deployment for running experiments. A simple implementation of such an automation is presented here
• Create a vLLM performance test. Here we are directly reusing performance test provided by the vLLM project. The required code is here

This figure shows the outline of the components and the parameters available for configuring each of them

The test results in the figure are the measurements recorded for the entity. The deployment parameters form the configuration space. Test parameters are partially inferred from the configuration space and partially from the context (Kubernetes endpoints, etc.)

Geospatial Model Experiments

The vLLM actuator includes support for benchmarking geospatial models, specifically IBM-NASA Prithvi models for Earth observation tasks.

Geospatial experiments are available for both endpoint and deployment testing, with support for pre-packaged datasets (india_url_in_b64_out, valencia_url_in_b64_out) and custom datasets. For detailed information on geospatial experiments, dataset formats, and usage examples, see the vLLM Performance Geospatial documentation.

The Actuator Package: Key Files

The actuator package is under ado_actuators/vllm_performance. Note all actuator packages should be placed under a directory called ado_actuators as this is the name of package that contains all ado plugins.

The key files are:

• Entry point registration in pyproject.toml

  • Registers the actuator class with ado using Python entry points.

  • Example:

    [project.entry-points."ado.actuators"]
    vllm_performance = "ado_actuators.vllm_performance.actuator:VLLMPerformanceTest"
    

• actuator.py

  • Implementation of the actuator logic.
  • The class must inherit from ActuatorBase and define an identifier attribute.

• experiments.yaml

  • This file contains the definitions of the experiments the actuator defines as YAML

• experiment_executor.py (OPTIONAL)

  • This file contains the code that
    • determines the values for the experiment parameters from the passed Entity and Experiment
    • execute the experiment and get measured property values
    • sends the measured property values back to the orchestrator

Customising Actuator Configurations

The actuator is configured using VLLMPerformanceTestParameters class

You can customise deployment_template, service_template and pvc_template for your OpenShift/K8s cluster. Refer to the default yamls for the templates referred to in Configuring the actuator and modify them appropriately

If you create a custom Docker image and upload it to a repository, please do not forget to create a corresponding Image pull secret in your assigned namespace. You must also update the value of the image_pull_secret_name parameter of the actuator configuration.

Environment lifecycle and garbage collection parameters

By default, vLLM deployments that finish an experiment are kept in a free pool for up to 5 minutes so that subsequent experiments sharing the same deployment configuration can reuse them rather than paying the full start-up cost.

Three parameters control this behaviour:

• free_environment_ttl (default: 300): Time-to-live in seconds for idle (free) Kubernetes environments. When an environment has been idle for at least this many seconds the garbage-collector deletes it. Set to 0 to delete every environment immediately upon release (disables the free pool).

• gc_force_delete (default: false): When true, the garbage-collector issues a force-delete (clears Kubernetes finalizers and uses grace_period_seconds=0) for any deployment that is still present in Kubernetes after gc_force_delete_threshold GC cycles following a normal delete. When false (the default) the GC logs a warning each cycle and keeps waiting for the deployment to disappear naturally.

• gc_force_delete_threshold (default: 3): Number of GC cycles a deployment may remain present in Kubernetes after a delete has been issued before a force-delete is triggered. Only used when gc_force_delete is true.

Customising Experiment Protocol

The values for the parameters in the entity space must be a subset of the acceptable values defined for the experiment (the experiment protocol). Therefore, depending on your environment and use case, you may need to update the set of values to expand the configuration space being studied.

For example, you may want to benchmark a different LLM or you may want to change the GPU type to the one installed in your cluster. In the former case, you will add values to model and in the latter case, you will have to modify the domain of the gpu_type parameter to avoid validation errors.

To do this, open the experiment definition YAML file in a text editor, and add your GPU model to the list of values of gpu_type.

Then, reinstall this actuator by running:

pip install .

After that, you can use the new value of gpu_type in your experiments. For example, in the sample space definition file, the location to update will be:

- identifier: "gpu_type"
  propertyDomain:
    values: ["NVIDIA-A100-80GB-PCIe"]

Notes on the Random walk operation

VLLM testing is using external environment (deployment + service) to run tests. Creating such an environment is resource-intensive. To speed up experiments execution it is recommended to use group samplers for running VLLM testing. This allows to create an environment once and use it for all experiments that can be used for it. In this case the group definition looks as follows:

grouping:
  - model
  - image
  - n_gpus
  - gpu_type
  - n_cpus
  - memory
  - max_batch_tokens
  - gpu_memory_utilization
  - dtype
  - cpu_offload
  - max_num_seq

For the complete example of configuring random walk operation for the group samplers, look here

A few ideas for further exploration

Try:

• Testing throughput for different sequence length for multiple models using this actuator (See discoveryspace_override_defaults_small.yaml for an example with multiple values for max_batch_tokens)