Deploy to Cloud

You can deploy Roboflow Inference containers to virtual machines in the cloud. These VMs are configured to run CPU or GPU-based Inference Servers under the hood, so you don't have to deal with OS, GPU drivers, or Docker installations. The Inference CLI currently supports deploying the Roboflow Inference container images into a virtual machine running on Google Cloud (GCP) or Amazon Cloud (AWS).

The Inference CLI assumes the corresponding cloud CLI is configured for the project you want to deploy the virtual machine into. Read the instructions for setting up the Google gcloud CLI or the Amazon AWS CLI.

Roboflow Inference cloud deploy is powered by the SkyPilot project.

pip install "inference-cli[cloud-deploy]"

inference cloud --help

inference cloud deploy --help

inference cloud deploy

Deploy GPU or CPU inference to AWS or GCP:

# Deploy the Roboflow Inference GPU container into a GPU-enabled VM in AWS
inference cloud deploy --provider aws --compute-type gpu

# Deploy the Roboflow Inference CPU container into a CPU-only VM in GCP
inference cloud deploy --provider gcp --compute-type cpu

Note the "cluster name" printed after the deployment completes. This handle is used in many subsequent commands. The deploy command also prints helpful debug and cost information about your VM.

Deploying Inference into a cloud VM also prints out an endpoint of the form http://1.2.3.4:9001; you can now run inference against this endpoint.

Note that port 9001 is automatically opened. Check with your security admin if this is acceptable for your cloud or project.

inference cloud status

To check the status of your deployment, run:

inference cloud status

Stop and start deployments

You can start and stop your deployment using:

inference cloud start <deployment_handle>

and

# Stop the VM, you only pay for disk storage while the VM is stopped
inference cloud stop <deployment_handle>

inference cloud undeploy

To delete (undeploy) your deployment, run:

inference cloud undeploy <deployment_handle>

SSH into the cloud deployment

You can SSH into your cloud deployment with the following command:

ssh <deployment_handle>

The required SSH key is automatically added to your ~/.ssh/config; you don't need to configure this manually.

Cloud deploy customization

Roboflow Inference cloud deploy creates VMs based on internally tested templates.

For advanced use cases, you can customize the template by passing your own sky YAML template on the command line:

inference cloud deploy --custom /path/to/sky-template.yaml

You can download the standard template stored in the Roboflow CLI and then modify it for your needs:

# This command will print out the standard gcp/cpu sky template.
inference cloud deploy --dry-run --provider gcp --compute-type cpu

Then you can deploy a custom template based on your changes.

You can also use the sky CLI to control your deployments and access more advanced functionality.

Roboflow Inference deploy currently supports AWS and GCP. Please open an issue on the Inference GitHub repository if you would like to see other cloud providers supported.