Enterprise feature. This feature is only available to Roboflow Enterprise users. Contact the sales team to learn more about Roboflow Enterprise.
You can run multiple models in parallel with parallel processing, a version of Roboflow Inference that processes inference requests asynchronously.
Inference Parallel supports the same features as Roboflow Inference, except that it does not support core models (CLIP and SAM).
With Inference Parallel, preprocessing, auto batching, inference, and post-processing all run in separate threads to increase server FPS throughput. Separate requests to the same model are batched on the fly as allowed by $MAX_BATCH_SIZE, and response handling then occurs independently. Images are passed via Python's SharedMemory module to maximize throughput.
These changes result in as much as a 76% speedup on one measured workload.
How to run Inference with parallel processing
First, build the parallel server:
./inference/enterprise/parallel/build.shThen run the server:
./inference/enterprise/parallel/run.shA message appears in the terminal indicating that the server is running and ready for use.
A container is published on Docker Hub:
docker pull roboflow/roboflow-inference-server-gpu-parallel:latestIf you are pulling a pinned tag, change the $TAG variable in run.sh to match.
Benchmarks
We evaluated the performance of Inference Parallel on a variety of models from Roboflow Universe, comparing it against Inference Server 0.9.5.rc on the same hardware: a computer with eight cores and one GPU. Instance segmentation metrics were calculated with "mask_decode_mode": "fast" in the request body. Requests were posted concurrently with a parallelism of 1000.
| Workspace | Model | Model type | Split | 0.9.5.rc FPS | 0.9.5.parallel FPS |
|---|---|---|---|---|---|
| senior-design-project-j9gpp | nbafootage/3 | object-detection | train | 30.2 fps | 44.03 fps |
| niklas-bommersbach-jyjff | dart-scorer/8 | object-detection | train | 26.6 fps | 47.0 fps |
| geonu | water-08xpr/1 | instance-segmentation | valid | 4.7 fps | 6.1 fps |
| university-of-bradford | detecting-drusen_1/2 | instance-segmentation | train | 6.2 fps | 7.2 fps |
| fy-project-y9ecd | cataract-detection-viwsu/2 | classification | train | 48.5 fps | 65.4 fps |
| hesunyu | playing-cards-ir0wr/1 | classification | train | 44.6 fps | 57.7 fps |
Inference with parallel processing enabled achieved higher FPS on every test. On some models the FPS increase was greater than 10 FPS.