Perception Encoder

Perception Encoder is Meta's vision-language embedding model. It maps images and text into a shared embedding space for similarity search, zero-shot classification, and retrieval.

We support three Perception Encoder endpoints:

• /perception_encoder/embed_image - embed an image
• /perception_encoder/embed_text - embed a string
• /perception_encoder/compare - compute similarity between an image and a list of text prompts

Perception Encoder API

export ROBOFLOW_API_KEY="your-key-here"

pip install -U requests opencv-python supervision

import base64
import os
import cv2
import requests
import supervision as sv

URL = "https://your-deployment.roboflow.cloud"

image = sv.load_image_from_url("https://media.roboflow.com/notebooks/examples/dog.jpeg")

_, buffer = cv2.imencode(".jpg", image)
image_base64 = base64.b64encode(buffer).decode("utf-8")

response = requests.post(
    f"{URL}/perception_encoder/embed_image",
    json={
        "api_key": os.environ["ROBOFLOW_API_KEY"],
        "image": {"type": "base64", "value": image_base64},
    },
)
result = response.json()
embedding = result["embeddings"][0]
print(f"Embedding length: {len(embedding)}")
print(f"First values: {embedding[:5]}")

The code above prints the embedding shape to the terminal:

Embedding length: 1024
First values: [0.0545, -0.0338, -0.0355, -0.0062, 0.0154]

Perception Encoder inference speed

Latency measured with Roboflow Inference on 1x NVIDIA L4, batch size 1, mean after warmup.

Measured with embed_image on the PE-Core-L14-336 checkpoint (image embedding only).

Run Perception Encoder with self-hosted Inference

Perception Encoder can be loaded directly with the inference package, which is the fastest path when embedding many images or video frames locally.

pip install "inference[transformers]"

from inference.core.utils.postprocess import cosine_similarity
from inference.models import PerceptionEncoder

pe = PerceptionEncoder(model_id="perception_encoder/PE-Core-B16-224")

image_embedding = pe.embed_image("https://media.roboflow.com/inference/people-walking.jpg")
text_embedding = pe.embed_text("a crowd of people walking")

print(cosine_similarity(image_embedding[0], text_embedding[0]))

Available checkpoints

model_id selects the backbone:

• perception_encoder/PE-Core-B16-224
• perception_encoder/PE-Core-L14-336
• perception_encoder/PE-Core-G14-448

Only the CLIP-style interface is supported; the language-aligned and spatially-aligned Perception Encoder variants are not available yet.

Use in Workflows

Perception Encoder is available in Workflows through the Perception Encoder Embedding Model block, which generates image or text embeddings without writing code.