Elastic Reports Faster Jina Model Inference on NVIDIA RTX Pro 6000 GPUs

Elastic says moving selected Jina API models to NVIDIA RTX Pro 6000 Blackwell Server Edition GPUs has cut latency across embedding, reranking, and document OCR. The strongest reported gain is in slow embedding requests: production measurements show 99th-percentile latency falling from 5.86 seconds on NVIDIA L4 GPUs to 0.73 seconds on the newer hardware. For teams building search and agent workflows, the results point to a practical benefit: faster model responses, including when large and small requests share capacity.

The migration is underway, rather than complete. Elastic says some Jina API models already run on the new GPUs, with further models in the Jina API and Elastic Inference Service (EIS) planned for migration. Customers using a model after it moves receive the performance change without needing to alter their integration. EIS provides GPU-accelerated endpoints integrated with Elasticsearch; the Jina API offers direct access to Jina models.

Why mixed traffic matters

Model size alone doesn't tell a service operator how quickly requests will finish. A shared endpoint may receive a large reranking job with many candidate documents at the same time as a small query that needs a quick result. If the larger request keeps the GPU busy, the smaller one can wait in a queue. That delay can matter to an application even when the model itself is compact.

Elastic says its RTX Pro 6000 evaluation focused on this mix of workloads. In a controlled test, small reranking requests alongside large jobs took 66 milliseconds on the RTX Pro 6000, compared with 708 milliseconds on the L4 setup, a reported 10.7-fold improvement. That result is notable because it measures the waiting experience of smaller requests under shared load, rather than only the completion time of one large job.

A separate controlled test measured a large reranking job completing in 160 milliseconds on one RTX Pro 6000 GPU, versus 702 milliseconds across three L4 GPUs. The test used the same software. These results suggest the newer GPU can handle both the large job and concurrent smaller requests more efficiently in the tested setup. They don't, on their own, show how every deployment will perform: request sizes, traffic patterns, batching, and service configuration all affect latency.

What the production measurements show

Elastic compared production traffic before and after the migration, and also ran controlled tests for workloads that production traffic couldn't isolate. Percentiles help distinguish typical response times from slower outliers: at the 99th percentile, 99 out of 100 requests finished within the reported time. For interactive search or agent systems, reducing these slower responses can make the service feel more consistent.

The reranking production comparison is especially useful to read alongside its traffic detail: Elastic reports the lower latency while average request volume was about 40% higher. Still, the results come from Elastic's own service and tests. They don't establish a universal speedup for customer workloads or provide a cost comparison, so teams evaluating a move will need to consider their own traffic and GPU capacity needs.

Where the models fit in a search pipeline

Jina models cover several steps between raw content and a useful answer. Reader and OCR models turn documents and websites into structured text, retaining elements such as headings, tables, images, and formulas. Elastic says jina-ocr-v1 handles more than 100 languages and scripts, and can produce text with markdown and HTML structure as well as LaTeX for formulas. The company describes it as the leading model under 600 million active parameters on the olmOCR-bench benchmark.

Embedding models map content into a form that search systems can compare. Jina's omni embedding models support text, images, audio, and video across more than 100 languages, placing those content types in a shared embedding space. Elastic says jina-embeddings-v5-omni-small performs best among open-weight omni models under 2 billion parameters on MTEB. A reranker then takes the initial search results and scores them against the original query to move stronger matches higher. Elastic says its 600-million-parameter jina-reranker-v3.5 outperforms the 4-billion-parameter Qwen3-Reranker-4B on the STaRK structured-data benchmark.

That pipeline explains why response time can accumulate across multiple model calls. Faster extraction can speed up ingestion; quicker embeddings and reranking can reduce the wait between a query and its best results. For agent-based applications, where a system may make several calls before responding, improvements at each step can matter. The source doesn't quantify the combined end-to-end effect, so the component results shouldn't be read as a measured improvement for a full agent workflow.

What to watch as migration continues

Elastic's reported gains make the RTX Pro 6000 move relevant to teams using hosted Jina models, especially those affected by slow requests under mixed traffic. The operational attraction is that migrated Jina API models require no customer-side integration change. For EIS users, the key question is when each endpoint they rely on moves and how its performance compares on their own request mix.

The next useful evidence will be results across more migrated models, along with details that let customers assess capacity and cost tradeoffs. For now, Elastic's measurements show substantial improvements in selected workloads, with the largest reported benefit in embedding latency and small reranking requests running beside large jobs. Teams can review the EIS documentation and try the Jina API.