High Throughput Inference: AI Model Selection Guide

High Throughput Inference

Use Case Overview

This page focuses on the High Throughput Inference use case, covering core requirements, recommended models, and evaluation criteria.

Selection Criteria

• Open-weight license

Evaluation Criteria

• Open-weight license

Best Practices

• Provide system prompt for context
• Monitor token usage and cost

Recommended Models

See the recommended models list for this use case (path: /en/use-cases/high-throughput/).