Claude 3.5 Sonnet: Complete Benchmark Performance Guide

anthropic 3.5-sonnet 2024-06-20 Proprietary

In-depth analysis of Claude 3.5 Sonnet's performance across MMLU, HumanEval, GSM8K, MATH, and BBH benchmarks with pricing and use cases.

Key Specifications

Vendoranthropic
Version3.5-sonnet
Release Date2024-06-20
Context Window200000 tokens
Input Modalitiestext, image
Output Modalitiestext
LicenseProprietary
Documentationhttps://docs.anthropic.com/claude/docs

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU88.7%2024-06-205-shotview
HUMANEVAL92pass@12024-06-20—view
GSM8K96.4%2024-06-200-shot CoTview
MATH71.1%2024-06-200-shot CoTview
BBH84.5%2024-06-203-shot CoTview

Pricing

TierPriceCurrency
Input$3 / MtokUSD
Output$15 / MtokUSD
Cache Read$0.3 / MtokUSD
Cache Write$3.75 / MtokUSD

Source: https://www.anthropic.com/pricing · as of 2024-08-01

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✗

Claude 3.5 Sonnet

Model Overview

Claude 3.5 Sonnet is Anthropic’s mid-2024 flagship, released on June 20, 2024. It raises the bar on coding, vision reasoning, and long-document understanding while maintaining a 200K context window. Despite being positioned as the “mid-tier” model in the Claude 3.5 family, Sonnet outperforms the previous Claude 3 Opus on nearly every academic benchmark and operates at twice the speed. Anthropic emphasizes Constitutional AI alignment, predictable behavior, and a strong preference for enterprise use cases involving complex tool use and agentic workflows. Claude 3.5 Sonnet is also the first model to ship with the Artifacts feature in the Claude.ai web UI, allowing interactive code and content generation alongside the conversation thread.

Key Specifications

AttributeValue
VendorAnthropic
Version3.5-sonnet
Release Date2024-06-20
Context Window200,000 tokens
Input Modalitiestext, image
Output Modalitiestext
LicenseProprietary
Documentationhttps://docs.anthropic.com/claude/docs

Benchmark Performance

BenchmarkScoreUnitNotes
MMLU88.7%5-shot
HumanEval92.0pass@1—
GSM8K96.4%0-shot CoT
MATH71.1%0-shot CoT
BBH84.5%3-shot CoT

Pricing

TierPrice (per 1M tokens)
Input$3.00
Output$15.00
Cache Read$0.30
Cache Write$3.75

Pricing source: https://www.anthropic.com/pricing (as of 2024-08-01).

Strengths

  • Industry-leading HumanEval score (92.0 pass@1), making it the top choice for code generation.
  • Generous 200K context window for long-document analysis and codebase reasoning.
  • Vision input support for charts, screenshots, and document understanding.
  • Strong tool-use and agentic workflow reliability.

Weaknesses

  • No audio modality support (text and image input only).
  • Pricing is 20-50% higher than GPT-4o for input tokens.
  • No self-host option; only available via Anthropic API or cloud partners.

Use Cases

  • Production code generation and refactoring.
  • Long-document summarization and legal/financial analysis.
  • Multimodal reasoning over mixed text+image inputs.
  • Agentic automation involving tool calls and structured outputs.

References