Claude 3 Sonnet (2024-02-29): Complete Benchmark Performance Guide

anthropic 3-sonnet-20240229 2024-02-29 Proprietary

In-depth analysis of Claude 3 Sonnet (2024-02-29) performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendoranthropic
Version3-sonnet-20240229
Release Date2024-02-29
Context Window200000 tokens
Input Modalitiestext, image
Output Modalitiestext
LicenseProprietary
Documentationhttps://docs.anthropic.com/claude/docs

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU84%2024-02-295-shotview
HUMANEVAL80.1pass@12024-02-29—view
GSM8K82.6%2024-02-290-shot CoTview
MATH46.4%2024-02-290-shot CoTview
BBH76.2%2024-02-293-shot CoTview
GPQA41.7%2024-02-290-shotview
IFEVAL75.5%2024-02-29prompt_strictview
ARC94.8%2024-02-29challengeview
MUSR63.5%2024-02-290-shotview
WINOGRANDE80.3%2024-02-290-shotview

Pricing

TierPriceCurrency
Input$3 / MtokUSD
Output$15 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://www.anthropic.com/pricing · as of 2024-02-29

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✗

Claude 3 Sonnet (2024-02-29)

Model Overview

Anthropic Claude 3 Sonnet 2024-02-29 快照版本, 200K 上下文, 速度与智能平衡型。

Core Specifications

VendorVersionRelease DateContext WindowInput ModalitiesOutput ModalitiesLicense
Anthropic3-sonnet-202402292024-02-29200Ktext, imagetextProprietary

Benchmark Performance

BenchmarkScoreUnitNotes
MMLU (Massive Multitask Language Understanding)84.0%5-shot
HumanEval80.1pass@1—
GSM8K (Grade School Math 8K)82.6%0-shot CoT
MATH46.4%0-shot CoT
BBH (BIG-Bench Hard)76.2%3-shot CoT
GPQA41.7%0-shot
IFEval75.5%prompt_strict
ARC94.8%challenge
MUSR63.5%0-shot
WinoGrande80.3%0-shot

Pricing

InputOutputCache ReadCache Write
————

per million tokens

Strengths

  • MMLU score 84.0, strong knowledge reasoning.
  • HumanEval 80.1, excellent code generation.
  • Input Modalities: text, image, audio.
  • Context window 200K.

Weaknesses

  • Proprietary, not self-hostable.

Use Cases

  • Code generation and debugging
  • Long document summarization
  • Vision and image understanding

References