GPT-3.5 Turbo 16K: Complete Benchmark Performance Guide

openai 3.5-turbo-16k 2023-06-13 Proprietary

In-depth analysis of GPT-3.5 Turbo 16K performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendoropenai
Version3.5-turbo-16k
Release Date2023-06-13
Context Window16385 tokens
Input Modalitiestext
Output Modalitiestext
LicenseProprietary
Documentationhttps://platform.openai.com/docs/models

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU64.2%2023-06-135-shotview
HUMANEVAL36.8pass@12023-06-13—view
GSM8K52.8%2023-06-130-shot CoTview
MATH30.4%2023-06-130-shot CoTview
BBH50.2%2023-06-133-shot CoTview
GPQA21.6%2023-06-130-shotview
IFEVAL48.5%2023-06-13prompt_strictview
ARC83.1%2023-06-13challengeview
MUSR29.1%2023-06-130-shotview
WINOGRANDE74%2023-06-130-shotview

Pricing

TierPriceCurrency
Input$3 / MtokUSD
Output$4 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://openai.com/api/pricing/ · as of 2023-06-13

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✓
  • GDPR: ✓
  • ISO 27001: ✓

GPT-3.5 Turbo 16K

Model Overview

OpenAI GPT-3.5 Turbo 16K 长上下文版本, 支持 16385 token, 适合中等长度文档处理。

Core Specifications

VendorVersionRelease DateContext WindowInput ModalitiesOutput ModalitiesLicense
Openai3.5-turbo-16k2023-06-1316KtexttextProprietary

Benchmark Performance

BenchmarkScoreUnitNotes
MMLU (Massive Multitask Language Understanding)64.2%5-shot
HumanEval36.8pass@1—
GSM8K (Grade School Math 8K)52.8%0-shot CoT
MATH30.4%0-shot CoT
BBH (BIG-Bench Hard)50.2%3-shot CoT
GPQA21.6%0-shot
IFEval48.5%prompt_strict
ARC83.1%challenge
MUSR29.1%0-shot
WinoGrande74.0%0-shot

Pricing

InputOutputCache ReadCache Write
————

per million tokens

Strengths

  • Context window 16K.

Weaknesses

  • HumanEval 36.8, coding weak.
  • Proprietary, not self-hostable.
  • Context window 16K is limited.

Use Cases

  • Long document summarization

References