Ministral 8B: Complete Benchmark Performance Guide

mistral ministral-8b 2024-10-16 Mistral Research License

In-depth analysis of Ministral 8B performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendormistral
Versionministral-8b
Release Date2024-10-16
Context Window128000 tokens
Input Modalitiestext
Output Modalitiestext
LicenseMistral Research License
Documentationhttps://docs.mistral.ai/

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU64.9%2024-10-165-shotview
HUMANEVAL63.3pass@12024-10-16—view
GSM8K51.7%2024-10-160-shot CoTview
MATH39.9%2024-10-160-shot CoTview
BBH60%2024-10-163-shot CoTview
GPQA35.1%2024-10-160-shotview
IFEVAL56.1%2024-10-16prompt_strictview
ARC85.5%2024-10-16challengeview
MUSR37%2024-10-160-shotview
WINOGRANDE75.3%2024-10-160-shotview

Pricing

TierPriceCurrency
Input$0.1 / MtokUSD
Output$0.1 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://mistral.ai/technology/ · as of 2024-10-16

Compliance

  • Data Residency: EU
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✓

Ministral 8B

Model Overview

Mistral Ministral 8B 边缘模型, 128K 上下文, 8B 参数, 为本地与边缘计算优化, 性能超越 Llama 3 8B。

Core Specifications

VendorVersionRelease DateContext WindowInput ModalitiesOutput ModalitiesLicense
Mistralministral-8b2024-10-16128KtexttextMistral Research License

Benchmark Performance

BenchmarkScoreUnitNotes
MMLU (Massive Multitask Language Understanding)64.9%5-shot
HumanEval63.3pass@1—
GSM8K (Grade School Math 8K)51.7%0-shot CoT
MATH39.9%0-shot CoT
BBH (BIG-Bench Hard)60.0%3-shot CoT
GPQA35.1%0-shot
IFEval56.1%prompt_strict
ARC85.5%challenge
MUSR37.0%0-shot
WinoGrande75.3%0-shot

Pricing

InputOutputCache ReadCache Write
————

per million tokens

Strengths

  • Context window 128K.

Weaknesses

  • Proprietary, not self-hostable.

Use Cases

  • Long document summarization

References