Ministral 3B: Complete Benchmark Performance Guide

mistral ministral-3b 2024-10-16 Mistral Research License

In-depth analysis of Ministral 3B performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendormistral
Versionministral-3b
Release Date2024-10-16
Context Window128000 tokens
Input Modalitiestext
Output Modalitiestext
LicenseMistral Research License
Documentationhttps://docs.mistral.ai/

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU42.4%2024-10-165-shotview
HUMANEVAL37.2pass@12024-10-16—view
GSM8K32.2%2024-10-160-shot CoTview
MATH10.1%2024-10-160-shot CoTview
BBH50.1%2024-10-163-shot CoTview
GPQA19%2024-10-160-shotview
IFEVAL47.8%2024-10-16prompt_strictview
ARC81.5%2024-10-16challengeview
MUSR29.4%2024-10-160-shotview
WINOGRANDE64.8%2024-10-160-shotview

Pricing

TierPriceCurrency
Input$0.04 / MtokUSD
Output$0.04 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://mistral.ai/technology/ · as of 2024-10-16

Compliance

  • Data Residency: EU
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✓

Ministral 3B

Model Overview

Mistral Ministral 3B 超轻量边缘模型, 128K 上下文, 3B 参数, 最具性价比的边缘部署模型。

Core Specifications

VendorVersionRelease DateContext WindowInput ModalitiesOutput ModalitiesLicense
Mistralministral-3b2024-10-16128KtexttextMistral Research License

Benchmark Performance

BenchmarkScoreUnitNotes
MMLU (Massive Multitask Language Understanding)42.4%5-shot
HumanEval37.2pass@1—
GSM8K (Grade School Math 8K)32.2%0-shot CoT
MATH10.1%0-shot CoT
BBH (BIG-Bench Hard)50.1%3-shot CoT
GPQA19.0%0-shot
IFEval47.8%prompt_strict
ARC81.5%challenge
MUSR29.4%0-shot
WinoGrande64.8%0-shot

Pricing

InputOutputCache ReadCache Write
————

per million tokens

Strengths

  • Context window 128K.

Weaknesses

  • MMLU 42.4, weak knowledge reasoning.
  • HumanEval 37.2, coding weak.
  • Proprietary, not self-hostable.

Use Cases

  • Long document summarization

References