Llama 3.1 Nemotron 70B: Complete Benchmark Performance Guide

meta 3.1-nemotron-70b 2024-10-17 Llama 3.1 Community License

In-depth analysis of Llama 3.1 Nemotron 70B performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendormeta
Version3.1-nemotron-70b
Release Date2024-10-17
Context Window128000 tokens
Input Modalitiestext
Output Modalitiestext
LicenseLlama 3.1 Community License
Documentationhttps://llama.meta.com/docs/

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU81.5%2024-10-175-shotview
HUMANEVAL79.8pass@12024-10-17—view
GSM8K84.6%2024-10-170-shot CoTview
MATH40%2024-10-170-shot CoTview
BBH70.3%2024-10-173-shot CoTview
GPQA42.6%2024-10-170-shotview
IFEVAL74.5%2024-10-17prompt_strictview
ARC93.7%2024-10-17challengeview
MUSR54.1%2024-10-170-shotview
WINOGRANDE84.7%2024-10-170-shotview

Pricing

TierPriceCurrency
Input$0.9 / MtokUSD
Output$0.9 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://ai.meta.com/blog/ · as of 2024-10-17

Compliance

  • Data Residency: self-host
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

Llama 3.1 Nemotron 70B

Model Overview

Meta/NVIDIA Llama 3.1 Nemotron 70B, 128K 上下文, NVIDIA 优化版本, 在 Arena 排行榜上接近 GPT-4o。

Core Specifications

VendorVersionRelease DateContext WindowInput ModalitiesOutput ModalitiesLicense
Meta3.1-nemotron-70b2024-10-17128KtexttextLlama 3.1 Community License

Benchmark Performance

BenchmarkScoreUnitNotes
MMLU (Massive Multitask Language Understanding)81.5%5-shot
HumanEval79.8pass@1—
GSM8K (Grade School Math 8K)84.6%0-shot CoT
MATH40.0%0-shot CoT
BBH (BIG-Bench Hard)70.3%3-shot CoT
GPQA42.6%0-shot
IFEval74.5%prompt_strict
ARC93.7%challenge
MUSR54.1%0-shot
WinoGrande84.7%0-shot

Pricing

InputOutputCache ReadCache Write
————

per million tokens

Strengths

  • MMLU score 81.5, strong knowledge reasoning.

Weaknesses

  • Proprietary, not self-hostable.

Use Cases

  • Code generation and debugging

References