Key Specifications

Benchmark Performance

Benchmark	Score	Unit	Evaluated At	Notes	Source
MMLU	81.4	%	2024-02-19	5-shot	view
HUMANEVAL	78.7	pass@1	2024-02-19	—	view
GSM8K	82.5	%	2024-02-19	0-shot CoT	view
MATH	42.6	%	2024-02-19	0-shot CoT	view
BBH	72.4	%	2024-02-19	3-shot CoT	view
GPQA	35.1	%	2024-02-19	0-shot	view
IFEVAL	75.3	%	2024-02-19	prompt_strict	view
ARC	94.5	%	2024-02-19	challenge	view
MUSR	57.4	%	2024-02-19	0-shot	view
WINOGRANDE	82.1	%	2024-02-19	0-shot	view

Name: Nous Hermes 2 Mixtral 8x7B
Brand: other

NousResearch Hermes 2 Mixtral 8x7B 微调模型, 32K 上下文, 基于 Mixtral 8x7B, 改进指令遵循能力。

Vendor	Version	Release Date	Context Window	Input Modalities	Output Modalities	License
Other	nous-hermes-2-mixtral-8x7b	2024-02-19	32K	text	text	Apache 2.0

Benchmark	Score	Unit	Notes
MMLU (Massive Multitask Language Understanding)	81.4	%	5-shot
HumanEval	78.7	pass@1	—
GSM8K (Grade School Math 8K)	82.5	%	0-shot CoT
MATH	42.6	%	0-shot CoT
BBH (BIG-Bench Hard)	72.4	%	3-shot CoT
GPQA	35.1	%	0-shot
IFEval	75.3	%	prompt_strict
ARC	94.5	%	challenge
MUSR	57.4	%	0-shot
WinoGrande	82.1	%	0-shot

Input	Output	Cache Read	Cache Write
—	—	—	—

per million tokens