OLMo-7B-0724-Instruct-hf

What is OLMo-7B-0724-Instruct-hf?

OLMo-7B-0724-Instruct-hf is an advanced language model developed by Allen Institute for AI (AI2) as part of their initiative to accelerate the science of language models. This instruct-tuned version is built upon the base OLMo model and has been specifically optimized through SFT (Supervised Fine-Tuning) and DPO (Direct Preference Optimization) using the Tulu 2 SFT Mix and cleaned UltraFeedback datasets.

Implementation Details

The model implements a Transformer-style autoregressive architecture with significant technical capabilities. It utilizes 32 layers, 4096 hidden size, and 32 attention heads, supporting a context length of 4096 tokens. The training process involved two phases: initial SFT with a learning rate of 2×10^-6 and DPO with 5×10^-7, both running for 3 epochs.

• Trained on Dolma dataset with 2.7T tokens
• Implements modern attention mechanisms with 32 heads
• Supports both float16 quantization and 8-bit inference
• Compatible with transformers library version 4.40.0 and newer

Core Capabilities

• Strong performance on MMLU (52.8% zero-shot)
• High AlpacaEval performance (83.5% win rate)
• Excellent toxicity control (1.7% on ToxiGen)
• Strong truthfulness metrics (70.3% on TruthfulQA)