| Property | Value |
|---|---|
| License | GovTech Singapore |
| Paper | Technical Report |
| Max Context Length | 514 tokens |
| Task Type | Binary Classification |
What is stsb-roberta-base-off-topic?
stsb-roberta-base-off-topic is a specialized model developed by GovTech for determining whether user prompts are off-topic relative to a system's intended purpose. Built on the Cross Encoder STSB RoBERTa Base architecture, this model achieves impressive performance metrics with 0.99 ROC-AUC and F1 scores.
Implementation Details
The model is implemented as a fine-tuned version of the stsb-roberta-base architecture, optimized for binary classification tasks. It supports both ONNX and PyTorch/SafeTensors deployment options, making it versatile for different production environments.
- Achieves 0.99 precision and 0.99 recall on benchmark datasets
- Supports maximum context length of 514 tokens
- Extensively evaluated on synthetic and external datasets including JailbreakBench, HarmBench, and TrustLLM
Core Capabilities
- Binary classification for on-topic/off-topic detection
- Enterprise-grade performance for LLM applications
- Flexible deployment options with ONNX and PyTorch support
- Robust evaluation across multiple benchmark datasets
Frequently Asked Questions
Q: What makes this model unique?
This model stands out for its exceptional performance in off-topic detection, achieving near-perfect scores across all key metrics (ROC-AUC, F1, Precision, Recall) and outperforming both pre-trained models and prompt engineering approaches.
Q: What are the recommended use cases?
The model is ideal for enterprise LLM applications requiring robust content moderation, specifically for detecting off-topic user inputs that deviate from the system's intended purpose. It's particularly useful for maintaining conversation relevance in production environments.