Clinical-Longformer
| Property | Value |
|---|---|
| Author | yikuan8 |
| Paper | View Paper |
| Downloads | 1,013,852 |
| Tags | Fill-Mask, Transformers, Clinical, Longformer |
What is Clinical-Longformer?
Clinical-Longformer is a specialized language model designed specifically for processing clinical text data. Built upon the Longformer architecture, it has been further pre-trained using MIMIC-III clinical notes to enhance its understanding of medical terminology and context. The model can process sequences of up to 4,096 tokens, making it particularly suitable for long clinical documents.
Implementation Details
The model was initialized from the base version of Longformer and underwent extensive pre-training on 6 32GB Tesla V100 GPUs. The training process included 200,000 steps with a batch size of 6×3 and learning rates of 3e-5, utilizing FP16 precision for accelerated training. The entire pre-training process took over two weeks to complete.
- Distributed training across 6 Tesla V100 GPUs
- FP16 precision for optimization
- 200,000 training steps
- Batch size of 6×3
Core Capabilities
- Extended context window of 4,096 tokens
- Superior performance in clinical NLP tasks
- Specialized in Named Entity Recognition (NER)
- Enhanced Question Answering (QA) capabilities
- Natural Language Inference (NLI)
- Clinical text classification
Frequently Asked Questions
Q: What makes this model unique?
Clinical-Longformer consistently outperforms ClinicalBERT by at least 2% across 10 baseline datasets, specifically in clinical domain tasks. Its extended context window and specialized training on MIMIC-III notes make it particularly effective for long clinical documents.
Q: What are the recommended use cases?
The model is ideal for clinical text analysis tasks including named entity recognition, question answering, natural language inference, and text classification in medical contexts. It's particularly useful when dealing with long clinical notes or documents that require extended context understanding.
