UltraBERT
Apache-2.0 English encoder family (150M, 380M) with 32K native context that reports beating ModernBERT, plus a 150M embedder.
Overview
UltraBERT is an encoder model family from DMIS Lab, published October 14, 2026 (paper arXiv 2610.15028), with 150M and 380M parameter models and a 150M embedding variant, UltraBERT-Embed-150M. All have a native 32K-token context, with 99% passkey recall up to 32K. The authors report higher scores than same-size ModernBERT on 10 benchmarks, e.g. GLUE 89.5 vs 88.4 and BEIR 45.2 vs 42.6 at 150M, with NomicBERT ahead on MultiCoNER in the base table. UltraBERT-Embed-150M scores 67.24 on MTEB(eng, v2), which the authors call the best dense result under 200M parameters, trained with public data only.
This is a sourced directory entry. Features and capabilities are reported in the linked sources; this entry does not document independent Aymorphic testing.
Key features
- 150M and 380M encoders, English, 32K native context
- 99% passkey recall at 32K for both sizes
- GLUE 89.5 (150M) and 91.0 (380M) vs ModernBERT 88.4 and 90.4
- UltraBERT-Embed-150M: 67.24 MTEB(eng, v2), 73.30 MTEB Code, 79.30 LongEmbed
- Apache 2.0; native in transformers>=5.20; Sentence Transformers for the embedder
- Embedder trained in two contrastive stages without distillation, on public data
Use cases
- Long-document retrieval and embedding with a small model
- Fine-tuning for classification, NER and reward modeling
- Code and math retrieval (RAR-b)
Pricing
Free, open weights under Apache 2.0.
Pricing reflects the source record and may change. Confirm current terms with the provider.
Dates and sources
- Released
- 14 Oct 2026
- Record updated
- 19 Oct 2026
- Record last checked
- 19 Oct 2026