IAAIALBERT
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Reduces BERT's parameters with factorized embeddings and cross-layer parameter sharing, so every layer reuses the same weights, plus an inter-sentence coherence loss.
Inside the method
Embedding (factorized) → [ Shared encoder layer ] × L
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Fixed depth with all layers shared
- KV / state strategy
- Not applicable (encoder)
Reading note
Achieved state-of-the-art GLUE, RACE and SQuAD results with fewer parameters than BERT-large; a canonical example of full-stack weight sharing in pretrained Transformers.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{albert2019,
title = {ALBERT: A Lite BERT for Self-supervised Learning of Language Representations},
author = {Zhenzhong Lan and Mingda Chen and Sebastian Goodman and Kevin Gimpel and Piyush Sharma and Radu Soricut},
year = {2019},
eprint = {1909.11942},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1909.11942}
}