Lightweight Spanish Language Models

ALBETO and DistilBETO: Lightweight Spanish Language Models

ALBETO and DistilBETO are versions of ALBERT and DistilBERT pre-trained exclusively on Spanish corpora. We train several versions of ALBETO ranging from 5M to 223M parameters and one of DistilBETO with 67M parameters.

All models (pre-trained and fine-tuned) can be found on our organization on the HuggingFace Hub.

The following tables show the results of every model in different evaluation tasks with links to the model on the HuggingFace Hub.

Pre-trained models

Model	Parameters	Evaluation Average	Size	Performance
BETO uncased	110M	77.48	1x	0.95x
BETO cased	110M	81.02	1x	1x
DistilBETO	67M	73.22	1.64x	0.90x
ALBETO tiny	5M	70.86	22x	0.87x
ALBETO base	12M	79.35	9.16x	0.97x
ALBETO large	18M	78.12	6.11x	0.96x
ALBETO xlarge	59M	80.20	1.86x	0.98x
ALBETO xxlarge	223M	81.34	0.49x	1x

Fine-tuned models

POS / NER

	POS	NER
BETO uncased	97.70	83.76
BETO cased	98.84	88.24
DistilBETO	97.50	81.19
ALBETO tiny	97.04	75.11
ALBETO base	98.08	83.35
ALBETO large	97.87	83.72
ALBETO xlarge	98.06	82.30
ALBETO xxlarge	98.35	84.36

MLDoc / PAWS-X / XNLI

	MLDoc	PAWS-X	XNLI
BETO uncased	96.38	84.25	77.76
BETO cased	96.65	89.80	81.98
DistilBETO	96.35	75.80	76.59
ALBETO tiny	95.82	80.20	73.43
ALBETO base	96.07	87.95	79.88
ALBETO large	92.22	86.05	78.94
ALBETO xlarge	95.70	89.05	81.68
ALBETO xxlarge	96.85	89.85	82.42

QA

Model	MLQA	SQAC	TAR, XQuAD
BETO uncased	64.12 / 40.83	72.22 / 53.45	74.81 / 54.62
BETO cased	67.65 / 43.38	78.65 / 60.94	77.81 / 56.97
DistilBETO	57.97 / 35.50	64.41 / 45.34	66.97 / 46.55
ALBETO tiny	51.84 / 28.28	59.28 / 39.16	66.43 / 45.71
ALBETO base	66.12 / 41.10	77.71 / 59.84	77.18 / 57.05
ALBETO large	65.56 / 40.98	76.36 / 56.54	76.72 / 56.21
ALBETO xlarge	68.26 / 43.76	78.64 / 59.26	80.15 / 59.66
ALBETO xxlarge	70.17 / 45.99	81.49 / 62.67	79.13 / 58.40

Citation

ALBETO and DistilBETO: Lightweight Spanish Language Models

To cite this resource in a publication please use the following:

 @inproceedings{canete2022albeto,
   title="ALBETO and DistilBETO: Lightweight Spanish Language Models",
   author="Ca{\~n}ete, Jos{\'e} and Donoso, Sebasti{\'a}n and Bravo-Marquez, Felipe and Carvallo, Andr{\'e}s and Araujo, Vladimir",
   booktitle = "Proceedings of the 13th Language Resources and Evaluation Conference",
   year = "2022",
   address = "Marseille, France",
   publisher = "European Language Resources Association",
 }