We find that BERT was significantly undertrained and propose an im-proved recipe for training BERT models, which we call RoBERTa, that can match or exceed the performance of all of the post-BERT methods.
eoberta: a robustly optimized bert pretraining approach, doberta: a robustly optimized bert pretraining approach, foberta: a robustly optimized bert pretraining approach, toberta: a robustly optimized bert pretraining approach, 5oberta: a robustly optimized bert pretraining approach, 4oberta: a robustly optimized bert pretraining approach, riberta: a robustly optimized bert pretraining approach, rkberta: a robustly optimized bert pretraining approach, rlberta: a robustly optimized bert pretraining approach, rpberta: a robustly optimized bert pretraining approach, r0berta: a robustly optimized bert pretraining approach, r9berta: a robustly optimized bert pretraining approach, roverta: a robustly optimized bert pretraining approach, ronerta: a robustly optimized bert pretraining approach, roherta: a robustly optimized bert pretraining approach, rogerta: a robustly optimized bert pretraining approach, robwrta: a robustly optimized bert pretraining approach, robsrta: a robustly optimized bert pretraining approach, robdrta: a robustly optimized bert pretraining approach, robrrta: a robustly optimized bert pretraining approach, rob4rta: a robustly optimized bert pretraining approach, rob3rta: a robustly optimized bert pretraining approach, robeeta: a robustly optimized bert pretraining approach, robedta: a robustly optimized bert pretraining approach, robefta: a robustly optimized bert pretraining approach, robetta: a robustly optimized bert pretraining approach, robe5ta: a robustly optimized bert pretraining approach, robe4ta: a robustly optimized bert pretraining approach, roberra: a robustly optimized bert pretraining approach, roberfa: a robustly optimized bert pretraining approach, roberga: a robustly optimized bert pretraining approach, roberya: a robustly optimized bert pretraining approach, rober6a: a robustly optimized bert pretraining approach, rober5a: a robustly optimized bert pretraining approach, robertz: a robustly optimized bert pretraining approach, roberts: a robustly optimized bert pretraining approach, robertw: a robustly optimized bert pretraining approach, robertq: a robustly optimized bert pretraining approach, roberta: z robustly optimized bert pretraining approach, roberta: s robustly optimized bert pretraining approach, roberta: w robustly optimized bert pretraining approach, roberta: q robustly optimized bert pretraining approach, roberta: a eobustly optimized bert pretraining approach, roberta: a dobustly optimized bert pretraining approach, roberta: a fobustly optimized bert pretraining approach, roberta: a tobustly optimized bert pretraining approach, roberta: a 5obustly optimized bert pretraining approach, roberta: a 4obustly optimized bert pretraining approach, roberta: a ribustly optimized bert pretraining approach, roberta: a rkbustly optimized bert pretraining approach, roberta: a rlbustly optimized bert pretraining approach, roberta: a rpbustly optimized bert pretraining approach, roberta: a r0bustly optimized bert pretraining approach, roberta: a r9bustly optimized bert pretraining approach, roberta: a rovustly optimized bert pretraining approach, roberta: a ronustly optimized bert pretraining approach, roberta: a rohustly optimized bert pretraining approach, roberta: a rogustly optimized bert pretraining approach, roberta: a robystly optimized bert pretraining approach, roberta: a robhstly optimized bert pretraining approach, roberta: a robjstly optimized bert pretraining approach, roberta: a robistly optimized bert pretraining approach, roberta: a rob8stly optimized bert pretraining approach, roberta: a rob7stly optimized bert pretraining approach, roberta: a robuatly optimized bert pretraining approach, roberta: a robuztly optimized bert pretraining approach, roberta: a robuxtly optimized bert pretraining approach, roberta: a robudtly optimized bert pretraining approach, roberta: a robuetly optimized bert pretraining approach, roberta: a robuwtly optimized bert pretraining approach, roberta: a robusrly optimized bert pretraining approach, roberta: a robusfly optimized bert pretraining approach, roberta: a robusgly optimized bert pretraining approach, roberta: a robusyly optimized bert pretraining approach, roberta: a robus6ly optimized bert pretraining approach, roberta: a robus5ly optimized bert pretraining approach, roberta: a robustky optimized bert pretraining approach, roberta: a robustpy optimized bert pretraining approach, roberta: a robustoy optimized bert pretraining approach, roberta: a robustlt optimized bert pretraining approach, roberta: a robustlg optimized bert pretraining approach, roberta: a robustlh optimized bert pretraining approach, roberta: a robustlu optimized bert pretraining approach, roberta: a robustl7 optimized bert pretraining approach, roberta: a robustl6 optimized bert pretraining approach, roberta: a robustly iptimized bert pretraining approach, roberta: a robustly kptimized bert pretraining approach, roberta: a robustly lptimized bert pretraining approach, roberta: a robustly pptimized bert pretraining approach, roberta: a robustly 0ptimized bert pretraining approach, roberta: a robustly 9ptimized bert pretraining approach, roberta: a robustly ootimized bert pretraining approach, roberta: a robustly oltimized bert pretraining approach, roberta: a robustly o-timized bert pretraining approach, roberta: a robustly o0timized bert pretraining approach, roberta: a robustly oprimized bert pretraining approach, roberta: a robustly opfimized bert pretraining approach, roberta: a robustly opgimized bert pretraining approach, roberta: a robustly opyimized bert pretraining approach, roberta: a robustly op6imized bert pretraining approach, roberta: a robustly op5imized bert pretraining approach, roberta: a robustly optumized bert pretraining approach, roberta: a robustly optjmized bert pretraining approach, roberta: a robustly optkmized bert pretraining approach, roberta: a robustly optomized bert pretraining approach, roberta: a robustly opt9mized bert pretraining approach, roberta: a robustly opt8mized bert pretraining approach, roberta: a robustly optinized bert pretraining approach, roberta: a robustly optikized bert pretraining approach, roberta: a robustly optijized bert pretraining approach, roberta: a robustly optimuzed bert pretraining approach, roberta: a robustly optimjzed bert pretraining approach, roberta: a robustly optimkzed bert pretraining approach, roberta: a robustly optimozed bert pretraining approach, roberta: a robustly optim9zed bert pretraining approach, roberta: a robustly optim8zed bert pretraining approach, roberta: a robustly optimixed bert pretraining approach, roberta: a robustly optimised bert pretraining approach, roberta: a robustly optimiaed bert pretraining approach, roberta: a robustly optimizwd bert pretraining approach