RoBERTa: A Robustly Optimized BERT Pretraining Approach

May 12, 2026

We find that BERT was significantly undertrained and propose an im-proved recipe for training BERT models, which we call RoBERTa, that can match or exceed the performance of all of the post-BERT methods.

https://arxiv.org/pdf/1907.11692

RoBERTa: A Robustly Optimized BERT Pretraining Approach <body> <h1> Array ( [0] => eoberta: a robustly optimized bert pretraining approach [1] => doberta: a robustly optimized bert pretraining approach [2] => foberta: a robustly optimized bert pretraining approach [3] => toberta: a robustly optimized bert pretraining approach [4] => 5oberta: a robustly optimized bert pretraining approach [5] => 4oberta: a robustly optimized bert pretraining approach [6] => riberta: a robustly optimized bert pretraining approach [7] => rkberta: a robustly optimized bert pretraining approach [8] => rlberta: a robustly optimized bert pretraining approach [9] => rpberta: a robustly optimized bert pretraining approach [10] => r0berta: a robustly optimized bert pretraining approach [11] => r9berta: a robustly optimized bert pretraining approach [12] => roverta: a robustly optimized bert pretraining approach [13] => ronerta: a robustly optimized bert pretraining approach [14] => roherta: a robustly optimized bert pretraining approach [15] => rogerta: a robustly optimized bert pretraining approach [16] => robwrta: a robustly optimized bert pretraining approach [17] => robsrta: a robustly optimized bert pretraining approach [18] => robdrta: a robustly optimized bert pretraining approach [19] => robrrta: a robustly optimized bert pretraining approach [20] => rob4rta: a robustly optimized bert pretraining approach [21] => rob3rta: a robustly optimized bert pretraining approach [22] => robeeta: a robustly optimized bert pretraining approach [23] => robedta: a robustly optimized bert pretraining approach [24] => robefta: a robustly optimized bert pretraining approach [25] => robetta: a robustly optimized bert pretraining approach [26] => robe5ta: a robustly optimized bert pretraining approach [27] => robe4ta: a robustly optimized bert pretraining approach [28] => roberra: a robustly optimized bert pretraining approach [29] => roberfa: a robustly optimized bert pretraining approach [30] => roberga: a robustly optimized bert pretraining approach [31] => roberya: a robustly optimized bert pretraining approach [32] => rober6a: a robustly optimized bert pretraining approach [33] => rober5a: a robustly optimized bert pretraining approach [34] => robertz: a robustly optimized bert pretraining approach [35] => roberts: a robustly optimized bert pretraining approach [36] => robertw: a robustly optimized bert pretraining approach [37] => robertq: a robustly optimized bert pretraining approach [38] => roberta: z robustly optimized bert pretraining approach [39] => roberta: s robustly optimized bert pretraining approach [40] => roberta: w robustly optimized bert pretraining approach [41] => roberta: q robustly optimized bert pretraining approach [42] => roberta: a eobustly optimized bert pretraining approach [43] => roberta: a dobustly optimized bert pretraining approach [44] => roberta: a fobustly optimized bert pretraining approach [45] => roberta: a tobustly optimized bert pretraining approach [46] => roberta: a 5obustly optimized bert pretraining approach [47] => roberta: a 4obustly optimized bert pretraining approach [48] => roberta: a ribustly optimized bert pretraining approach [49] => roberta: a rkbustly optimized bert pretraining approach [50] => roberta: a rlbustly optimized bert pretraining approach [51] => roberta: a rpbustly optimized bert pretraining approach [52] => roberta: a r0bustly optimized bert pretraining approach [53] => roberta: a r9bustly optimized bert pretraining approach [54] => roberta: a rovustly optimized bert pretraining approach [55] => roberta: a ronustly optimized bert pretraining approach [56] => roberta: a rohustly optimized bert pretraining approach [57] => roberta: a rogustly optimized bert pretraining approach [58] => roberta: a robystly optimized bert pretraining approach [59] => roberta: a robhstly optimized bert pretraining approach [60] => roberta: a robjstly optimized bert pretraining approach [61] => roberta: a robistly optimized bert pretraining approach [62] => roberta: a rob8stly optimized bert pretraining approach [63] => roberta: a rob7stly optimized bert pretraining approach [64] => roberta: a robuatly optimized bert pretraining approach [65] => roberta: a robuztly optimized bert pretraining approach [66] => roberta: a robuxtly optimized bert pretraining approach [67] => roberta: a robudtly optimized bert pretraining approach [68] => roberta: a robuetly optimized bert pretraining approach [69] => roberta: a robuwtly optimized bert pretraining approach [70] => roberta: a robusrly optimized bert pretraining approach [71] => roberta: a robusfly optimized bert pretraining approach [72] => roberta: a robusgly optimized bert pretraining approach [73] => roberta: a robusyly optimized bert pretraining approach [74] => roberta: a robus6ly optimized bert pretraining approach [75] => roberta: a robus5ly optimized bert pretraining approach [76] => roberta: a robustky optimized bert pretraining approach [77] => roberta: a robustpy optimized bert pretraining approach [78] => roberta: a robustoy optimized bert pretraining approach [79] => roberta: a robustlt optimized bert pretraining approach [80] => roberta: a robustlg optimized bert pretraining approach [81] => roberta: a robustlh optimized bert pretraining approach [82] => roberta: a robustlu optimized bert pretraining approach [83] => roberta: a robustl7 optimized bert pretraining approach [84] => roberta: a robustl6 optimized bert pretraining approach [85] => roberta: a robustly iptimized bert pretraining approach [86] => roberta: a robustly kptimized bert pretraining approach [87] => roberta: a robustly lptimized bert pretraining approach [88] => roberta: a robustly pptimized bert pretraining approach [89] => roberta: a robustly 0ptimized bert pretraining approach [90] => roberta: a robustly 9ptimized bert pretraining approach [91] => roberta: a robustly ootimized bert pretraining approach [92] => roberta: a robustly oltimized bert pretraining approach [93] => roberta: a robustly o-timized bert pretraining approach [94] => roberta: a robustly o0timized bert pretraining approach [95] => roberta: a robustly oprimized bert pretraining approach [96] => roberta: a robustly opfimized bert pretraining approach [97] => roberta: a robustly opgimized bert pretraining approach [98] => roberta: a robustly opyimized bert pretraining approach [99] => roberta: a robustly op6imized bert pretraining approach [100] => roberta: a robustly op5imized bert pretraining approach [101] => roberta: a robustly optumized bert pretraining approach [102] => roberta: a robustly optjmized bert pretraining approach [103] => roberta: a robustly optkmized bert pretraining approach [104] => roberta: a robustly optomized bert pretraining approach [105] => roberta: a robustly opt9mized bert pretraining approach [106] => roberta: a robustly opt8mized bert pretraining approach [107] => roberta: a robustly optinized bert pretraining approach [108] => roberta: a robustly optikized bert pretraining approach [109] => roberta: a robustly optijized bert pretraining approach [110] => roberta: a robustly optimuzed bert pretraining approach [111] => roberta: a robustly optimjzed bert pretraining approach [112] => roberta: a robustly optimkzed bert pretraining approach [113] => roberta: a robustly optimozed bert pretraining approach [114] => roberta: a robustly optim9zed bert pretraining approach [115] => roberta: a robustly optim8zed bert pretraining approach [116] => roberta: a robustly optimixed bert pretraining approach [117] => roberta: a robustly optimised bert pretraining approach [118] => roberta: a robustly optimiaed bert pretraining approach [119] => roberta: a robustly optimizwd bert pretraining approach [120] => roberta: a robustly optimizsd bert pretraining approach [121] => roberta: a robustly optimizdd bert pretraining approach [122] => roberta: a robustly optimizrd bert pretraining approach [123] => roberta: a robustly optimiz4d bert pretraining approach [124] => roberta: a robustly optimiz3d bert pretraining approach [125] => roberta: a robustly optimizes bert pretraining approach [126] => roberta: a robustly optimizex bert pretraining approach [127] => roberta: a robustly optimizec bert pretraining approach [128] => roberta: a robustly optimizef bert pretraining approach [129] => roberta: a robustly optimizer bert pretraining approach [130] => roberta: a robustly optimizee bert pretraining approach [131] => roberta: a robustly optimized vert pretraining approach [132] => roberta: a robustly optimized nert pretraining approach [133] => roberta: a robustly optimized hert pretraining approach [134] => roberta: a robustly optimized gert pretraining approach [135] => roberta: a robustly optimized bwrt pretraining approach [136] => roberta: a robustly optimized bsrt pretraining approach [137] => roberta: a robustly optimized bdrt pretraining approach [138] => roberta: a robustly optimized brrt pretraining approach [139] => roberta: a robustly optimized b4rt pretraining approach [140] => roberta: a robustly optimized b3rt pretraining approach [141] => roberta: a robustly optimized beet pretraining approach [142] => roberta: a robustly optimized bedt pretraining approach [143] => roberta: a robustly optimized beft pretraining approach [144] => roberta: a robustly optimized bett pretraining approach [145] => roberta: a robustly optimized be5t pretraining approach [146] => roberta: a robustly optimized be4t pretraining approach [147] => roberta: a robustly optimized berr pretraining approach [148] => roberta: a robustly optimized berf pretraining approach [149] => roberta: a robustly optimized berg pretraining approach [150] => roberta: a robustly optimized bery pretraining approach [151] => roberta: a robustly optimized ber6 pretraining approach [152] => roberta: a robustly optimized ber5 pretraining approach [153] => roberta: a robustly optimized bert oretraining approach [154] => roberta: a robustly optimized bert lretraining approach [155] => roberta: a robustly optimized bert -retraining approach [156] => roberta: a robustly optimized bert 0retraining approach [157] => roberta: a robustly optimized bert peetraining approach [158] => roberta: a robustly optimized bert pdetraining approach [159] => roberta: a robustly optimized bert pfetraining approach [160] => roberta: a robustly optimized bert ptetraining approach [161] => roberta: a robustly optimized bert p5etraining approach [162] => roberta: a robustly optimized bert p4etraining approach [163] => roberta: a robustly optimized bert prwtraining approach [164] => roberta: a robustly optimized bert prstraining approach [165] => roberta: a robustly optimized bert prdtraining approach [166] => roberta: a robustly optimized bert prrtraining approach [167] => roberta: a robustly optimized bert pr4training approach [168] => roberta: a robustly optimized bert pr3training approach [169] => roberta: a robustly optimized bert prerraining approach [170] => roberta: a robustly optimized bert prefraining approach [171] => roberta: a robustly optimized bert pregraining approach [172] => roberta: a robustly optimized bert preyraining approach [173] => roberta: a robustly optimized bert pre6raining approach [174] => roberta: a robustly optimized bert pre5raining approach [175] => roberta: a robustly optimized bert preteaining approach [176] => roberta: a robustly optimized bert pretdaining approach [177] => roberta: a robustly optimized bert pretfaining approach [178] => roberta: a robustly optimized bert prettaining approach [179] => roberta: a robustly optimized bert pret5aining approach [180] => roberta: a robustly optimized bert pret4aining approach [181] => roberta: a robustly optimized bert pretrzining approach [182] => roberta: a robustly optimized bert pretrsining approach [183] => roberta: a robustly optimized bert pretrwining approach [184] => roberta: a robustly optimized bert pretrqining approach [185] => roberta: a robustly optimized bert pretrauning approach [186] => roberta: a robustly optimized bert pretrajning approach [187] => roberta: a robustly optimized bert pretrakning approach [188] => roberta: a robustly optimized bert pretraoning approach [189] => roberta: a robustly optimized bert pretra9ning approach [190] => roberta: a robustly optimized bert pretra8ning approach [191] => roberta: a robustly optimized bert pretraibing approach [192] => roberta: a robustly optimized bert pretraiming approach [193] => roberta: a robustly optimized bert pretraijing approach [194] => roberta: a robustly optimized bert pretraihing approach [195] => roberta: a robustly optimized bert pretrainung approach [196] => roberta: a robustly optimized bert pretrainjng approach [197] => roberta: a robustly optimized bert pretrainkng approach [198] => roberta: a robustly optimized bert pretrainong approach [199] => roberta: a robustly optimized bert pretrain9ng approach [200] => roberta: a robustly optimized bert pretrain8ng approach [201] => roberta: a robustly optimized bert pretrainibg approach [202] => roberta: a robustly optimized bert pretrainimg approach [203] => roberta: a robustly optimized bert pretrainijg approach [204] => roberta: a robustly optimized bert pretrainihg approach [205] => roberta: a robustly optimized bert pretraininf approach [206] => roberta: a robustly optimized bert pretraininv approach [207] => roberta: a robustly optimized bert pretraininb approach [208] => roberta: a robustly optimized bert pretraininh approach [209] => roberta: a robustly optimized bert pretraininy approach [210] => roberta: a robustly optimized bert pretrainint approach [211] => roberta: a robustly optimized bert pretraining zpproach [212] => roberta: a robustly optimized bert pretraining spproach [213] => roberta: a robustly optimized bert pretraining wpproach [214] => roberta: a robustly optimized bert pretraining qpproach [215] => roberta: a robustly optimized bert pretraining aoproach [216] => roberta: a robustly optimized bert pretraining alproach [217] => roberta: a robustly optimized bert pretraining a-proach [218] => roberta: a robustly optimized bert pretraining a0proach [219] => roberta: a robustly optimized bert pretraining aporoach [220] => roberta: a robustly optimized bert pretraining aplroach [221] => roberta: a robustly optimized bert pretraining ap-roach [222] => roberta: a robustly optimized bert pretraining ap0roach [223] => roberta: a robustly optimized bert pretraining appeoach [224] => roberta: a robustly optimized bert pretraining appdoach [225] => roberta: a robustly optimized bert pretraining appfoach [226] => roberta: a robustly optimized bert pretraining apptoach [227] => roberta: a robustly optimized bert pretraining app5oach [228] => roberta: a robustly optimized bert pretraining app4oach [229] => roberta: a robustly optimized bert pretraining appriach [230] => roberta: a robustly optimized bert pretraining apprkach [231] => roberta: a robustly optimized bert pretraining apprlach [232] => roberta: a robustly optimized bert pretraining apprpach [233] => roberta: a robustly optimized bert pretraining appr0ach [234] => roberta: a robustly optimized bert pretraining appr9ach [235] => roberta: a robustly optimized bert pretraining approzch [236] => roberta: a robustly optimized bert pretraining approsch [237] => roberta: a robustly optimized bert pretraining approwch [238] => roberta: a robustly optimized bert pretraining approqch [239] => roberta: a robustly optimized bert pretraining approaxh [240] => roberta: a robustly optimized bert pretraining approavh [241] => roberta: a robustly optimized bert pretraining approafh [242] => roberta: a robustly optimized bert pretraining approadh [243] => roberta: a robustly optimized bert pretraining approacg [244] => roberta: a robustly optimized bert pretraining approacb [245] => roberta: a robustly optimized bert pretraining approacn [246] => roberta: a robustly optimized bert pretraining approacj [247] => roberta: a robustly optimized bert pretraining approacu [248] => roberta: a robustly optimized bert pretraining approacy ) </h1> </body>