RoBERTa: A Robustly Optimized BERT Pretraining Approach

June 12, 2025

We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it.

https://arxiv.org/abs/1907.11692

RoBERTa: A Robustly Optimized BERT Pretraining Approach <body> <h1> Array ( [0] => eoberta: a robustly optimized bert pretraining approach [1] => doberta: a robustly optimized bert pretraining approach [2] => foberta: a robustly optimized bert pretraining approach [3] => toberta: a robustly optimized bert pretraining approach [4] => 5oberta: a robustly optimized bert pretraining approach [5] => 4oberta: a robustly optimized bert pretraining approach [6] => riberta: a robustly optimized bert pretraining approach [7] => rkberta: a robustly optimized bert pretraining approach [8] => rlberta: a robustly optimized bert pretraining approach [9] => rpberta: a robustly optimized bert pretraining approach [10] => r0berta: a robustly optimized bert pretraining approach [11] => r9berta: a robustly optimized bert pretraining approach [12] => roverta: a robustly optimized bert pretraining approach [13] => ronerta: a robustly optimized bert pretraining approach [14] => roherta: a robustly optimized bert pretraining approach [15] => rogerta: a robustly optimized bert pretraining approach [16] => robwrta: a robustly optimized bert pretraining approach [17] => robsrta: a robustly optimized bert pretraining approach [18] => robdrta: a robustly optimized bert pretraining approach [19] => robrrta: a robustly optimized bert pretraining approach [20] => rob4rta: a robustly optimized bert pretraining approach [21] => rob3rta: a robustly optimized bert pretraining approach [22] => robeeta: a robustly optimized bert pretraining approach [23] => robedta: a robustly optimized bert pretraining approach [24] => robefta: a robustly optimized bert pretraining approach [25] => robetta: a robustly optimized bert pretraining approach [26] => robe5ta: a robustly optimized bert pretraining approach [27] => robe4ta: a robustly optimized bert pretraining approach [28] => roberra: a robustly optimized bert pretraining approach [29] => roberfa: a robustly optimized bert pretraining approach [30] => roberga: a robustly optimized bert pretraining approach [31] => roberya: a robustly optimized bert pretraining approach [32] => rober6a: a robustly optimized bert pretraining approach [33] => rober5a: a robustly optimized bert pretraining approach [34] => robertz: a robustly optimized bert pretraining approach [35] => roberts: a robustly optimized bert pretraining approach [36] => robertw: a robustly optimized bert pretraining approach [37] => robertq: a robustly optimized bert pretraining approach [38] => roberta: z robustly optimized bert pretraining approach [39] => roberta: s robustly optimized bert pretraining approach [40] => roberta: w robustly optimized bert pretraining approach [41] => roberta: q robustly optimized bert pretraining approach [42] => roberta: a eobustly optimized bert pretraining approach [43] => roberta: a dobustly optimized bert pretraining approach [44] => roberta: a fobustly optimized bert pretraining approach [45] => roberta: a tobustly optimized bert pretraining approach [46] => roberta: a 5obustly optimized bert pretraining approach [47] => roberta: a 4obustly optimized bert pretraining approach [48] => roberta: a ribustly optimized bert pretraining approach [49] => roberta: a rkbustly optimized bert pretraining approach [50] => roberta: a rlbustly optimized bert pretraining approach [51] => roberta: a rpbustly optimized bert pretraining approach [52] => roberta: a r0bustly optimized bert pretraining approach [53] => roberta: a r9bustly optimized bert pretraining approach [54] => roberta: a rovustly optimized bert pretraining approach [55] => roberta: a ronustly optimized bert pretraining approach [56] => roberta: a rohustly optimized bert pretraining approach [57] => roberta: a rogustly optimized bert pretraining approach [58] => roberta: a robystly optimized bert pretraining approach [59] => roberta: a robhstly optimized bert pretraining approach [60] => roberta: a robjstly optimized bert pretraining approach [61] => roberta: a robistly optimized bert pretraining approach [62] => roberta: a rob8stly optimized bert pretraining approach [63] => roberta: a rob7stly optimized bert pretraining approach [64] => roberta: a robuatly optimized bert pretraining approach [65] => roberta: a robuztly optimized bert pretraining approach [66] => roberta: a robuxtly optimized bert pretraining approach [67] => roberta: a robudtly optimized bert pretraining approach [68] => roberta: a robuetly optimized bert pretraining approach [69] => roberta: a robuwtly optimized bert pretraining approach [70] => roberta: a robusrly optimized bert pretraining approach [71] => roberta: a robusfly optimized bert pretraining approach [72] => roberta: a robusgly optimized bert pretraining approach [73] => roberta: a robusyly optimized bert pretraining approach [74] => roberta: a robus6ly optimized bert pretraining approach [75] => roberta: a robus5ly optimized bert pretraining approach [76] => roberta: a robustky optimized bert pretraining approach [77] => roberta: a robustpy optimized bert pretraining approach [78] => roberta: a robustoy optimized bert pretraining approach [79] => roberta: a robustlt optimized bert pretraining approach [80] => roberta: a robustlg optimized bert pretraining approach [81] => roberta: a robustlh optimized bert pretraining approach [82] => roberta: a robustlu optimized bert pretraining approach [83] => roberta: a robustl7 optimized bert pretraining approach [84] => roberta: a robustl6 optimized bert pretraining approach [85] => roberta: a robustly iptimized bert pretraining approach [86] => roberta: a robustly kptimized bert pretraining approach [87] => roberta: a robustly lptimized bert pretraining approach [88] => roberta: a robustly pptimized bert pretraining approach [89] => roberta: a robustly 0ptimized bert pretraining approach [90] => roberta: a robustly 9ptimized bert pretraining approach [91] => roberta: a robustly ootimized bert pretraining approach [92] => roberta: a robustly oltimized bert pretraining approach [93] => roberta: a robustly o-timized bert pretraining approach [94] => roberta: a robustly o0timized bert pretraining approach [95] => roberta: a robustly oprimized bert pretraining approach [96] => roberta: a robustly opfimized bert pretraining approach [97] => roberta: a robustly opgimized bert pretraining approach [98] => roberta: a robustly opyimized bert pretraining approach [99] => roberta: a robustly op6imized bert pretraining approach [100] => roberta: a robustly op5imized bert pretraining approach [101] => roberta: a robustly optumized bert pretraining approach [102] => roberta: a robustly optjmized bert pretraining approach [103] => roberta: a robustly optkmized bert pretraining approach [104] => roberta: a robustly optomized bert pretraining approach [105] => roberta: a robustly opt9mized bert pretraining approach [106] => roberta: a robustly opt8mized bert pretraining approach [107] => roberta: a robustly optinized bert pretraining approach [108] => roberta: a robustly optikized bert pretraining approach [109] => roberta: a robustly optijized bert pretraining approach [110] => roberta: a robustly optimuzed bert pretraining approach [111] => roberta: a robustly optimjzed bert pretraining approach [112] => roberta: a robustly optimkzed bert pretraining approach [113] => roberta: a robustly optimozed bert pretraining approach [114] => roberta: a robustly optim9zed bert pretraining approach [115] => roberta: a robustly optim8zed bert pretraining approach [116] => roberta: a robustly optimixed bert pretraining approach [117] => roberta: a robustly optimised bert pretraining approach [118] => roberta: a robustly optimiaed bert pretraining approach [119] => roberta: a robustly optimizwd bert pretraining approach [120] => roberta: a robustly optimizsd bert pretraining approach [121] => roberta: a robustly optimizdd bert pretraining approach [122] => roberta: a robustly optimizrd bert pretraining approach [123] => roberta: a robustly optimiz4d bert pretraining approach [124] => roberta: a robustly optimiz3d bert pretraining approach [125] => roberta: a robustly optimizes bert pretraining approach [126] => roberta: a robustly optimizex bert pretraining approach [127] => roberta: a robustly optimizec bert pretraining approach [128] => roberta: a robustly optimizef bert pretraining approach [129] => roberta: a robustly optimizer bert pretraining approach [130] => roberta: a robustly optimizee bert pretraining approach [131] => roberta: a robustly optimized vert pretraining approach [132] => roberta: a robustly optimized nert pretraining approach [133] => roberta: a robustly optimized hert pretraining approach [134] => roberta: a robustly optimized gert pretraining approach [135] => roberta: a robustly optimized bwrt pretraining approach [136] => roberta: a robustly optimized bsrt pretraining approach [137] => roberta: a robustly optimized bdrt pretraining approach [138] => roberta: a robustly optimized brrt pretraining approach [139] => roberta: a robustly optimized b4rt pretraining approach [140] => roberta: a robustly optimized b3rt pretraining approach [141] => roberta: a robustly optimized beet pretraining approach [142] => roberta: a robustly optimized bedt pretraining approach [143] => roberta: a robustly optimized beft pretraining approach [144] => roberta: a robustly optimized bett pretraining approach [145] => roberta: a robustly optimized be5t pretraining approach [146] => roberta: a robustly optimized be4t pretraining approach [147] => roberta: a robustly optimized berr pretraining approach [148] => roberta: a robustly optimized berf pretraining approach [149] => roberta: a robustly optimized berg pretraining approach [150] => roberta: a robustly optimized bery pretraining approach [151] => roberta: a robustly optimized ber6 pretraining approach [152] => roberta: a robustly optimized ber5 pretraining approach [153] => roberta: a robustly optimized bert oretraining approach [154] => roberta: a robustly optimized bert lretraining approach [155] => roberta: a robustly optimized bert -retraining approach [156] => roberta: a robustly optimized bert 0retraining approach [157] => roberta: a robustly optimized bert peetraining approach [158] => roberta: a robustly optimized bert pdetraining approach [159] => roberta: a robustly optimized bert pfetraining approach [160] => roberta: a robustly optimized bert ptetraining approach [161] => roberta: a robustly optimized bert p5etraining approach [162] => roberta: a robustly optimized bert p4etraining approach [163] => roberta: a robustly optimized bert prwtraining approach [164] => roberta: a robustly optimized bert prstraining approach [165] => roberta: a robustly optimized bert prdtraining approach [166] => roberta: a robustly optimized bert prrtraining approach [167] => roberta: a robustly optimized bert pr4training approach [168] => roberta: a robustly optimized bert pr3training approach [169] => roberta: a robustly optimized bert prerraining approach [170] => roberta: a robustly optimized bert prefraining approach [171] => roberta: a robustly optimized bert pregraining approach [172] => roberta: a robustly optimized bert preyraining approach [173] => roberta: a robustly optimized bert pre6raining approach [174] => roberta: a robustly optimized bert pre5raining approach [175] => roberta: a robustly optimized bert preteaining approach [176] => roberta: a robustly optimized bert pretdaining approach [177] => roberta: a robustly optimized bert pretfaining approach [178] => roberta: a robustly optimized bert prettaining approach [179] => roberta: a robustly optimized bert pret5aining approach [180] => roberta: a robustly optimized bert pret4aining approach [181] => roberta: a robustly optimized bert pretrzining approach [182] => roberta: a robustly optimized bert pretrsining approach [183] => roberta: a robustly optimized bert pretrwining approach [184] => roberta: a robustly optimized bert pretrqining approach [185] => roberta: a robustly optimized bert pretrauning approach [186] => roberta: a robustly optimized bert pretrajning approach [187] => roberta: a robustly optimized bert pretrakning approach [188] => roberta: a robustly optimized bert pretraoning approach [189] => roberta: a robustly optimized bert pretra9ning approach [190] => roberta: a robustly optimized bert pretra8ning approach [191] => roberta: a robustly optimized bert pretraibing approach [192] => roberta: a robustly optimized bert pretraiming approach [193] => roberta: a robustly optimized bert pretraijing approach [194] => roberta: a robustly optimized bert pretraihing approach [195] => roberta: a robustly optimized bert pretrainung approach [196] => roberta: a robustly optimized bert pretrainjng approach [197] => roberta: a robustly optimized bert pretrainkng approach [198] => roberta: a robustly optimized bert pretrainong approach [199] => roberta: a robustly optimized bert pretrain9ng approach [200] => roberta: a robustly optimized bert pretrain8ng approach [201] => roberta: a robustly optimized bert pretrainibg approach [202] => roberta: a robustly optimized bert pretrainimg approach [203] => roberta: a robustly optimized bert pretrainijg approach [204] => roberta: a robustly optimized bert pretrainihg approach [205] => roberta: a robustly optimized bert pretraininf approach [206] => roberta: a robustly optimized bert pretraininv approach [207] => roberta: a robustly optimized bert pretraininb approach [208] => roberta: a robustly optimized bert pretraininh approach [209] => roberta: a robustly optimized bert pretraininy approach [210] => roberta: a robustly optimized bert pretrainint approach [211] => roberta: a robustly optimized bert pretraining zpproach [212] => roberta: a robustly optimized bert pretraining spproach [213] => roberta: a robustly optimized bert pretraining wpproach [214] => roberta: a robustly optimized bert pretraining qpproach [215] => roberta: a robustly optimized bert pretraining aoproach [216] => roberta: a robustly optimized bert pretraining alproach [217] => roberta: a robustly optimized bert pretraining a-proach [218] => roberta: a robustly optimized bert pretraining a0proach [219] => roberta: a robustly optimized bert pretraining aporoach [220] => roberta: a robustly optimized bert pretraining aplroach [221] => roberta: a robustly optimized bert pretraining ap-roach [222] => roberta: a robustly optimized bert pretraining ap0roach [223] => roberta: a robustly optimized bert pretraining appeoach [224] => roberta: a robustly optimized bert pretraining appdoach [225] => roberta: a robustly optimized bert pretraining appfoach [226] => roberta: a robustly optimized bert pretraining apptoach [227] => roberta: a robustly optimized bert pretraining app5oach [228] => roberta: a robustly optimized bert pretraining app4oach [229] => roberta: a robustly optimized bert pretraining appriach [230] => roberta: a robustly optimized bert pretraining apprkach [231] => roberta: a robustly optimized bert pretraining apprlach [232] => roberta: a robustly optimized bert pretraining apprpach [233] => roberta: a robustly optimized bert pretraining appr0ach [234] => roberta: a robustly optimized bert pretraining appr9ach [235] => roberta: a robustly optimized bert pretraining approzch [236] => roberta: a robustly optimized bert pretraining approsch [237] => roberta: a robustly optimized bert pretraining approwch [238] => roberta: a robustly optimized bert pretraining approqch [239] => roberta: a robustly optimized bert pretraining approaxh [240] => roberta: a robustly optimized bert pretraining approavh [241] => roberta: a robustly optimized bert pretraining approafh [242] => roberta: a robustly optimized bert pretraining approadh [243] => roberta: a robustly optimized bert pretraining approacg [244] => roberta: a robustly optimized bert pretraining approacb [245] => roberta: a robustly optimized bert pretraining approacn [246] => roberta: a robustly optimized bert pretraining approacj [247] => roberta: a robustly optimized bert pretraining approacu [248] => roberta: a robustly optimized bert pretraining approacy ) </h1> </body>