Masked-Language-Model

• Masked Language Model

  Predicts randomly masked tokens from context; primary pre-training objective for bidirectional encoders like BERT.

• Cloze Task

  Predicting masked tokens from context; unsupervised pre-training objective where random words are hidden and must be inferred.