Byte-Pair Encoding Tokenization
InterviewCodingtokenizationBPENLP
Related module: Sequence models and attention
Problem statement
Implement the Byte-Pair Encoding (BPE) algorithm for learning a tokenizer vocabulary.
Given a corpus, BPE iteratively merges the most frequent adjacent pair of tokens until reaching a target vocabulary size.
Implement both the vocabulary learning and encoding phases.