In this repo, I will be playing around with the byte-pair encoding algorithm as the mechanism for tokenization based on the paper "Language Models are unsupervised multitask learners" and the Youtube Video on GPT Tokenizer by Andrej Karpathy
| Name | Name | Last commit date | ||
|---|---|---|---|---|
In this repo, I will be playing around with the byte-pair encoding algorithm as the mechanism for tokenization based on the paper "Language Models are unsupervised multitask learners" and the Youtube Video on GPT Tokenizer by Andrej Karpathy