Skip to content

Add 2-byte pair lookup table to skip hashmap for initial BPE merge scan#551

Open
augustasio wants to merge 2 commits into
openai:mainfrom
augustasio:pair-table-optimization
Open

Add 2-byte pair lookup table to skip hashmap for initial BPE merge scan#551
augustasio wants to merge 2 commits into
openai:mainfrom
augustasio:pair-table-optimization

Test: assert pair-table path produces identical output to hashmap path

4d95458
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs