[Feat] Allow tokenizer_name override#321

tianmu-li wants to merge 3 commits into

mlcommons:mainmlcommons/endpoints:mainfrom

tianmu-li:feat/separate_tokenizer_nametianmu-li/endpoints:feat/separate_tokenizer_nameCopy head branch name to clipboard