r/LocalLLaMA
Posted by u/InternationalAsk1490
Fun fact: Anthropic has never open-sourced any LLMs
Tools 702 points
94 comments
3 weeks ago
I’ve been working on a little side project comparing tokenizer efficiency across different companies’ models for multilingual encoding. Then I saw Anthropic’s announcement today and suddenly realized: there’s no way to analyze claude’s tokenizer lmao! edit: Google once mentioned in a paper that Gemma and Gemini share the same tokenizer. OpenAI has already open‑sourced their tokenizers (and gpt‑oss). And don’t even get me started on Llama (Llama 5 pls 😭).
More from r/LocalLLaMA
r/LocalLLaMA · u/KvAk_AKPlaysYT
Recent
Hot
Anthropic: "We’ve identified industrial-scale distillation attacks on our models by DeepSeek, Moonshot AI, and MiniMax." 🚨
Tools
3.1K 674 3 weeks ago
r/LocalLLaMA · u/HeadAcanthisitt...
Recent
Hot
I feel personally attacked
Tools
3.0K 151 1 week ago
r/LocalLLaMA · u/Xhehab_
Recent
Hot
Distillation when you do it. Training when we do it.
Tools
2.6K 156 3 weeks ago