r/LocalLLaMA
Posted by u/InternationalAsk1490
Fun fact: Anthropic has never open-sourced any LLMs
Tools 702 points
94 comments
2 months ago
I’ve been working on a little side project comparing tokenizer efficiency across different companies’ models for multilingual encoding. Then I saw Anthropic’s announcement today and suddenly realized: there’s no way to analyze claude’s tokenizer lmao! edit: Google once mentioned in a paper that Gemma and Gemini share the same tokenizer. OpenAI has already open‑sourced their tokenizers (and gpt‑oss). And don’t even get me started on Llama (Llama 5 pls 😭).
More from r/LocalLLaMA
r/LocalLLaMA · u/jacek2023
Recent
Hot
This is where we are right now, LocalLLaMA
the future is now
Tools
3.2K 439 0 months ago
r/LocalLLaMA · u/KvAk_AKPlaysYT
Hot
Anthropic: "We’ve identified industrial-scale distillation attacks on our models by DeepSeek, Moonshot AI, and MiniMax." 🚨
Tools
3.1K 674 2 months ago
r/LocalLLaMA · u/HeadAcanthisitt...
Hot
I feel personally attacked
Tools
3.0K 151 2 months ago