Speaking AI's Language - How Computers Read Text
Computers only understand numbers. So how do they read "Hello, world"?
It's like translating English to Morse code, but more complex. Let me show you exactly how AI converts your words into numbers it can understand.
🔢The Translation Problem
Every word, every letter, every space gets converted to numbers. This process is called tokenization.
Tokenization: Breaking Words into Pieces
AI doesn't always see whole words. It breaks text into "tokens" - like syllables:
Why Not Just Use Letters?
You might wonder: why not just use A=1, B=2, etc.?
The Problem with Letters
The Token Solution
The Dictionary: AI's Vocabulary
Every AI model has a vocabulary - typically 30,000 to 50,000 tokens:
Token Dictionary Example
Real Example: How ChatGPT Sees Your Message
You type:
ChatGPT sees:
The Power of Context: Word Embeddings
Here's where it gets interesting. Each token isn't just a number - it's a location in "meaning space":
Imagine a Map of Words
Close together (similar meanings):
- • "King" and "Queen" are neighbors
- • "Happy" and "Joyful" are near each other
- • "Cat" and "Kitten" are close
Far apart (different meanings):
- • "King" and "Bicycle" are distant
- • "Happy" and "Thermometer" are far
- • "Cat" and "Mathematics" are separated
The Famous Word Math
This actually works in AI:
How? Because words are stored as coordinates in space!
Languages: Same Concepts, Different Tokens
But in meaning space, they all occupy similar positions! This is why AI can translate between languages.
Special Tokens: AI's Stage Directions
AI uses special tokens like stage directions in a play:
The Tokenization Challenge: Different Languages
English is relatively easy. Other languages are harder:
| Language | Word | Token Count |
|---|---|---|
| English | "Hello" | 1 token |
| Chinese | "你好" | 2-3 tokens |
| Arabic | "مرحبا" | 3-4 tokens |
| Emoji | "😀" | 1 token |
| Code | "function()" | 2-3 tokens |
This is why AI sometimes struggles with non-English languages - they need more tokens for the same meaning!
Hands-On: Play with Tokenization
Try This Tokenizer Experiment:
1. Go to: OpenAI Tokenizer (free online tool)
Search for "OpenAI Tokenizer" to find the tool
2. Type: "Hello world"
See: 2 tokens
3. Type: "Supercalifragilisticexpialidocious"
See: Gets broken into chunks
4. Type: "你好世界" (Hello world in Chinese)
See: More tokens than English
5. Type: Some emojis
See: Each emoji is usually 1 token
Interesting Discoveries:
- • Spaces matter! "Hello" vs " Hello" are different tokens
- • Capital letters often create new tokens
- • Common words = fewer tokens
- • Rare words = broken into pieces
Why This Matters for You
Understanding tokens helps you:
Write better prompts
Shorter tokens = faster responses
Understand costs
APIs charge per token
Debug weird behavior
Sometimes AI splits words oddly
Optimize performance
Fewer tokens = better performance
🎓 Key Takeaways
- ✓Computers only understand numbers - text must be converted to tokens
- ✓Tokenization breaks words into pieces - not always full words
- ✓Embeddings create meaning space - similar words are close together
- ✓Word math actually works - King - Man + Woman = Queen
- ✓Languages use different token counts - English is more efficient
- ✓Tokens matter for cost and performance - understand them to optimize
Ready to Understand Neural Networks?
In Chapter 6, discover how neural networks work through simple Lego block analogies. Learn the AI brain architecture!
Continue to Chapter 6