Part 3: Creating AIChapter 7 of 12

Building Your First Dataset - My 77,000 Example Journey

20 min5,500 words289 reading now
Building Your First Dataset: The 77,000 Example Journey

In 2023, I decided to create a PostgreSQL expert AI. Not because I had to, but because I was curious: Could I make an AI that knew PostgreSQL better than most developers?

6 months later, I had built 77,175 training examples. The result? An AI that could debug PostgreSQL like a senior DBA. Let me show you exactly how I did it.

📊My Story: From Zero to 77,000 Training Examples

The Challenge

  • ✗No existing PostgreSQL-specific models
  • ✗Generic models gave incorrect SQL advice
  • ✗Documentation was scattered everywhere

The Journey

  • ✓Time invested: 6 months
  • ✓Examples created: 77,175
  • ✓Coffee consumed: ~500 cups
  • ✓Result: AI debugging like a senior DBA

What is Training Data, Really?

Training data is like a cookbook for AI:

Recipe Card Format:
Ingredients (Input) → Dish (Output)
Training Example Format:
Question/Problem (Input) → Answer/Solution (Output)

Real Examples from My Dataset

Example 1:
"input":
"My query is running slowly on PostgreSQL 15"
"output":
"First, run EXPLAIN ANALYZE on your query to see the execution plan. Check for sequential scans on large tables - these often indicate missing indexes. Also verify your statistics are current with ANALYZE."
Example 2:
"input":
"ERROR: duplicate key value violates unique constraint"
"output":
"This error occurs when trying to insert a value that already exists in a unique column. Solutions: 1) Use INSERT ... ON CONFLICT to handle duplicates, 2) Check your sequence if using serial columns, 3) Ensure your application isn't sending duplicate requests."

The Data Collection Strategy

Here's exactly how I built my dataset:

🔍

Source 1: Stack Overflow Mining

18,000 examples

What I did:

  1. 1. Scraped PostgreSQL tagged questions
  2. 2. Filtered for answered questions with 5+ upvotes
  3. 3. Cleaned and formatted Q&A pairs

Quality tricks:

  • • Only kept accepted answers
  • • Removed outdated version-specific info
  • • Combined multiple good answers into comprehensive responses
💎

Source 2: Mailing List Gold

32,385 examples

PostgreSQL mailing lists = 20+ years of expert discussions

My process:

  1. 1. Downloaded pgsql-general archives
  2. 2. Extracted problem-solution threads
  3. 3. Converted discussions to Q&A format

Why this was golden:

  • • Real production problems
  • • Solutions from PostgreSQL core developers
  • • Edge cases you won't find anywhere else
📚

Source 3: Documentation Examples

5,985 examples

Official docs are great but dense. I transformed them:

Before:
"The CREATE INDEX command builds an index on the specified column(s) of the specified table..."
After:
Q: "How do I create an index in PostgreSQL?"
A: "Use CREATE INDEX. Basic syntax: CREATE INDEX idx_name ON table_name(column_name); For better performance on large tables, use CREATE INDEX CONCURRENTLY to avoid locking."
🐛

Source 4: GitHub Issues

3,039 examples

Popular PostgreSQL projects have issue trackers full of real problems:

  1. 1. Scraped issues from pg_admin, postgres, major extensions
  2. 2. Extracted problem descriptions and solutions
  3. 3. Included error messages and stack traces

This gave me real-world debugging scenarios!

🔮

Source 5: My Secret Weapon - Synthetic Data

10,005 examples

I generated variations using templates:

# Template approach
templates = [
"How to optimize {query_type} on {table_size} table?",
"Best index for {column_type} with {operation}?",
"Solving {error_code} in PostgreSQL {version}?"
]
# Generated thousands of combinations

Quality Control: The Make or Break

Bad data = Bad AI. Here's how I ensured quality:

The Validation Process

Step 1: Automated checks

  • • No empty responses
  • • Minimum 50 characters per answer
  • • No obvious errors (like MySQL syntax in PostgreSQL context)

Step 2: Random sampling

  • • Manually reviewed 5% of examples
  • • If error rate > 2%, reviewed entire batch

Step 3: Technical accuracy

  • • Tested SQL examples in actual PostgreSQL
  • • Verified version-specific features
  • • Checked for deprecated syntax

Step 4: Diversity check

  • • Ensured mix of difficulty levels
  • • Covered all major PostgreSQL features
  • • Included both common and edge cases

The Format That Works

After much experimentation, here's the winning format:

{
  "conversations": [
    {
      "role": "system",
      "content": "You are a PostgreSQL expert database administrator with 20 years of experience."
    },
    {
      "role": "user",
      "content": "My database backup is taking 5 hours. How can I speed it up?"
    },
    {
      "role": "assistant",
      "content": "Here are several ways to speed up PostgreSQL backups:

1. Use parallel backup with pg_dump -j option:
   pg_dump -j 4 -Fd -f backup_dir dbname

2. Consider using pg_basebackup for physical backups

3. Implement incremental backups with pgBackRest or WAL-G

4. Optimize your I/O: backup to different disk than database

5. Use compression carefully - sometimes CPU becomes bottleneck"
    }
  ]
}

Common Mistakes I Made (So You Don't Have To)

Mistake 1: Too Similar Examples

Bad:
"How to create index?" → Answer
"How do I create index?" → Same answer
"Creating an index?" → Same answer
Good:
"How to create index?" → Basic answer
"Best practices for index creation?" → Detailed strategy
"Creating index without locking table?" → CONCURRENTLY focus

Mistake 2: Forgetting Context

Bad:
Q: "It's slow"
A: "Add an index"
Good:
Q: "My SELECT query with JOIN on large tables is slow"
A: "For JOIN performance, ensure indexes on join columns..."

Mistake 3: One-Size-Fits-All Answers

Bad:
Every performance question → "Run VACUUM ANALYZE"
Good:
Varied solutions based on specific symptoms
🎯

Your Turn: Start Small Dataset (100 Examples)

Pick Your Domain

Something you know well:

  • • Your job field
  • • A hobby
  • • A skill you have

Create 100 Examples Using This Framework:

20 examples: Common beginner questions
20 examples: Intermediate challenges
20 examples: Advanced problems
20 examples: Error messages and fixes
20 examples: Best practices

Format:

{ "input": "[Specific question/problem]", "output": "[Detailed, helpful answer]" }

Tools You'll Need:

  • • Spreadsheet or text editor
  • • JSON formatter (free online)
  • • Domain knowledge or research ability

The Results: Was It Worth It?

After 6 months and 77,175 examples:

The Good

  • ✓Model knew PostgreSQL inside-out
  • ✓Could debug complex issues
  • ✓Knew version-specific features and quirks
  • ✓Query optimization suggestions were spot-on

The Investment

  • Time: ~500 hours
  • Cost: ~$50 (OpenAI API for validation)
  • Learning: Priceless
  • Satisfaction: Enormous

The Outcome

  • →Model performs better than GPT-4 on PostgreSQL tasks
  • →Being used by 1000+ developers
  • →Saved countless debugging hours
  • →Proved that individuals can create specialized AI

Lessons Learned

1

Quality > Quantity

1,000 excellent examples > 10,000 mediocre ones

2

Real Data > Synthetic

But synthetic fills gaps well

3

Diversity Matters

Cover edge cases, not just common cases

4

Test Everything

Bad data compounds during training

5

Document Sources

You'll need to update/improve later

🎓 Key Takeaways

  • ✓Training data is the foundation - quality datasets make quality AI
  • ✓Multiple sources are best - Stack Overflow, mailing lists, docs, GitHub issues, synthetic data
  • ✓Quality control is critical - automate checks, manually sample, test accuracy
  • ✓Consistency matters - use a standardized format for all examples
  • ✓Start small - 100 examples is enough to begin your journey
PREVIOUS
Tokens and Context Windows - The Memory Game
NEXT CHAPTER
Training AI - Teaching Machines to Think

Ready to Learn How to Train AI?

In Chapter 8, discover pre-training vs fine-tuning, learning rates, and the complete training process with real code examples!

Continue to Chapter 8
Free Tools & Calculators