AI Coding Assistant Training Data Extraction Toolkit

Complete toolkit to extract ALL chat, agent, and code context data from AI coding assistants for machine learning training.

🎯 What This Does

Automatically discovers and extracts complete conversation history including:

• ✅ User messages & AI responses
• ✅ Code context (file paths, line numbers, snippets)
• ✅ Code diffs and suggested edits
• ✅ Multi-file contexts
• ✅ Tool use and execution results
• ✅ Timestamps and metadata

📦 Included Scripts

1. extract_claude_code.py

Extracts from Claude Code / Claude Desktop

• Searches: ~/.claude, ~/.claude-code, ~/.claude-local, ~/.claude-m2, ~/.claude-zai
• Formats: JSONL session files
• Includes: Messages, tool use, file contexts, diffs

2. extract_codex.py

Extracts from Codex (if installed)

• Searches: ~/.codex, ~/.codex-local
• Formats: Rollout JSONL files
• Includes: User/agent messages, tool results, diffs

3. extract_cursor.py

Extracts from Cursor (Chat + Composer + Agent) - ALL VERSIONS

• Searches: ~/Library/Application Support/Cursor (macOS) or equivalent
• Formats: SQLite databases (state.vscdb, cursorDiskKV)
• Handles:
  • Old Chat mode (workspace storage)
  • Composer inline storage (v1.x - messages in composerData array)
  • Composer separate storage (v1.x-v2.0 transition - messages in bubbleId keys)
  • Latest Composer/Agent (v2.0+)
• Includes:
  • Code context, selections, diffs
  • Suggested edits and code blocks
  • Tool results and execution outputs

4. extract_trae.py

Extracts from Trae

• Searches: ~/.trae, ~/Library/Application Support/Trae
• Formats: JSONL and SQLite databases
• Includes: Chat, agent data, tool use, diffs

5. extract_windsurf.py

Extracts from Windsurf

• Searches: ~/Library/Application Support/Windsurf or equivalent
• Formats: SQLite databases (VSCode-like format)
• Includes: Chat, agent/flow conversations, code context

6. extract_continue.py

Extracts from Continue AI Assistant

• Searches: ~/.continue/sessions/
• Formats: JSON session files
• Includes:
  • User/assistant messages
  • Tool calls and results
  • Reasoning blocks
  • Context items
  • Workspace information

7. extract_gemini.py

Extracts from Google Gemini CLI

• Searches: ~/.gemini/tmp/[hash]/chats/
• Formats: JSON session files
• Includes:
  • User/assistant messages
  • Thoughts (reasoning steps with timestamps)
  • Token usage breakdown
  • Model information
  • Project hash and workspace linking

8. extract_opencode.py

Extracts from OpenCode (CLI + Desktop)

• Searches:
  • CLI: ~/.local/share/opencode/storage/ (Linux), ~/Library/Application Support/opencode (macOS)
  • Desktop: ~/.local/share/ai.opencode.app (Linux), ~/Library/Application Support/ai.opencode.app (macOS)
• Formats: JSON files (sessions/messages/parts) and Tauri .dat files (desktop)
• Includes:
  • User/assistant messages with full conversation hierarchy
  • Tool calls and tool results
  • Code blocks and text content
  • Token usage and cost tracking
  • Model and provider information
  • Agent mode and session metadata
  • Project directory and version info
  • Parent/child session relationships

🚀 Quick Start

Installation

# No dependencies required - uses Python 3 standard library
python3 --version  # Ensure Python 3.6+ is installed

Basic Usage

# Extract from Claude Code
python3 extract_claude_code.py

# Extract from Cursor
python3 extract_cursor.py

# Extract from Codex
python3 extract_codex.py

# Extract from Trae
python3 extract_trae.py

# Extract from Windsurf
python3 extract_windsurf.py

# Extract from Continue
python3 extract_continue.py

# Extract from Gemini CLI
python3 extract_gemini.py

# Extract from OpenCode
python3 extract_opencode.py

# Extract from ALL tools at once
./extract_all.sh

Output

All scripts create an extracted_data/ directory with timestamped JSONL files:

extracted_data/
├── claude_code_conversations_20250116_143022.jsonl
├── cursor_complete_20250116_143045.jsonl
├── gemini_conversations_20250116_143145.jsonl
├── codex_conversations_20250116_143102.jsonl
├── trae_conversations_20250116_143115.jsonl
├── windsurf_conversations_20250116_143130.jsonl
├── continue_conversations_20250116_143145.jsonl
└── opencode_conversations_20250116_143200.jsonl

📊 Output Format

Each conversation is a single JSON line in JSONL format:

{
  "messages": [
    {
      "role": "user",
      "content": "How do I fix this TypeScript error?",
      "code_context": [
        {
          "file": "/Users/user/project/src/index.ts",
          "code": "const x: string = 123;",
          "range": {
            "selectionStartLineNumber": 10,
            "positionLineNumber": 10
          }
        }
      ],
      "timestamp": "2025-01-16T14:30:22.123Z"
    },
    {
      "role": "assistant",
      "content": "The error occurs because you're assigning a number to a string type...",
      "suggested_diffs": [...],
      "model": "claude-sonnet-4-5",
      "timestamp": "2025-01-16T14:30:25.456Z"
    }
  ],
  "source": "cursor-composer",
  "name": "TypeScript Type Error Fix",
  "created_at": 1705414222000
}

🔍 How It Works

Auto-Discovery Process

Each script follows this pattern:

1. Detect Operating System (macOS, Linux, Windows)
2. Search Common Locations:
   • macOS: ~/Library/Application Support, ~/.config, ~/
   • Linux: ~/.config, ~/.local/share, ~/
   • Windows: %APPDATA%, %LOCALAPPDATA%, ~/
3. Find All Installations of the target tool
4. Scan Storage Locations:
   • SQLite databases (.vscdb, .db)
   • JSONL session files
   • Project-specific directories
5. Extract Complete Data including context and diffs
6. Save to Organized JSONL with timestamps

Storage Formats Handled

Claude Code / Codex

• Format: JSONL files (one event per line)
• Location: ~/.claude/projects/[project]/[session].jsonl
• Structure: Event-based with type markers

Cursor (v0.43 - v2.0+)

• Format: SQLite databases
• Locations:
  • Workspace: ~/Library/Application Support/Cursor/User/workspaceStorage/[hash]/state.vscdb
  • Global: ~/Library/Application Support/Cursor/User/globalStorage/state.vscdb
• Tables: ItemTable (Chat), cursorDiskKV (Composer/Agent)
• Storage Evolution:
  • v0.x - v1.x: Chat mode in workspace ItemTable
  • v1.x: Composer inline (messages in composerData.conversation[])
  • v1.x - v2.0 transition: Composer separate (messages in bubbleId:{composer}:{bubble} keys)
  • v2.0+: Latest format with enhanced metadata
• Keys:
  • workbench.panel.aichat.view.aichat.chatdata (Chat mode)
  • composerData:{uuid} (Composer metadata + conversation)
  • bubbleId:{composer}:{bubble} (Individual messages - transitional format)
  • codeBlockDiff:{id} (Code block diffs)

Trae / Windsurf

• Format: Hybrid (JSONL + SQLite)
• Location: Similar to VSCode/Cursor structure
• Structure: VSCode extension data format

🎓 Understanding the Data

Message Roles

• user: Human developer messages
• assistant: AI assistant responses

Code Context Fields

• code_context: File selections and code snippets
• suggested_diffs: AI-proposed code changes
• tool_use: Code execution, file operations
• tool_results: Execution outputs, diffs applied
• diff_histories: Full edit history

Metadata Fields

• source: Which tool (e.g., "cursor-composer", "claude-code")
• session_id/composer_id: Unique conversation ID
• project_path: Working directory
• timestamp: Message time
• model: AI model used (if available)

🔧 Advanced Usage

Merge All Extractions

# Combine all JSONL files
cat extracted_data/*.jsonl > all_conversations.jsonl

# Count total conversations
wc -l all_conversations.jsonl

# Count by source
grep -o '"source":"[^"]*"' all_conversations.jsonl | sort | uniq -c

Filter by Date

import json
from datetime import datetime

with open('extracted_data/cursor_complete_20250116.jsonl') as f:
    for line in f:
        conv = json.loads(line)
        created = conv.get('created_at', 0)
        if created > 1704067200000:  # After Jan 1, 2024
            print(json.dumps(conv))

Extract Only Conversations with Diffs

import json

with open('extracted_data/cursor_complete.jsonl') as f:
    for line in f:
        conv = json.loads(line)
        if any('suggested_diffs' in m or 'diff_histories' in m
               for m in conv['messages']):
            print(json.dumps(conv))

📋 Data Quality

What Gets Extracted

✅ Complete Conversations:

• Both user prompts AND AI responses
• Multi-turn dialogues
• Full conversation context

✅ Code Context:

• File paths and names
• Selected code snippets
• Line number ranges
• Multi-file selections

✅ Diffs and Edits:

• Suggested code changes
• Applied diffs
• Edit histories
• File modifications

✅ Metadata:

• Timestamps
• Project paths
• Model information
• Conversation names

What Might Be Missing

⚠️ Partial Data:

• Conversations without AI responses (user-only)
• Deleted or archived sessions
• Corrupted database entries

⚠️ Privacy Considerations:

• May include proprietary code
• May include API keys/secrets
• May include personal file paths

🛡️ Privacy & Security

Before Using Extracted Data

1. Scan for Secrets:

   pip install detect-secrets
   detect-secrets scan extracted_data/*.jsonl

2. Review Sensitive Data:

• Check for API keys, passwords, tokens
• Verify no proprietary code exposed
• Sanitize file paths if needed

3. Storage:

• Keep on encrypted drives
• Don't commit to public repositories
• Secure backups recommended

🎯 Training Use Cases

Direct Fine-Tuning

from datasets import load_dataset

dataset = load_dataset(
    'json',
    data_files='extracted_data/*.jsonl',
    split='train'
)

# Filter complete conversations
dataset = dataset.filter(
    lambda x: any(m['role'] == 'assistant' for m in x['messages'])
)

With Unsloth

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "unsloth/qwen2.5-coder-7b-instruct",
    max_seq_length=4096,
    load_in_4bit=True,
)

def format_chat(example):
    return {
        'text': tokenizer.apply_chat_template(
            example['messages'],
            tokenize=False
        )
    }

dataset = dataset.map(format_chat)

🐛 Troubleshooting

No installations found

Problem: Script reports "No installations found"

Solutions:

1. Check if the tool is actually installed
2. Verify installation location manually
3. Add custom path to script:

   # Add to find_XXX_installations() function
   locations.append(Path("/custom/path/to/tool"))

Empty extracted_data directory

Problem: Extraction completes but no data found

Solutions:

1. Verify you've actually used the tool and have chat history
2. Check if data is in a non-standard location
3. Look for database files manually:

   find ~ -name "*.vscdb" -o -name "*.db" 2>/dev/null

Database locked errors

Problem: SQLite database is locked

Solutions:

1. Close the AI tool before running extraction
2. Use read-only mode:

   conn = sqlite3.connect(f'file:{db_path}?mode=ro', uri=True)

Permission denied

Problem: Cannot read certain files

Solutions:

1. Run with appropriate permissions
2. Check file ownership
3. Copy databases to accessible location first

📚 Platform-Specific Notes

macOS

• Uses ~/Library/Application Support for most tools
• May need Full Disk Access for system directories
• SQLite databases typically in ~/Library/Application Support/[Tool]/User/

Linux

• Uses ~/.config and ~/.local/share
• Check ~/.local/state for some tools
• May use $XDG_CONFIG_HOME if set

Windows

• Uses %APPDATA% and %LOCALAPPDATA%
• Paths: C:\Users\[User]\AppData\Roaming\[Tool]
• May need admin privileges for Program Files

🔄 Version Compatibility

Cursor

• ✅ v2 (0.43+): Composer/Agent in cursorDiskKV
• ✅ v1: Chat in workspace ItemTable
• ⚠️ Pre-v0.43: Different format, limited support

Claude Code

• ✅ All versions with JSONL session files
• ✅ Project-based structure

Codex

• ✅ Rollout JSONL format
• ✅ Time-based session organization

📈 Performance Tips

Large Datasets

# Process in chunks
split -l 1000 all_conversations.jsonl chunk_

# Compress for storage
gzip extracted_data/*.jsonl

Speed Optimization

# Use multiprocessing for large scans
from multiprocessing import Pool

with Pool() as pool:
    results = pool.map(extract_from_db, db_files)

🤝 Contributing

Found a new storage format or tool? Contributions welcome!

1. Follow existing script structure
2. Add auto-discovery logic
3. Extract complete data (messages + context + diffs)
4. Output to organized JSONL
5. Update this README

📄 License

MIT License - Use freely for training ML models

⚠️ Disclaimer

This toolkit extracts YOUR OWN data from locally installed AI tools. Users are responsible for:

• Ensuring they have rights to extracted data
• Handling sensitive/proprietary information appropriately
• Complying with tool Terms of Service
• Scanning for secrets before sharing/training

Generated: January 16, 2025
Status: Production Ready
Compatibility: Python 3.6+, macOS/Linux/Windows