exo-cuda

🎯 What This Fork Adds

The original exo focuses on Apple Silicon (MLX). This fork restores full NVIDIA CUDA support via tinygrad:

⚡ Quick Start

# Clone this repo
git clone https://github.com/Scottcjn/exo-cuda.git
cd exo-cuda

# Create venv and install
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

# Upgrade tinygrad to latest (fixes CUDA issues)
pip install --upgrade git+https://github.com/tinygrad/tinygrad.git

# Start with CUDA backend
exo --inference-engine tinygrad --chatgpt-api-port 8001 --disable-tui

📋 Requirements

Install CUDA Toolkit

# Ubuntu/Debian
sudo apt install nvidia-cuda-toolkit

# Verify
nvcc --version
nvidia-smi

✅ Verified Hardware

Tested December 2024 - January 2025:

🖥️ Multi-Node Cluster

Node 1 (Primary + API)

exo --inference-engine tinygrad --chatgpt-api-port 8001 --disable-tui

Node 2+ (Workers)

exo --inference-engine tinygrad --disable-tui

That's it! Nodes auto-discover via UDP broadcast. No manual configuration.

Manual Peer Configuration (Optional)

# Create peers.json
echo '{"peers": ["192.168.1.100:5678", "192.168.1.101:5678"]}' > peers.json

# Start with manual discovery
exo --inference-engine tinygrad --discovery-module manual \
    --discovery-config-path peers.json

Minimal Smoke Test (NVIDIA path)

Run this exact sequence to confirm CUDA inference is actually healthy:

nvidia-smi
nvcc --version
python3 -c "from tinygrad import Device; print(Device.DEFAULT)"
exo --inference-engine tinygrad --chatgpt-api-port 8001 --disable-tui

In another terminal:

curl -sS http://localhost:8001/v1/models
curl -sS http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-3.2-1b","messages":[{"role":"user","content":"ping"}],"max_tokens":16}'

If this works, your CUDA stack is ready for multi-node expansion.

🔌 API Usage

exo provides a ChatGPT-compatible API:

# Chat completion
curl http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.2-3b",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'

# List models
curl http://localhost:8001/v1/models

Supported Models

All tinygrad-compatible models work:

🔧 Environment Variables

# Debug logging (0-9, higher = more verbose)
DEBUG=2 exo --inference-engine tinygrad

# Tinygrad-specific debug (1-6)
TINYGRAD_DEBUG=2 exo --inference-engine tinygrad

# Limit GPU visibility
CUDA_VISIBLE_DEVICES=0,1 exo --inference-engine tinygrad

🐛 Troubleshooting

Common Fixes

# Fix tinygrad CUDA issues
pip install --upgrade git+https://github.com/tinygrad/tinygrad.git

# Verify CUDA is working
python3 -c "from tinygrad import Device; print(Device.DEFAULT)"
# Should print: CUDA

# Test GPU memory
nvidia-smi --query-gpu=memory.free --format=csv

📊 Performance Tips

1. Use SXM2 GPUs - NVLink provides faster inter-GPU communication
2. Match GPU types - Heterogeneous clusters work but homogeneous is faster
3. 10GbE+ networking - For multi-node clusters, network is the bottleneck
4. Disable TUI - --disable-tui reduces overhead

🔗 Related Projects

🙏 Credits

• Original exo by exo labs
• tinygrad for the CUDA backend
• NVIDIA for CUDA toolkit

📜 License

GPL-3.0 (same as original exo)

Troubleshooting FAQ

Installation Issues

Q: Build fails with CUDA not found
A: Ensure CUDA toolkit is installed and nvcc is in your PATH:

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

Q: CMake can't find CUDA
A: Set CUDA_HOME explicitly:

export CUDA_HOME=/usr/local/cuda
cmake -DCUDA_TOOLKIT_ROOT_DIR=$CUDA_HOME ..

Runtime Issues

Q: CUDA out of memory error
A: Reduce batch size or model size, or use a GPU with more VRAM.

Q: Slow performance
A: Check GPU utilization with nvidia-smi. Ensure you're using the GPU, not CPU fallback.

Q: Driver version mismatch
A: Update NVIDIA drivers to match your CUDA version:

nvidia-smi  # Check current version
# Update drivers as needed

Common Errors

Q: libcudart.so not found
A: Add CUDA lib to library path:

sudo ldconfig /usr/local/cuda/lib64

Q: Compilation warnings about deprecated APIs
A: These are usually safe to ignore, but consider updating to newer CUDA APIs.

For more help, open an issue with your error message and system details.

Part of the Elyan Labs Ecosystem

• RustChain — Proof-of-Antiquity blockchain rewarding vintage hardware
• BoTTube — AI video platform where 119+ agents create content
• GitHub