🚀 Liquid AI × Weights & Biases Tokyo Hackathon 2025

Welcome to the Liquid AI × Weights & Biases Tokyo Hackathon! This repository demonstrates how to
fine-tune Liquid AI's LFM2 with advanced experiment tracking and LM development
observability using Weights & Biases'
Models and Weave**.

🎯 What's Inside

This example showcases a complete workflow for fine-tuning LFM2 models with best-practices observability and provenance
supported by Weights & Biases:

• 🔄 Periodic Evaluation: Automatically evaluate model performance during training
• 🔍 Weave Tracing: Full observability of your evaluation process with automatic visualization
  Models](https://docs.wandb.ai/guides/models/) + Weave
• 📈 Rich Visualizations: Automatic charts, progress visualization, and metadata tracking for full model provenance
• 🎯 Multiple Metrics: Track multiple use case-specific metrics across evals; we provide simplified conversation,
  response length, coherence, and diversity scoring as examples

✨ Key Features

Custom Evaluation Callback

• WeaveEvaluationCallback: A custom training callback that runs Weave evaluations periodically
• Automatic metric logging to W&B
• Full traceability of evaluation results with Weave, all centrally stored with your run

Comprehensive Scoring System

• Conversation Scorer: Measures similarity between model output and reference text
• Improvement Scorer: Tracks performance improvements over the base model
• Response Length Scorer: Evaluates appropriate response length
• Coherence Scorer: Assesses logical flow and sentence connectivity
• Diversity Scorer: Measures vocabulary richness

Efficient Fine-tuning

• Unsloth Integration: 2x faster inference with memory-efficient training
• (optional) LoRA Adapters: Parameter-efficient fine-tuning
• (optional) 4-bit Dynamic Quantization: Reduced memory usage for accessible
  training
• Flexible Model Selection: Choose from any of our LFM2 family of models
  • LFM2 base models
  • Liquid Nanos, select
    task-specific models which can be further fine-tuned for your use case
  • LFM2-VL our vision-language model
    family

📋 Prerequisites

Before you start, make sure you have:

• Weights & Biases account
• HuggingFace account with access token
• Lambda Labs account

🛠️ Setup

1. Environment Configuration

For Local Development:

Copy the .env.example file to .env in the project root:

cp .env.example .env

And update the variables with your credentials.

HF_TOKEN=your_huggingface_token
WANDB_API_KEY=your_wandb_api_key
WANDB_ENTITY=your_wandb_entity
WANDB_PROJECT=your_project_name

🎮 Usage Workflow

Step 1: Configure Your Project

Update the following variables in the script:

# Your HuggingFace profile
HF_PROFILE_NAME = "your_username"

# Model registry collection name
WANDB_REGISTRY_COLLECTION_NAME = "your-model-name"

# Choose your base model
base_model_repo = "unsloth/LFM2-350M-unsloth-bnb-4bit"  # or 700M, 1.2B, etc. 

Step 2: Initialize W&B and Weave

The script automatically initializes both services:

wandb.login()
weave.init(f"{WANDB_ENTITY}/{WANDB_PROJECT}")

Step 3: Prepare Your Dataset

The example uses the FineTome-100k-dedup dataset, but you can use your own:

dataset_name = "your_hf/dataset_repo"
train_dataset, val_dataset, test_dataset = prepare_dataset(
    tokenizer=lfm2_ft.tokenizer,
    dataset_name=dataset_name,
    num_samples=3000
)

Step 4: Configure Training

Customize training parameters:

train_configs = TrainConfig(
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    num_train_epochs=1,
    max_steps=100,
    learning_rate=2e-4,
    # ... more parameters
)

Step 5: Create Trainer with Evaluation Callback

Set up periodic evaluation during training:

trainer, evaluation_callback = get_trainer_with_evaluation_callback(
    model=lfm2_ft.model,
    tokenizer=lfm2_ft.tokenizer,
    config=train_configs,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    weave_model=lfm2_ft,
    eval_interval="steps",  # or "epoch"
    eval_steps=20,  # evaluate every 20 steps
    max_eval_samples=50
)

Step 6: Train!

Start fine-tuning with automatic evaluation:

trainer_stats = trainer.train()

During training, you'll see:

• 📊 Periodic evaluation metrics logged to W&B
• 🔍 Weave traces for full observability
• 📈 Automatic visualization in the Weave UI

Step 7: Final Evaluation

Run comprehensive evaluation on the test set:

final_evaluation = weave.Evaluation(
    dataset=test_subset,
    scorers=all_scorers,
    evaluation_name="Final Test Evaluation"
)

final_summary, final_call = await final_evaluation.evaluate(lfm2_ft)

📊 Understanding the Results

W&B Dashboard

• Training Metrics: Loss, learning rate, GPU usage
• Evaluation Metrics: Periodic scores for all custom metrics
• Model Artifacts: Saved checkpoints and metadata

Weave UI

• Radar Charts: Multi-dimensional performance visualization
• Bar Charts: Metric comparisons across training steps
• Trace View: Detailed execution paths and timing information
• Call Logs: Complete history of model predictions and evaluations

🏗️ Project Structure

.
├── Liquid_AI_x_Weights_&_Biases_Tokyo_Hackathon_2025_*.py
│   └── Main training script with all functionality
├── README.md
│   └── This file
└── .env
    └── Environment variables (create this from .env.example)

🎓 Key Concepts

LoRA (Low-Rank Adaptation)

Efficient fine-tuning method that adds trainable rank decomposition matrices to model layers, dramatically reducing the
number of parameters to train.

Weave Evaluation

W&B's framework for tracking and visualizing ML model evaluations with automatic tracing, versioning, and comparison
capabilities.

Unsloth

Optimized library for faster LLM training and inference, providing 2x speed improvements and memory efficiency.

💡 Tips for Success

1. Start Small: Begin with the 350M model and smaller datasets to validate your workflow
2. Monitor GPU Memory: Check memory usage and adjust batch size accordingly
3. Experiment with Metrics: Customize scorer functions for your specific use case
4. Save Checkpoints: Enable checkpoint saving for long training runs
5. Use Weave UI: Regularly check the Weave interface for insights into model behavior

🔗 Resources

• Liquid AI Documentation
• Weights & Biases Documentation
• Weave Documentation
• Unsloth Documentation
• HuggingFace Transformers

🤝 Support

• Hackathon Support: Reach out to the event organizers
• Liquid AI Community: Join the Discord server and the Discord channel
  #tokyo-hack
• W&B Community: Join the Slack community
• Issues: Open an issue in this repository

📝 License

This example is provided for educational purposes as part of the Liquid AI × Weights & Biases Tokyo Hackathon 2025.

Happy Hacking! 🎉

Built with ❤️ by Liquid AI and Weights & Biases teams for the Tokyo Hackathon 2025.