Query Profile

Standardized query profiling for health AI benchmarks, analogous to clinical trial reporting for evaluation datasets.

The Problem

Clinical trials require transparent inclusion criteria for generalizability. Benchmarks validating health LLMs rarely characterize the query populations they contain.

The validity of extrapolating benchmark performance to real-world use depends on prompt composition. Omitting this is like publishing trial results without describing who was enrolled.

Without understanding dataset composition, aggregate metrics misrepresent model readiness for clinical use.

The Solution

Query Profile applies a standardized taxonomy to health queries. See our paper for methodology and findings.

Compositional profiles reveal:

• Intents tested (symptom checking, education, triage, chronic management)
• Topics covered and medical domains missing
• Populations represented (age groups, vulnerable populations)
• Context provided (lab values, medications, wearable data, raw clinical text)
• Risk levels present (routine wellness vs. safety-critical scenarios)

Systematic comparison exposes gaps before they become deployment failures.

Benchmark Profiles

We analyzed 18,842 consumer health queries across six public benchmarks:

Key gaps:

• High-risk scenarios underrepresented: 6.3% test safety-critical situations (crisis intervention, drug interactions, acute emergencies)
• Vulnerable populations missing: 89.5% target "adult, unspecified"; minimal pediatric, elderly, or pregnant coverage
• Single-turn dominance: 91.8% single-turn queries; multi-turn clinical reasoning largely untested

View the full combined profile

How It Works

Input: Health queries in standard format

{"id": "q1", "messages": [{"role": "user", "content": "My 4-year-old has had a fever of 103°F for two days. Should I go to the ER?"}]}

Output: Structured tags revealing query characteristics

{
  "tags": {
    "intent_top": "symptom_check",
    "intent_sub": "triage",
    "topic_area_path": ["Pediatrics", "Fever"],
    "population": "peds_under5",
    "risk_sensitivity": "high",
    "objective_data": ["vitals_basic"],
    "context_depth": "moderate"
  }
}

Profile: Aggregate across your dataset to see composition at a glance.

Quick Start

Try the Example (No API Key Needed)

git clone https://github.com/apple/ml-health-query-profiles.git
cd ml-health-query-profiles
pip install -e .
python examples/quickstart.py

Tag Your Own Data

# 1. Set your API key
export OPENAI_API_KEY=sk-...

# 2. Prepare data (JSONL with messages array)
cat my_queries.jsonl
# {"id": "q1", "messages": [{"role": "user", "content": "What causes migraines?"}]}

# 3. Tag
python -m query_profile.taggers --input my_queries.jsonl --model gpt-4o

# 4. Generate profile
python -m query_profile.analytics --input my_queries_tagged.jsonl --format markdown

Installation

git clone https://github.com/apple/ml-health-query-profiles.git
cd ml-health-query-profiles
pip install -e .

Requirements: Python 3.10+

The Taxonomy

See docs/TAGGING_SCHEMA.md for the complete field reference.

Documentation

CLI Reference

Tagging

# Standard tagging (uses LiteLLM, works with any provider)
python -m query_profile.taggers --input queries.jsonl --model gpt-4o

# OpenAI Batch API (50% cheaper, up to 24h turnaround)
python -m query_profile.taggers --input queries.jsonl --backend openai

# Test loading without API calls
python -m query_profile.taggers --input queries.jsonl --dry-run

Analytics

# Markdown profile
python -m query_profile.analytics --input tagged.jsonl --format markdown

# JSON output
python -m query_profile.analytics --input tagged.jsonl --format json

# Filter to consumer queries only
python -m query_profile.analytics --input tagged.jsonl --consumer-only

Python API

from query_profile import LiteLLMTagger, load_jsonl, generate_profile, ProfileFormatter

# Load and tag
records = load_jsonl("queries.jsonl")
tagger = LiteLLMTagger(model="gpt-4o")
tagged = list(tagger.tag_batch(records))

# Generate profile
profile = generate_profile(tagged, "my_dataset", filters={"user_type": "consumer"})
print(ProfileFormatter.to_markdown(profile))

Repository Structure

query-profile/
├── query_profile/          # Core package
│   ├── taggers/            # LLM tagging
│   ├── analytics/          # Profile generation
│   └── data/               # Data utilities
├── prompts/                # Versioned tagging prompts
├── outputs/                # Generated profiles, tables, figures
├── docs/                   # Documentation
├── examples/               # Worked examples
└── paper/                  # Research paper reproducibility
    ├── datasets/           # Dataset preparation
    ├── analysis/           # Analysis scripts
    └── manuscript/         # LaTeX source

Paper Reproducibility

This package accompanies our research paper. The paper/ directory reproduces analysis across six public datasets:

• HealthBench (main, hard, consensus)
• HealthSearchQA
• MashQA
• MedRedQA
• GoogleFitbit (fitness, sleep)

See paper/REPRODUCIBILITY.md for instructions.

Contributing

See CONTRIBUTING.md for guidelines.

License

Apple Sample Code License

Citation

If you use this package in your research, please cite:

@article{rajkomar2026validity,
  title={The Validity Gap in Health AI Evaluation: A Cross-Sectional Analysis of Benchmark Composition},
  author={Rajkomar, Alvin and Sudarshan, Pavan and Lai, Angela and Peng, Lily},
  journal={arXiv preprint arXiv:2603.18294},
  year={2026}
}