Internet2EgoExo

Turn internet-scale video into egocentric and exocentric training data.

Search the open web for first-person and third-person footage, confirm the
viewpoint against real frames before paying for anything, then let agents on the
Memories.ai video datalake curate and annotate whatever
survives.

Demo

One real run: a plain-language ask, a live search, the frame check that rejects
the wrong viewpoint before anything is paid for, the cost ledger, and a finished
clip with its task and action tree.

Watch the full 92-second run (1280x720). The search in it
really took 71 seconds, and that stretch is labelled as sped up on screen rather
than passed off as real time.

What it does

You state a goal the way you would say it out loud, not as a set of filters:

find me first-person cooking footage, hands must be visible, two hours of it

and the loop runs:

What comes out is a directory of clips with annotations, a manifest, and a bill
for what an hour of it cost.

What it is strict about

No hands, no clip. First-person footage without hands in it is just ordinary
footage. This is the one rule with no override.

Never mix hour counts. worn, delivered, accepted and
accepted_labeled are four different numbers. Only accepted_labeled is safe
to quote outward.

Unmeasured is not passed. A quality check that could not be measured is
dropped from the score rather than assumed to have passed.

Where to look

What you get

How the agent decides what to do

Your sentence becomes slots. An agent loop then picks tools until it has enough,
and reports what it found.

Slots. A Gemini model reads the request and pulls out what is actually being
asked for. It runs through OpenRouter by default, or Google directly; see
The models.

# "first-person cooking footage from YouTube, at least 5 minutes, 2 hours of it"
ParsedQuery(
    platforms=["youtube"],
    topics=["cooking"],
    viewpoint=Viewpoint.EGOCENTRIC,
    min_duration_seconds=300,
    target_hours=2.0,
)

The loop. Up to ten steps by default, set with MAX_AGENT_STEPS. Results are
filtered inside the loop, before the model sees them, so it never reasons over
candidates that already fail the request.

flowchart TD
    A["Request, parsed into slots"] --> B{"Model picks a tool"}
    B -->|"search, fetch, classify"| C["Tool registry runs it,
with retry and fallback"]
    C --> D["Results filtered against the slots
before the model sees them"]
    D --> B
    B -->|"enough, or step cap reached"| E["Candidates, viewpoint verdicts,
manifest, cost"]

The answer. An AgentResponse: the written answer, video_references
with their metadata and viewpoint verdicts, a dataset summary, usage_metrics
for tokens and cost, and the parsed_query it actually worked from.

For what happens to a candidate after it is found, read
the pipeline.

Installation

# Clone the repository
git clone https://github.com/Memories-ai-labs/Internet2EgoExo.git
cd Internet2EgoExo

# Install dependencies
pip install -e .

# Or with uv
uv pip install -e .

Configuration

Create a .env file with your API keys:

# Required
GOOGLE_API_KEY=your_google_api_key
YOUTUBE_API_KEY=your_youtube_data_api_key
MEMORIES_API_KEY=sk-mai-your_datalake_api_key
MEMORIES_BASE_URL=https://api.memories.ai/serve/datalake/v1

# Optional
MEMORIES_COLLECTION_ID=            # index into an existing collection
MEMORIES_COLLECTION_NAME=video-searching-agent
MEMORIES_INDEX_FPS=1.0
MEMORIES_INDEX_WAIT_SECONDS=120    # how long one call waits for indexing
EXA_API_KEY=your_exa_api_key
APIFY_API_TOKEN=your_apify_api_token   # also how YouTube videos get downloaded
VIEWPOINT_CHECK=frames                 # off | frames | watch — see PRE-SIGHT

Getting API Keys

1. Google API Key (required): Get your Gemini API key at Google AI Studio

2. YouTube Data API Key (required):

   • Go to Google Cloud Console
   • Create a project and enable YouTube Data API v3
   • Create an API key

3. Memories.ai API Key (required): Create a Video Datalake key (sk-mai-...) in the Console → API keys. Indexing is billed per minute of video — see pricing

4. Exa.ai API Key (optional): Sign up at exa.ai for neural web search

5. Apify API Token (recommended): Sign up at apify.com. It scrapes TikTok, Instagram and X — and it is how YouTube videos are downloaded, which matters more than it sounds:

   yt-dlp no longer works against YouTube from a datacentre address. It fails
   with "Sign in to confirm you're not a bot", which is not a transient
   error — it is YouTube declining to serve any IP range that looks like a
   server, and every host this pipeline realistically runs on (a container, a
   CI runner, a serverless function) sits in one of those ranges. So for
   YouTube:

   • metadata comes from the YouTube Data API, which is free, answers in
     one round trip, and states outright whether a video is Creative Commons
     (status.license) — a Gate 0 fact that the extractor was always vague
     about;
   • the file comes from an Apify downloader actor, which fetches it from a
     residential address into Apify's key-value store, from where this process
     streams it to disk on the way to the Datalake.

   The other platforms are fetched the same way, through the scrapers the
   search already calls, with their download flag on.

   There is no extractor path at all any more. yt-dlp was the download
   route for everything except YouTube, and it does not work: TikTok fails with
   "Unexpected response from webpage request" on every attempt, so a run
   surfaced good TikTok candidates, charged for the search, and then reported
   failed for reasons that said nothing about the footage. Fetching is now
   routed per platform across providers:

   DOWNLOAD_PROVIDERS	Credentials	State
   apify (default)	APIFY_API_TOKEN	Verified against the live API for YouTube, TikTok, Instagram and X
   rapidapi	RAPIDAPI_KEY, RAPIDAPI_HOST	Written to the documented API, never executed
   brightdata	BRIGHTDATA_TOKEN, BRIGHTDATA_ZONE	Written to the documented API, never executed
   oxylabs	OXYLABS_USERNAME, OXYLABS_PASSWORD	Written to the documented API, never executed

   Set it to an ordered list — DOWNLOAD_PROVIDERS=apify,brightdata — and each
   is tried until one hands back a media URL. A provider with no credentials
   reports itself unavailable and is never asked, so an unverified one cannot
   quietly become the provider a run depends on. When all of them refuse, the
   error names every attempt rather than one extractor.

Quick Start

import asyncio
from video_searching_agent import VideoSearchingAgent

async def main():
    agent = VideoSearchingAgent()

    response = await agent.query(
        "find me first-person cooking footage, hands must be visible"
    )
    print(response.answer)

    # Every candidate carries its viewpoint verdict and the cues behind it.
    for ref in response.video_references:
        print(f"- {ref.title}: {ref.url}")

asyncio.run(main())

The pipeline, end to end

One request — "find me first-person cooking footage, hands must be visible" —
runs the whole way through. Nothing below is a separate product: it is one path,
and each leg is owned by an agent whose job is narrow enough to be judged on its
own output.

                      ┌──────────────────────── you ────────────────────────┐
                      │  "all first-person cooking videos, hands visible"   │
                      └───────────────────────────┬─────────────────────────┘
                                                  ▼
 1  SEARCH        search agent        YouTube · TikTok · Instagram · X · web
                                     viewpoint classification, usability ranking
                                                  ▼
 2  SCREEN        cleaning agent      licence · length · viewpoint     ← spends nothing
                                                  ▼
 3  LOOK          cleaning agent      a few real frames, not the words about them
                                     wrong viewpoint stops here, for ~$0.002
                                                  ▼
 4  DOWNLOAD      provider routing    platform pages are not fetchable media
                                                  ▼
 5  INDEX         Video Datalake      upload → captions · transcription · embeddings
                                                  ▼
 6  CLEAN         cleaning agent      hands · other people · editing · resolution
                                     then: where does each action start and stop
                                                  ▼
 7  ANNOTATE      annotation agent    task → action → event, on time anchors
                                                  ▼
 8  CURATE        curation agent      hours ledger · diversity · duplicates · grade
                                                  ▼
                      ┌──────────────────────────────────────────────────────┐
                      │  clean clips, every one with hands, each with a tree  │
                      └──────────────────────────────────────────────────────┘

Every verdict is written back onto the video as tags, so each pass narrows the
next one: clean_pass is the annotation agent's worklist, first_person_view
and hands_visible are how the next query finds only footage that already
qualified.

The agents

They are separate on purpose. Filtering is a judgement about footage, annotation
is a judgement about language, and curation is a judgement about a set — mixing
them into one prompt produces an agent that is mediocre at all three and whose
mistakes cannot be attributed.

Looking, rather than reading about it

Every judgement here began as a statement about caption wording: whether hands
are in frame, whether the camera is worn, whether a span is one action. Caption
text is a real signal and a lossy one, and the failures were the kind you only
find by looking — a vertical phone clip of somebody using a washing machine was
rejected on a hand density of 43% computed from words like "pours" and "places",
when what actually disqualified it (9:16, with a burned-in watermark) was sitting
in the pixels.

So two of the agents run as ReAct loops with eyes. agent/eyes.py cuts a
span through the Datalake, pulls frames out of the file and hands them back as
images; agent/react_loop.py is the think-act-observe runtime they share. On each
turn the agent either calls a tool — look at frames, re-read a caption window —
or answers. Every step lands in the trace, so the reasoning is inspectable rather
than a black box that emitted a label.

Three bounds keep it honest and affordable:

• steps — a loop that has not converged in a handful of turns will not;
• money — a cut is $0.005, tools report what they spend, and the loop stops
  when the budget is gone rather than when it feels finished;
• the answer contract — running out of steps is not an answer. It returns
  nothing and the caller abstains.

The clipping agent (agent/clipping_agent.py) takes the caption walk's
proposal — which has never seen the footage — and corrects it where the frames
disagree. On a real IKEA wardrobe build it merged two proposed spans, moved four
boundaries, and gave reasons the captions do not contain ("inserting and
hammering metal threaded inserts"). It may not invent a span covering footage it
never examined, boundaries are clamped to the video, siblings are separated so
G2-TREE-2 holds, and a span it could not see stays exactly as proposed. When
the loop does not converge the proposal stands — the honest outcome, not an
empty one.

The annotating agent (agent/annotating_agent.py) labels one span, and looks
when the captions do not settle what it is being asked. That recovers the field
that used to be lost most often. A hand is still never invented — but "I saw it"
is now a way of knowing, and each hand field records which:

Both of those came back null before, as "hand assignment not stated in the
captions". A hand claimed from frames when no look happened is discarded in
code, not merely discouraged in the prompt.

Auditing what came out

The first four agents each make a judgement and then act on it. The
quality-check agent makes none of its own about footage: it audits their
output, adversarially, and it is allowed to fail a clip that all of them
accepted.

That independence is the point. A pipeline that grades its own homework will
tell you a dataset is fine in exactly the cases where its own reasoning was
wrong. Twenty videos in and fifty clips out is only a result if the fifty clips
would survive somebody reading them, so the audit asks a narrower and harsher
question of the finished artefact: if someone handed me this dataset, would I
accept it?

The evidence check is asked as a refutation — the model is told to find the
reason the label is wrong — because a model asked to confirm a label will
confirm it. Silence is not contradiction: a window with no captions is a
warning, never a failure, and captions vaguer than the annotation still support
it. A finding with no evidence behind it is a bug in the audit, not a finding.

Spans are sampled across a clip rather than taken from the front, because
later anchors are the ones most likely to have drifted.

Running the whole thing on real task queries

python qa/run_pipeline.py                  # five task queries, end to end
python qa/run_pipeline.py --query laundry   # one of them
python qa/run_pipeline.py --dry-run         # search and check candidates only, free

qa/run_qa.py proves the path works. This proves the output is worth
having: it runs laundry, kitchen, bike repair, packing and furniture assembly
all the way through — search → collect → index → clean → annotate → curate —
and then hands the delivered clips to the audit. It costs real money (a download
through Apify, a minute of indexing and a caption pass per clip), so it is a
separate script rather than part of the recurring sweep, and its exit code is
non-zero when the audit rejects a set.

Two things it found on its first real run, both fixed here:

• A clip whose indexing outran the request budget came back neither accepted
  nor rejected, with no reason given — so the run reported "nothing survived
  collection: []", an empty list of reasons, for work that had in fact reached
  the Datalake. That outcome is now its own stage, pending, which says what
  happened and hands back the video_id to curate later.
• Two twelve-minute videos do not fit in one 300-second serverless request, so
  the stream is cut before the summary event arrives. The per-clip events are
  the reliable record; the summary is a convenience that may never be sent.

Agentic filtering

Filtering runs twice, at the two points where it is cheapest.

Before the download, from platform metadata alone — nothing is fetched and
nothing is indexed, so a candidate that cannot qualify costs nothing:

Everything above except PRE-SIGHT reads words about a video — a title, a
description, tags. That is how a run can come back with clips that are licence
clear and completely off topic: a video called "POV cooking" is very often a
tripod pointed at a worktop, and no field in the metadata says so.

PRE-SIGHT looks at the video instead. It runs twice, and the first time is
the one that matters most: right after the search, on the candidates about to be
offered. A search for first-person cooking otherwise returns things like "10
Camera Angles and Shots for Cooking Videos" — nothing in that title says the
camera is on a tripod, so queueing four clips means watching all four get
skipped later. Checking at search time means the wrong ones are never offered,
and the manifest records them under frames show exocentric footage rather than
pretending they were never found. The second pass is the pre-download one below,
which catches anything queued by hand. Two tiers,
and the price gap between them is why there are two:

frames is the default because it is 140× cheaper for the same verdict on the
question being asked, and on the four-video check in tests/ it got all four
right — including abstaining on the one whose stills are title cards. watch is
better judgement and worth it for a handful of finalists; it is never the
default, because twenty candidates a query would be $5 a search.

The arithmetic is what makes this worth running on every candidate rather than
on finalists. Measured on this deployment: the look costs $0.002; the
download it might prevent costs $0.09–0.12 in Apify credits, plus a minute
of Datalake indexing, plus the caption reads the cleaning and annotation passes
make on top. One rejection pays for fifty looks.

The look can only ever stop a download, and only on a confident opposite
reading. Abstention passes through, a weak reading passes through, and a failed
call passes through with a note saying it did not run — the post-index caption
evidence sees all of the footage rather than three moments of it, and keeps the
last word.

After indexing, from the derived content — this is the pass that matters,
because it judges the footage rather than the description of it:

A clip with no hands is dropped. That is the point of the run, and it is the one
rule with no override: first-person footage without hands in it is just ordinary
video.

Two honesty rules hold everywhere:

• Nothing is scored on a number that was not measured. A check that cannot be
  computed from the available inputs reports measured: false and is excluded
  from the score, rather than being assumed to pass. G3-DUP (overlap with
  public corpora) needs embeddings this repo does not compute, so it reports
  unmeasured — always.
• The evidence is caption wording, not a detector. Every hand verdict
  carries the caveat "read from caption wording, not a hand-tracking or pose
  model", so nothing downstream can mistake it for detector output.

Agentic clipping

Caption segments carry timestamps, so a run of consecutive segments with work
happening in them is an action and a run of idle segments is not. The cleaning
agent merges, splits and drops:

• consecutive hand-present segments merge into one action;
• a gap longer than two seconds splits rather than merges;
• an idle segment breaks the run and is excluded;
• a span under two seconds is dropped as noise;
• the task span covers the actions that survived — usually less than the whole
  video, because the intro and the outro are not the task.

The output is a tree of time anchors on the original video — [start, end]
pairs — and never cut files. That is G2-TREE-5 in the standard and it is not
negotiable: cut clips lose the context on either side of a boundary, and a
boundary that turns out to be wrong can no longer be moved.

Agentic annotation

The annotation agent narrates what happens between the boundaries, at three
depths:

task     replace-inner-tube      "A punctured tube is swapped for a new one."
  action   lever-tyre-off        "Two levers walk the bead off the rim."
    event    lever-slips         "The second lever slips out of the bead."

Depth is the grade:

The structural rules are checked, not trusted:

And one rule the prompt enforces because the data is worthless without it:
hand assignment is never invented. If the captions do not say which hand did
what, left_hand and right_hand stay null. A guessed left/right is worse than
a blank one, because downstream it cannot be told apart from a real one.

Tags are written back in the form hoi/// — for
example hoi/chop-vegetables/right/move-knife — which is what makes the next
query able to ask for exactly that.

There are two ways in. annotate_video narrates anchors the cleaning agent
already found. run is the discovery loop the Datalake is built for:
search_moments shortlists spans → get_moment says what is really in them →
the model judges → update_video writes the verdict back. The share of the
shortlist that survives the read is reported as the run's survival rate;
retrieval proposes, the span decides.

Agentic data curation

Curation is what can only be judged across a set.

Four hour measures, never mixed. Reporting a delivered hour as an accepted
hour overstates a dataset by 30-40%:

media_yield is accepted / delivered, and it is reported rather than assumed.

Diversity, across the set:

The scorecard, 100 points: annotation 45, diversity 25, media 20, licensing
10 → A ≥85 (main training set, sellable), B 70-84 (main set, not
external), C 55-69 (pretrain / diversity supplement), D The rate limit does not survive serverless. The token bucket lives in

process memory, and on a platform like Vercel each request can land on a fresh
instance — so on that host the limit is effectively not enforced. What actually
bounds spend there is MAX_COLLECT_URLS (how many clips one request may
queue — indexing is billed per video-minute), a spend cap set on the keys
themselves at OpenRouter and memories.ai, and API_KEYS if you are willing to
require one. A real shared limit needs shared state (Vercel KV, Upstash), which
this repo does not assume you have.

Vercel

vercel.json and api/index.py are in the repo, and api/requirements.txt
holds a deliberately smaller dependency set than the project's own (no
playwright, no scraper SDKs — they are not importable in a serverless bundle and
are not needed to serve the API or the UI).

npm i -g vercel
vercel            # first deploy, links the project
vercel --prod

Then set environment variables in the Vercel dashboard (or vercel env add):

What does not fit serverless. A function has a request timeout — 60s on
Hobby, up to 300s on Pro — and indexing a long video takes minutes. The
collection stream handles that the way it was designed to: it reports
indexing still running with the video_id to come back with, rather than
hanging. For bulk collection, run the app on a host without a request timeout
(a VM, Render, Railway, Fly) and point the UI at it. Downloading is also
happier there: platform hosts rate-limit datacenter IPs, and serverless disk is
capped at 512 MB in /tmp.

The models: one key, or Google's

Either provider drives the agents, and the choice is a setting rather than a
rewrite:

LLM_PROVIDER forces a choice (openrouter / gemini); the default, auto,
prefers OpenRouter when its key is present. Set OPENROUTER_MODEL to use a
different model.

Streaming API

POST /api/v1/queries/stream runs a query and streams Server-Sent Events
(started, progress, tool_call, tool_result, clarification_needed,
complete, error):

curl -N http://localhost:8000/api/v1/queries/stream \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $API_KEY" \
  -d '{
        "query": "Top latte art videos from the past week",
        "sources": ["youtube", "tiktok"]
      }'

Pinned sources replace whatever platforms the query parser inferred, and the agent is told
to search only those.

POST /api/v1/collect/stream

Download candidates, index them into the Video Datalake, clean them and annotate
what survives. Events: started, clip_stage (one per stage, per clip),
clip_done, complete, error.

POST /api/v1/curate/stream

Clean, annotate and grade a worklist that is already indexed. Events: started,
clip_done, complete (hours ledger, Gate 3 checks, batch grade), error.

One of video_ids or tag is required.

Reading one video's content

Ask about a video's own content and the agent indexes it into the Video Datalake,
then reads back the captions, transcription and summary:

response = await agent.query(
    "What does this video actually show and say? https://example.com/clip.mp4"
)

Indexing takes time. If it is still running when the call's wait budget expires,
the tool reports status: "processing" with a video_id, and the next call reads
the results instead of re-indexing.

Response Structure

AgentResponse:
    session_id: str           # Unique session identifier
    query: str                # Original query
    answer: str               # Natural language answer
    video_references: list    # List of VideoReference objects
    platforms_searched: list  # Platforms that were searched
    total_videos_analyzed: int
    steps_taken: int          # Agent loop iterations
    tools_used: list          # Tools that were called
    execution_time_seconds: float

    # Extended fields
    usage_metrics: UsageMetrics   # Detailed cost tracking
    parsed_query: ParsedQuery     # Extracted slots from query
    tool_execution_details: list  # Success/failure for each tool call
    confidence_score: float       # Answer confidence (0-1)
    needs_clarification: bool     # Whether clarification is needed
    clarification_question: str   # Question to ask user if needed

UsageMetrics Structure

UsageMetrics:
    gemini: GeminiCost            # Gemini API costs
        token_usage: TokenUsage   # input_tokens, output_tokens, total_tokens
        input_cost_usd: float
        output_cost_usd: float
        total_cost_usd: float
    tool_costs: list[ToolUsageCost]  # Per-tool cost breakdown
    total_cost_usd: float         # Combined Gemini + tools cost
    gemini_calls: int             # Number of Gemini API calls
    tool_calls: int               # Total tool invocations

Architecture

VideoSearchingAgent
    ├── GeminiClient (Google Gemini API)
    ├── QueryParser (LLM-first slot extraction)
    ├── ClarificationManager (handles missing context)
    ├── RetryExecutor (retry with exponential backoff + fallbacks)
    ├── ToolRegistry
    │   ├── YouTube: YouTubeSearchTool, YouTubeChannelTool
    │   ├── Exa: ExaSearchTool, ExaSimilarTool, ExaContentTool, ExaResearchTool
    │   ├── TikTok (Apify): TikTokSearchTool, TikTokCreatorTool
    │   ├── Instagram (Apify): InstagramSearchTool, InstagramCreatorTool
    │   ├── Twitter (Apify): TwitterSearchTool, TwitterProfileTool
    │   ├── Video Datalake: VideoIndexTool, VideoAnalysisTool, VideoMomentSearchTool
    │   └── Unified: VideoSearchTool
    └── AgentSession (tracks query lifecycle)

Tools Reference

The agent has access to 16 specialized tools organized by category:

YouTube Tools (2)

Exa.ai Tools (4)

Apify Social Media Tools (6)

Memories.ai Video Datalake Tools (3)

The Datalake is the agent's long-term video memory: a video indexed once stays
searchable, so later questions read the lake instead of re-processing the video.

Cost control: indexing is billed per minute of video, so the tool policy blocks
these tools unless the query actually names a video or asks what is inside one —
a broad discovery query can never trigger paid indexing.

Unified Tools (1)

Query Slots

The agent extracts structured slots from natural language queries using LLM-first parsing. These slots control search behavior:

Platform Slots

Entity Slots

Metric Slots

Time Frame Slots

Quantity Slots

Data Models

Core Entities

Video:
    platform: Platform        # youtube, tiktok, instagram, twitter
    platform_id: str          # ID on source platform
    url: HttpUrl              # Direct video URL
    title: str | None
    creator: Creator | None
    metrics: VideoMetrics | None
    published_at: datetime | None
    hashtags: list[str]

Creator:
    username: str
    platform: Platform
    followers: int | None
    verified: bool
    total_videos: int | None

VideoMetrics:
    views: int | None
    likes: int | None
    comments: int | None
    shares: int | None
    engagement_rate: float | None  # Platform-specific calculation

Query Models

ParsedQuery:
    original_query: str
    query_type: QueryType     # industry_topic, brand_analysis, creator_profile, etc.
    platforms: list[str]
    topics: list[str]
    creators: list[str]
    metric: MetricType        # most_popular, highest_engagement, etc.
    time_frame: TimeFrame     # past_week, past_month, etc.
    quantity: int             # 1-100
    needs_clarification: bool

AgentSession:
    session_id: str
    user_query: str
    parsed_query: ParsedQuery | None
    current_step: int         # Curre