Field notes · AI agents, Hermes & Claude Code By Julian Goldie

Hermes Agent Video Analysis: How To Set Up video_analyze, The YouTube Skill And video-xray, Step By Step (2026)

Turn on the opt-in toolset, point it at a video-capable model, summarise YouTube links with the bundled skill, then add the community workbench for full reports. Exact steps and gotchas.

Get The AI Profit Stack Join AIPB →
  • 01 1,000+ done-for-you AI agent workflows
  • 02 5 live coaching calls / week with me
  • 03 7-day refund, cancel anytime
  • 04 3,400+ AI operators inside

Hermes agent video analysis is set up in three ways, and this walkthrough builds all of them: the core video_analyze tool, the bundled youtube-content skill for YouTube links, and the video-xray workbench plugin for full reports.

I'd set up the core tool first, because it takes five minutes and tells you quickly whether your model stack can see video at all.

The YouTube skill covers links where the transcript is all you need, and the plugin adds frame sampling, transcripts, timestamps and exports on top.

Want my team to look at your video workflow before you build it? Book a free SEO strategy session.

Step 1: Know Which Route Answers Which Question

The core tool answers one question about one clip.

The YouTube skill turns a video's captions into written content, without looking at a single frame.

You give it a URL or a file path and a question, and it returns a written answer.

The plugin runs a job: it samples frames across a window, observes them in segments, transcribes speech and writes a report you can export.

Pick the tool by the output you need, not by habit.

Settingvideo_analyzeyoutube-content skillvideo-xray
Ships with HermesYes, in the opt-in video toolsetYes, bundled skillNo, community catalog plugin
Model usedAuxiliary video-capable modelYour main chat model reads the transcriptYour current Desktop chat model
Inputsvideo_url plus questionA YouTube link or video IDAttached file, time window, granularity
OutputsOne answerSummary, chapters, thread, blog post, quotesReport .md, prompt .txt, SRT and JSON cues
Where it runsAny Hermes install that resolves a vision clientLinux, macOS and WindowsHermes Desktop, verified on Windows

Step 2: Turn On The Video Toolset

The tools reference lists video_analyze under its own toolset and marks it opt-in.

That means a fresh hermes-cli session will not have it.

Two ways to switch it on, both from the docs:

hermes --toolsets video

Or add video to the toolsets: list in your config.yaml so every session gets it.

The toolsets reference describes the set as video analysis and understanding tools, added explicitly.

Keep it separate in your head from video_gen, which makes clips rather than reading them.

Step 3: Point Hermes At A Video-Capable Model

This is the gotcha that costs people an hour.

Images can travel straight to a vision-capable main model, but the vision tools source says video_analyze keeps an auxiliary-only gate and has no native path.

Its tool description says it sends the video to a video-capable model, with Gemini as the example.

The handler reads the model from auxiliary.video.model first, then auxiliary.vision.model, then the legacy AUXILIARY_VIDEO_MODEL and AUXILIARY_VISION_MODEL variables.

The provider still comes from the vision side; a request for a fully separate video backend, issue 27313, is still open.

The provider side lives under auxiliary.vision in the configuration docs, with keys for provider, model, base URL, API key and timeouts.

auxiliary:
  vision:
    provider: "openrouter"
    model: "google/gemini-2.5-flash"

That model name is the one the Hermes error message suggests when a model rejects video, so it is a safe first test.

If no auxiliary vision client resolves, the tool does not register at all, which is why it can look "missing" after Step 2.

🔥 Want the whole Hermes stack wired before you test video? Inside the AI Profit Boardroom you get Hermes already set up inside the Agent OS, a video walkthrough for each step, four live coaching calls every week and 3,400+ members swapping configs. → Join the AI Profit Boardroom

Step 4: Run Your First video_analyze Call

Start with a short local file so you rule out download problems.

Use video_analyze on ~/Videos/demo.mp4.
Question: list every piece of on-screen text with a rough timestamp,
then tell me when the product first appears.

Hermes wraps your question in its own instruction to describe visual content, motion, audio cues, text overlays and scene transitions first.

So ask for the specific thing you want at the end, and the description arrives for free.

Formats accepted by the source: mp4, webm, mov, avi, mkv and mpeg.

Anything over 20 MB logs a warning that it may be slow, and the payload cap sits at about 50 MB.

Step 5: Handle URLs, Blocklists And Private Hosts

A public HTTP or HTTPS link works as the video source, and Hermes downloads it to its cache before sending it on.

Two guards apply to that download.

  • The website blocklist in the security docs covers video_analyze URLs, so a blocked domain is refused before any fetch.
  • With allow_private_urls left false, media downloads aimed at private addresses such as localhost are refused.

Video on a NAS or an internal server? Copy it to a local path the agent can read and pass that path instead.

Step 6: Install video-xray From The Catalog Entry

The 15 November 2026 canary notes list "Catalog: add video-xray (video analysis workbench)" from @LuuMoss.

The catalog entry pins version 0.2.2, requires Hermes 0.22.1 or newer, sets the tier to community and lists Windows as the only verified platform.

The plugins docs say hermes plugins install <name> installs a catalog entry at its reviewed, pinned commit, and general plugins stay off until enabled.

The plugin's own README documents a manual three-step install, because it also ships a desktop panel:

  1. Copy plugin.yaml, __init__.py, engine/ and dashboard/ into %LOCALAPPDATA%/hermes/plugins/video-xray/.
  2. Run hermes plugins enable video-xray, then fully quit and reopen Hermes Desktop.
  3. Copy desktop/plugin.js into %LOCALAPPDATA%/hermes/desktop-plugins/video-xray/ and press Ctrl+Shift+R.

Before any of that, put ffmpeg on your PATH; the README suggests winget install Gyan.FFmpeg.

You know it worked when a gold Video Analysis Workbench strip shows above the composer, and /video opens it.

Step 7: Choose A Transcription Engine

All engines are optional, and the plugin runs without any of them.

  • Whisper (local): the README's recommended start. Make a venv, install faster-whisper, then register that venv's Python in the panel.
  • Cloud ASR: any OpenAI-compatible transcription endpoint. Fill base URL, model and key, then hit Test connection.
  • R2T2 (local): the best local pick for Chinese, with a roughly 3.5 GB download and a fixed folder layout.
  • Off: frames only.

Saving an engine performs a real load on your machine, so a green light means it actually started.

From chat, an auto ladder picks whichever engine you installed.

📞 Rather have the video pipeline mapped for your business? Book a free SEO strategy session and my team will walk through which footage is worth analysing and where the reports should go. → Grab a free strategy call

Tuning A video-xray Run

Three dials decide speed, cost and detail.

  • Time window: analyse a slice instead of the whole file when you only care about one section.
  • Granularity: 8, 16, 32 or 64 frames per observation segment. Cost scales linearly, so start low.
  • Shot-cut threshold: adjustable from 0.05 to 0.95, used when the reverse prompt splits at real cuts.

Through the agent tools there is also an adaptive sampling mode that adds frames around detected scene changes.

Six built-in presets cover common jobs, and a custom option lets you write your own observation prompt.

Cancel is graceful, and Resume carries on from the interrupted segment without redoing finished ones.

Gotchas I'd Check On Day One

  • Auto-analysis is on. The catalog security note says every attached video starts a job when you press send. Untick "Analyze attached videos on send" if you only want manual runs.
  • Frames go to your chat model. A cloud chat model means frames go to that provider. Use a local model for private footage.
  • Engine not ready. The README fix is to send one normal chat message first so the model is up.
  • transcript=no. No speech in that window, which is normal for music-only clips.
  • Cloud key storage. A cloud ASR key you save is written unencrypted to a file only your user account can read, inside Hermes' data directory.
  • Whisper first run. faster-whisper downloads weights from Hugging Face the first time.

Worked Example: A Long Webinar Recording

Here's how I'd split a long recording between the two routes.

A full webinar file is usually far past the 50 MB cap, so video_analyze is the wrong first tool for it.

  1. Attach the file in Hermes Desktop with auto-analysis switched off.
  2. Open the workbench with /video and set a time window covering the first section only.
  3. Pick Whisper as the engine and 16 frames per segment.
  4. Run the job, then open the frame gallery to check the sampled frames actually cover the slides.
  5. If coverage looks thin, raise granularity for that window only, rather than for the whole file.
  6. Repeat for the next section, and use Resume if you had to cancel midway.
  7. Export the SRT for captions and the Markdown report for the written recap.

For a short promo clip cut from that webinar, go back to video_analyze and ask one sharp question, such as which frame shows the offer on screen.

That split keeps the heavy, segment-by-segment work in the plugin and the quick checks in the core tool.

It also keeps model calls predictable, since the workbench only spends frames on the windows you chose.

Using The Outputs

A finished job exports a Markdown report and a reversed prompt as a text file, each with a copy button.

The SRT and JSON cue files import straight into CapCut, Premiere or any player.

The reversed prompt targets MiniMax-H3; delete the formatting lines the README names and the shot descriptions work for other generators.

Planning to make new clips from those prompts? That is the generation side, covered in the Hermes Agent AI video generation guide.

Where Gateway Chats Fit

Video does not have to start on the desktop.

The Telegram docs say a video posted in an observed group without a trigger is cached with its saved path.

When someone later asks about it, the agent opens the file with video_analyze if the video toolset is on, and with its file tools if not.

The same canary also made a Telegram comment and the video sent right after it count as one turn, so a caption and its clip arrive together.

YouTube Links: The youtube-content Skill

For a YouTube link you rarely need frames at all, and Hermes ships a bundled skill for exactly that.

The youtube-content skill page describes it as YouTube transcripts to summaries, threads and blogs.

Every installed skill doubles as a slash command, so this is the whole call:

/youtube-content summarize https://youtu.be/VIDEO_ID

The skill runs a helper script through the terminal tool, which needs the youtube extra prepared in the Hermes environment first, as the skill's setup section explains.

It accepts full links, youtu.be short links, Shorts, embeds, live links or a bare 11-character video ID.

Output formats include chapters, a summary, chapter summaries, an X thread, a blog post and timestamped quotes; with no format named, it defaults to a summary.

Transcripts over about 50K characters are split into overlapping chunks and summarised before merging.

The catch: it reads captions, not pictures, so a video with transcripts disabled returns nothing useful, and on-screen text is invisible to it.

My Verdict On This Setup

Build Step 2 and Step 3 today, because they cost nothing and unlock video questions everywhere Hermes runs.

Reach for /youtube-content whenever the video is on YouTube and the spoken words are what you need.

Add video-xray only on a Windows machine running Desktop, and only once you need transcripts, timestamps and reports you will reuse.

Switch off auto-analysis on day one, keep private footage on a local model, and start every run at 8 or 16 frames per segment.

If you want Hermes pre-installed with the rest of the agent stack, the AI Profit Boardroom has the Agent OS and the trainings, or book a free SEO strategy session with my team.

Also On Our Network

FAQ: Hermes Agent Video Analysis

How do I switch on video analysis in Hermes Agent?

Add the opt-in video toolset with --toolsets video or list video under toolsets in config.yaml, then make sure an auxiliary vision provider is configured.

Which setting picks the model that watches the video?

video_analyze reads auxiliary.video.model, then auxiliary.vision.model, then the legacy env variables, and uses the auxiliary vision provider. It never uses a native path on the main model.

What file types does video_analyze accept?

mp4, webm, mov, avi, mkv and mpeg. Files over 20 MB may be slow, and the payload cap is about 50 MB.

Is the video-xray plugin made by Nous Research?

No. Its catalog entry is community tier and the maintainer is LuuMoss. Nous lists it in the catalog with a pinned commit.

Why does every video I attach start a job?

That is the plugin default. Untick Analyze attached videos on send in the workbench panel to run jobs by hand.

How do I summarise a YouTube video with Hermes?

Run /youtube-content summarize followed by the link. The bundled skill fetches the transcript and turns it into a summary, chapters, a thread or a blog post.

📺 Video notes + links to the tools 👉

🎥 Learn how I make these videos 👉

🆓 Get a FREE AI Course + Community + 1,000 AI Agents 👉

About Julian

I'm Julian Goldie, an SEO entrepreneur, author and founder of the AI Profit Boardroom, which has 3,400+ members.

I help business owners scale with AI agents, automation and SEO.

  • I've built a 7-figure SEO and link building agency, Goldie Agency, with a team of 70+ SEO experts.
  • I've grown a YouTube channel to 400,000+ subscribers.
  • I wrote "Link Building Mastery".
  • My Udemy courses have taught over 29,000 students.

→ Get my best AI training inside the AI Profit Boardroom

Enable the toolset, set the auxiliary model, test one clip, and if you want help beyond that, join the AI Profit Boardroom or book a free SEO strategy session about your hermes agent video analysis.

§ Field reports

Real wins from inside the AI Profit Boardroom

See all 3,400+ members →
AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot AIPB member win screenshot

Scroll sideways →

⚑ Next step

Ready to run AI agents that actually do the work?

Join 3,400+ operators inside the AI Profit Boardroom. Get 1,000+ plug-and-play AI agent workflows, 5 coaching calls a week, and a community that holds you accountable.

Join The AI Agent Community →

7-day no-questions refund · Cancel anytime

Found this useful? Get more of this site in your Google.

One tap adds us as a preferred source — Google then shows you our posts more often in Search, Top Stories, and AI Overviews.

← Back to all posts