Hermes agent video analysis is set up in three ways, and this walkthrough builds all of them: the core video_analyze tool, the bundled youtube-content skill for YouTube links, and the video-xray workbench plugin for full reports.
I'd set up the core tool first, because it takes five minutes and tells you quickly whether your model stack can see video at all.
The YouTube skill covers links where the transcript is all you need, and the plugin adds frame sampling, transcripts, timestamps and exports on top.
Want my team to look at your video workflow before you build it? Book a free SEO strategy session.
Step 1: Know Which Route Answers Which Question
The core tool answers one question about one clip.
The YouTube skill turns a video's captions into written content, without looking at a single frame.
You give it a URL or a file path and a question, and it returns a written answer.
The plugin runs a job: it samples frames across a window, observes them in segments, transcribes speech and writes a report you can export.
Pick the tool by the output you need, not by habit.
| Setting | video_analyze | youtube-content skill | video-xray |
|---|---|---|---|
| Ships with Hermes | Yes, in the opt-in video toolset | Yes, bundled skill | No, community catalog plugin |
| Model used | Auxiliary video-capable model | Your main chat model reads the transcript | Your current Desktop chat model |
| Inputs | video_url plus question | A YouTube link or video ID | Attached file, time window, granularity |
| Outputs | One answer | Summary, chapters, thread, blog post, quotes | Report .md, prompt .txt, SRT and JSON cues |
| Where it runs | Any Hermes install that resolves a vision client | Linux, macOS and Windows | Hermes Desktop, verified on Windows |
Step 2: Turn On The Video Toolset
The tools reference lists video_analyze under its own toolset and marks it opt-in.
That means a fresh hermes-cli session will not have it.
Two ways to switch it on, both from the docs:
hermes --toolsets video
Or add video to the toolsets: list in your config.yaml so every session gets it.
The toolsets reference describes the set as video analysis and understanding tools, added explicitly.
Keep it separate in your head from video_gen, which makes clips rather than reading them.
Step 3: Point Hermes At A Video-Capable Model
This is the gotcha that costs people an hour.
Images can travel straight to a vision-capable main model, but the vision tools source says video_analyze keeps an auxiliary-only gate and has no native path.
Its tool description says it sends the video to a video-capable model, with Gemini as the example.
The handler reads the model from auxiliary.video.model first, then auxiliary.vision.model, then the legacy AUXILIARY_VIDEO_MODEL and AUXILIARY_VISION_MODEL variables.
The provider still comes from the vision side; a request for a fully separate video backend, issue 27313, is still open.
The provider side lives under auxiliary.vision in the configuration docs, with keys for provider, model, base URL, API key and timeouts.
auxiliary:
vision:
provider: "openrouter"
model: "google/gemini-2.5-flash"
That model name is the one the Hermes error message suggests when a model rejects video, so it is a safe first test.
If no auxiliary vision client resolves, the tool does not register at all, which is why it can look "missing" after Step 2.
🔥 Want the whole Hermes stack wired before you test video? Inside the AI Profit Boardroom you get Hermes already set up inside the Agent OS, a video walkthrough for each step, four live coaching calls every week and 3,400+ members swapping configs. → Join the AI Profit Boardroom
Step 4: Run Your First video_analyze Call
Start with a short local file so you rule out download problems.
Use video_analyze on ~/Videos/demo.mp4.
Question: list every piece of on-screen text with a rough timestamp,
then tell me when the product first appears.
Hermes wraps your question in its own instruction to describe visual content, motion, audio cues, text overlays and scene transitions first.
So ask for the specific thing you want at the end, and the description arrives for free.
Formats accepted by the source: mp4, webm, mov, avi, mkv and mpeg.
Anything over 20 MB logs a warning that it may be slow, and the payload cap sits at about 50 MB.
Step 5: Handle URLs, Blocklists And Private Hosts
A public HTTP or HTTPS link works as the video source, and Hermes downloads it to its cache before sending it on.
Two guards apply to that download.
- The website blocklist in the security docs covers
video_analyzeURLs, so a blocked domain is refused before any fetch. - With
allow_private_urlsleft false, media downloads aimed at private addresses such aslocalhostare refused.
Video on a NAS or an internal server? Copy it to a local path the agent can read and pass that path instead.
Step 6: Install video-xray From The Catalog Entry
The 15 November 2026 canary notes list "Catalog: add video-xray (video analysis workbench)" from @LuuMoss.
The catalog entry pins version 0.2.2, requires Hermes 0.22.1 or newer, sets the tier to community and lists Windows as the only verified platform.
The plugins docs say hermes plugins install <name> installs a catalog entry at its reviewed, pinned commit, and general plugins stay off until enabled.
The plugin's own README documents a manual three-step install, because it also ships a desktop panel:
- Copy
plugin.yaml,__init__.py,engine/anddashboard/into%LOCALAPPDATA%/hermes/plugins/video-xray/. - Run
hermes plugins enable video-xray, then fully quit and reopen Hermes Desktop. - Copy
desktop/plugin.jsinto%LOCALAPPDATA%/hermes/desktop-plugins/video-xray/and pressCtrl+Shift+R.
Before any of that, put ffmpeg on your PATH; the README suggests winget install Gyan.FFmpeg.
You know it worked when a gold Video Analysis Workbench strip shows above the composer, and /video opens it.
Step 7: Choose A Transcription Engine
All engines are optional, and the plugin runs without any of them.
- Whisper (local): the README's recommended start. Make a venv, install
faster-whisper, then register that venv's Python in the panel. - Cloud ASR: any OpenAI-compatible transcription endpoint. Fill base URL, model and key, then hit Test connection.
- R2T2 (local): the best local pick for Chinese, with a roughly 3.5 GB download and a fixed folder layout.
- Off: frames only.
Saving an engine performs a real load on your machine, so a green light means it actually started.
From chat, an auto ladder picks whichever engine you installed.
📞 Rather have the video pipeline mapped for your business? Book a free SEO strategy session and my team will walk through which footage is worth analysing and where the reports should go. → Grab a free strategy call
Tuning A video-xray Run
Three dials decide speed, cost and detail.
- Time window: analyse a slice instead of the whole file when you only care about one section.
- Granularity: 8, 16, 32 or 64 frames per observation segment. Cost scales linearly, so start low.
- Shot-cut threshold: adjustable from 0.05 to 0.95, used when the reverse prompt splits at real cuts.
Through the agent tools there is also an adaptive sampling mode that adds frames around detected scene changes.
Six built-in presets cover common jobs, and a custom option lets you write your own observation prompt.
Cancel is graceful, and Resume carries on from the interrupted segment without redoing finished ones.
Gotchas I'd Check On Day One
- Auto-analysis is on. The catalog security note says every attached video starts a job when you press send. Untick "Analyze attached videos on send" if you only want manual runs.
- Frames go to your chat model. A cloud chat model means frames go to that provider. Use a local model for private footage.
- Engine not ready. The README fix is to send one normal chat message first so the model is up.
- transcript=no. No speech in that window, which is normal for music-only clips.
- Cloud key storage. A cloud ASR key you save is written unencrypted to a file only your user account can read, inside Hermes' data directory.
- Whisper first run. faster-whisper downloads weights from Hugging Face the first time.
Worked Example: A Long Webinar Recording
Here's how I'd split a long recording between the two routes.
A full webinar file is usually far past the 50 MB cap, so video_analyze is the wrong first tool for it.
- Attach the file in Hermes Desktop with auto-analysis switched off.
- Open the workbench with
/videoand set a time window covering the first section only. - Pick Whisper as the engine and 16 frames per segment.
- Run the job, then open the frame gallery to check the sampled frames actually cover the slides.
- If coverage looks thin, raise granularity for that window only, rather than for the whole file.
- Repeat for the next section, and use Resume if you had to cancel midway.
- Export the SRT for captions and the Markdown report for the written recap.
For a short promo clip cut from that webinar, go back to video_analyze and ask one sharp question, such as which frame shows the offer on screen.
That split keeps the heavy, segment-by-segment work in the plugin and the quick checks in the core tool.
It also keeps model calls predictable, since the workbench only spends frames on the windows you chose.
Using The Outputs
A finished job exports a Markdown report and a reversed prompt as a text file, each with a copy button.
The SRT and JSON cue files import straight into CapCut, Premiere or any player.
The reversed prompt targets MiniMax-H3; delete the formatting lines the README names and the shot descriptions work for other generators.
Planning to make new clips from those prompts? That is the generation side, covered in the Hermes Agent AI video generation guide.
Where Gateway Chats Fit
Video does not have to start on the desktop.
The Telegram docs say a video posted in an observed group without a trigger is cached with its saved path.
When someone later asks about it, the agent opens the file with video_analyze if the video toolset is on, and with its file tools if not.
The same canary also made a Telegram comment and the video sent right after it count as one turn, so a caption and its clip arrive together.
YouTube Links: The youtube-content Skill
For a YouTube link you rarely need frames at all, and Hermes ships a bundled skill for exactly that.
The youtube-content skill page describes it as YouTube transcripts to summaries, threads and blogs.
Every installed skill doubles as a slash command, so this is the whole call:
/youtube-content summarize https://youtu.be/VIDEO_ID
The skill runs a helper script through the terminal tool, which needs the youtube extra prepared in the Hermes environment first, as the skill's setup section explains.
It accepts full links, youtu.be short links, Shorts, embeds, live links or a bare 11-character video ID.
Output formats include chapters, a summary, chapter summaries, an X thread, a blog post and timestamped quotes; with no format named, it defaults to a summary.
Transcripts over about 50K characters are split into overlapping chunks and summarised before merging.
The catch: it reads captions, not pictures, so a video with transcripts disabled returns nothing useful, and on-screen text is invisible to it.
My Verdict On This Setup
Build Step 2 and Step 3 today, because they cost nothing and unlock video questions everywhere Hermes runs.
Reach for /youtube-content whenever the video is on YouTube and the spoken words are what you need.
Add video-xray only on a Windows machine running Desktop, and only once you need transcripts, timestamps and reports you will reuse.
Switch off auto-analysis on day one, keep private footage on a local model, and start every run at 8 or 16 frames per segment.
If you want Hermes pre-installed with the rest of the agent stack, the AI Profit Boardroom has the Agent OS and the trainings, or book a free SEO strategy session with my team.
Also On Our Network
- 🌐 A plain-English setup of Hermes video analysis for a shop, a service firm or an agency
- 🌐 The three ways to analyze video in Hermes, compared in one table
- 🌐 A step-by-step setup of Hermes video analysis inside an Agent OS profile
- 🌐 How to set up Hermes video analysis, with a short note on the model behind it
- 🌐 The complete Hermes Agent video analysis guide on AI Profit Boardroom
FAQ: Hermes Agent Video Analysis
How do I switch on video analysis in Hermes Agent?
Add the opt-in video toolset with --toolsets video or list video under toolsets in config.yaml, then make sure an auxiliary vision provider is configured.
Which setting picks the model that watches the video?
video_analyze reads auxiliary.video.model, then auxiliary.vision.model, then the legacy env variables, and uses the auxiliary vision provider. It never uses a native path on the main model.
What file types does video_analyze accept?
mp4, webm, mov, avi, mkv and mpeg. Files over 20 MB may be slow, and the payload cap is about 50 MB.
Is the video-xray plugin made by Nous Research?
No. Its catalog entry is community tier and the maintainer is LuuMoss. Nous lists it in the catalog with a pinned commit.
Why does every video I attach start a job?
That is the plugin default. Untick Analyze attached videos on send in the workbench panel to run jobs by hand.
How do I summarise a YouTube video with Hermes?
Run /youtube-content summarize followed by the link. The bundled skill fetches the transcript and turns it into a summary, chapters, a thread or a blog post.
📺 Video notes + links to the tools 👉
🎥 Learn how I make these videos 👉
🆓 Get a FREE AI Course + Community + 1,000 AI Agents 👉
About Julian
I'm Julian Goldie, an SEO entrepreneur, author and founder of the AI Profit Boardroom, which has 3,400+ members.
I help business owners scale with AI agents, automation and SEO.
- I've built a 7-figure SEO and link building agency, Goldie Agency, with a team of 70+ SEO experts.
- I've grown a YouTube channel to 400,000+ subscribers.
- I wrote "Link Building Mastery".
- My Udemy courses have taught over 29,000 students.
→ Get my best AI training inside the AI Profit Boardroom
Enable the toolset, set the auxiliary model, test one clip, and if you want help beyond that, join the AI Profit Boardroom or book a free SEO strategy session about your hermes agent video analysis.











