Getting Xiaomi MiMo Working with Hermes Agent: Voice, Search & Tool Use

Wed, 29 Jul 2026 00:00:00 +0000

A while back I wrote a personal Telegram bot, and at the time I thought: it’d be great if this could replace a voice assistant and hook into the smart home. But back then the effort far outweighed the payoff — LLM API prices were sky-high, and between daily conversations, tool calls, and accumulating context, running an AI agent was basically burning money. So I shelved the idea.

Then things changed. Xiaomi opened up its Home Assistant integration, agent frameworks matured, and DeepSeek drove model prices down to a level I was willing to pay. I spent some time with OpenClaw, then noticed Xiaomi had released MiMo. After checking the pricing and capabilities, I subscribed to a token plan. MiMo 2.5 is multimodal, and 2.5-pro is more than capable for non-coding work.

OpenClaw still fell short for me: its memory was unreliable between sessions, and automatically learned skills often needed several rounds of fixes. I was spending more time tinkering than saving. I switched to Hermes Agent, whose memory and skill systems suit me better. Since I had already paid for the MiMo token plan, I added ASR, TTS, and web search, then patched Hermes so that it would look things up when appropriate instead of answering from memory. These are the parts of that process worth keeping.

MiMo Web Search: A Low-Hassle Web Search Option

Hermes ships with a web_search tool backed by pluggable providers (Tavily, Firecrawl, Exa, Brave, and others). It chooses one based on the available API keys and falls back to DuckDuckGo. MiMo also offers server-side web search: a Chat Completions request makes it search, browse pages, and extract content before returning structured results. That avoids setting up a separate overseas search API.

The setup is straightforward. The call flow is:

Hermes web_search tool call
    ↓
MiMoWebSearchProvider.search(query, limit)
    ↓
POST {base_url}/chat/completions
    tools: [{type: "web_search", force_search: true, max_keyword: 3}]
    ↓
MiMo server-side search → returns annotations: [{type: "url_citation", ...}]
    ↓
Parsed into Hermes standard format: {title, url, description, position}

The plugin prefers url_citation annotations for results. The model occasionally returns structured JSON directly in the response content; the plugin parses that as a fallback.

The plugin is implemented as a Hermes WebSearchProvider; its code is at git.ntzyz.cn/mimo-websearch-for-hermes. There are two easy-to-miss details:

Prompt: To get structured results rather than free-form text, the prompt explicitly asks MiMo to follow a JSON schema:

Use the web_search tool to find current information for the query below,
then respond with ONLY a single JSON object — no prose, no markdown
fences — matching this exact schema:

{"results": [{"title": "string", "url": "string",
"description": "1-2 sentence summary"}]}

Configuration: Hermes’s config.yaml lets you adjust:

web:
  search_backend: "mimo"
  mimo:
    model: "mimo-v2.5"       # Options: mimo-v2.5-pro, mimo-v2-flash, etc.
    max_keyword: 3            # Maximum keywords per search round
    timeout: 30

max_keyword controls how many keywords can be used in a search round. More keywords may yield more complete results, but cost more. Web search itself is billed domestically at ¥16 per 1,000 calls, plus token consumption at the standard rate.

Once it was working, I tried searching for “Geekerwan latest popular video.” MiMo returned Bilibili and YouTube links, video titles, content summaries, even bullet comment reactions and chart rankings. It looks as though it really did read the pages rather than simply returning search results.

What’s more interesting is how different the answers are for the same question with and without search. Take “Is Apple Silicon’s lack of external GPU support a software limitation or a hardware architecture limitation”:

Without search (before patch, MiMo answered from memory):

It’s a hardware architecture design choice, not a software limitation. The core reason is Apple Silicon’s Unified Memory Architecture (UMA)…eGPU support was indeed removed after macOS Monterey, in sync with M-series chips.

The direction was right, but it was too absolute — outright denying the software factor.

With search (after patch, MiMo did 3 web_search calls):

Both, but mainly a software/architecture design choice, not a physical hardware impossibility.…On March 31, 2026, Apple officially approved a third-party eGPU driver developed by Tiny Corp (George Hotz’s company), allowing external AMD/Nvidia GPUs on Apple Silicon Macs via Thunderbolt/USB4. But only for compute workloads, not graphics acceleration.

After searching, MiMo picked up the March 2026 Tiny Corp third-party eGPU driver news — an event that happened after the model’s training data cutoff, impossible to know from memory alone.

MiMo ASR: Handling Voice Messages

Hermes supports voice interaction: users send voice messages in a channel, and the agent transcribes them before processing. Its STT (speech-to-text) system also uses pluggable providers, so I added MiMo ASR.

Plugin code at git.ntzyz.cn/mimo-asr-for-hermes.

MiMo ASR’s API is non-standard — it doesn’t use the usual /v1/audio/transcriptions endpoint. Instead it goes through /v1/chat/completions, stuffing the audio into messages as input_audio:

payload = {
    "model": "mimo-v2.5-asr",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
        ]
    }]
}

The audio is Base64-encoded and sent with the model and messages to /v1/chat/completions; the transcription is returned in choices[0].message.content. Only WAV and MP3 are supported directly; other formats (OGG and M4A, for example) are automatically converted to 16 kHz mono WAV with ffmpeg.

I first used it for Telegram voice messages lasting only a few dozen seconds, where neither limit came up. Video transcription was a different story: MiMo ASR has two hard limits:

10MB per request: Base64-encoded audio can’t exceed 10MB
8192 token context: Audio token density is roughly 1400 tokens/MB (@ 48kbps), so a single request handles about 5-7 minutes of audio

I mostly use it for video transcription. The audio from a 38-minute video is still well over 10MB even after compression, so it has to be split up. I ended up using overlapping segments: 5 minutes each, with a 4-minute step and 60 seconds of overlap:

# 5 min per segment, 4 min step (60s overlap)
ffmpeg -y -ss 0   -i input.mp3 -t 300 -ar 16000 -ac 1 -b:a 64k seg_00.mp3
ffmpeg -y -ss 240 -i input.mp3 -t 300 -ar 16000 -ac 1 -b:a 64k seg_01.mp3
ffmpeg -y -ss 480 -i input.mp3 -t 300 -ar 16000 -ac 1 -b:a 64k seg_02.mp3

At first I split it cleanly, which occasionally dropped words when a sentence crossed a boundary. Adding the overlap largely fixed that. After transcription, I use an LLM to deduplicate and merge the output, remove filler words, correct recognition errors, and add section headings.

I tested it on Geekerwan’s 38-minute battery life comparison video. After compressing the audio to 64kbps mono (~22MB), I split it into 10 segments, transcribed each, then merged with an LLM. MiMo ASR’s accuracy on Mandarin is quite high — technical terms (battery capacity locking, charging speed throttling, AI Agent battery testing, etc.) were all recognized correctly.

There are limitations though. Some dialects don’t work well — I tried a Nantong dialect comedy video and only got a few sentences that were close to Mandarin out of 1.7MB of audio. MiMo ASR also doesn’t support mixed input, so you can’t attach a text prompt alongside audio (like “output with timestamps”) — pure audio only. And there’s no timestamp feature, unlike Whisper which can output sentence-level or word-level timestamps, so aligning audio with video timelines isn’t really feasible.

I mainly use it for Telegram voice messages and Mandarin video now, and that is enough for the moment. It also uses the same API key and token plan, so there is no separate service to set up. I will worry about dialects and timestamps if I actually need them.

MiMo TTS: Adding Speech Output

Hermes’s text_to_speech tool also uses pluggable providers. With MiMo TTS added, the agent can reply with voice bubbles directly in a channel.

Plugin code at git.ntzyz.cn/mimo-tts-for-hermes.

MiMo TTS also goes through the chat/completions API, but the usage differs from ASR. The text to synthesize goes in an assistant message’s content, while voice and format are specified in the audio field:

{
    "model": "mimo-v2.5-tts",
    "messages": [
        {"role": "assistant", "content": "Text to synthesize"}
    ],
    "audio": {"format": "wav", "voice": "冰糖"}
}

Style is not a separate parameter. Instead, the API reuses the chat message structure: add a user message before the text to describe the desired delivery, for example, “Read this in a newscaster’s voice” or “Tell this as a story.”

MiMo TTS has 9 built-in voices — 4 Chinese, 4 English, plus 1 auto-select:

Voice	Language	Gender	Notes
`冰糖`	Chinese	F
`茉莉`	Chinese	F
`苏打`	Chinese	M
`白桦`	Chinese	M
`Mia`	English	F
`Chloe`	English	F
`Milo`	English	M
`Dean`	English	M
`mimo_default`	Auto	-	Selects based on text language

Beyond the built-in voices, there are two additional models: voice design (voicedesign) which generates custom voices from text descriptions, and voice cloning (voiceclone) which clones a voice from an audio sample.

First, a Chinese sample using the 冰糖 voice, reading Linjiangxian (a traditional tune pattern):

滚滚长江东逝水，浪花淘尽英雄。是非成败转头空，青山依旧在，几度夕阳红。

Something more casual, with the 茉莉 voice:

下午三点，大猫猫准时跳上桌子，一屁股坐在键盘上，然后用那种「你怎么还不摸我」的眼神看着我。我试图把它挪开，它就四脚朝天地躺下来，露出肚皮。行吧，今天下午的代码是写不完了。

And an English sample, with the “Dean” voice reading the Hermes Agent introduction:

Hermes Agent is a personal AI assistant that runs on your own machine. It connects to Telegram, Discord, Slack, and over twenty other platforms. It learns across sessions with memory and skills, delegates tasks to subagents, runs scheduled jobs, and drives your terminal and browser. Think of it as your own private AI butler, fully extensible through plugins and skills.

Chinese sounds more natural than I expected, and the English is listenable too—at least it does not make me want to stop it immediately. I did not measure latency carefully: non-streaming mode takes roughly 2–3 seconds for a ten-second clip, while streaming mode (stream: true, PCM16 24 kHz output) can play as it generates. In practice, Hermes only uses TTS for longer replies; short ones stay as text.

Patching Hermes: Getting It to Use Tools

MiMo already had access to web_search, so why did it initially answer from memory instead? The problem was not the plugin, but the system prompts Hermes injects for different models.

I traced this through the Hermes source. The key file is agent/system_prompt.py, which selects guidance based on the model name. One section, TOOL_USE_ENFORCEMENT_GUIDANCE, essentially says: “Use tools to take actions; do not merely describe what you would do.”

This guidance is only injected for models whose names contain these keywords:

`1`	`TOOL_USE_ENFORCEMENT_MODELS = ("gpt", "codex", "gemini", "gemma", "grok", "glm", "qwen", "deepseek")`

mimo isn’t in the list, so MiMo never gets the “use tools when you should” guidance and just answers from memory.

There’s also a stricter layer in the source called OPENAI_MODEL_EXECUTION_GUIDANCE, which contains a <mandatory_tool_use> block listing rules like “current facts, news, versions → must use web_search” — but this only gets injected for gpt/codex/grok.

There are only two changes, both under ~/.hermes/hermes-agent/agent/:

First: prompt_builder.py — add "mimo" to the end of the TOOL_USE_ENFORCEMENT_MODELS tuple:

# Before
TOOL_USE_ENFORCEMENT_MODELS = ("gpt", "codex", "gemini", "gemma", "grok", "glm", "qwen", "deepseek")

# After
TOOL_USE_ENFORCEMENT_MODELS = ("gpt", "codex", "gemini", "gemma", "grok", "glm", "qwen", "deepseek", "mimo")

This gives MiMo the instruction to use a tool when needed instead of merely describing the action.

Second: system_prompt.py — add or "mimo" in _model_lower to the OPENAI_MODEL_EXECUTION_GUIDANCE injection condition:

# Before
if "gpt" in _model_lower or "codex" in _model_lower or "grok" in _model_lower:

# After
if "gpt" in _model_lower or "codex" in _model_lower or "grok" in _model_lower or "mimo" in _model_lower:

This also gives MiMo the mandatory-tool rules for situations such as current facts and arithmetic.

After both changes, clear pycache and restart the gateway:

1
2
3

find ~/.hermes/hermes-agent/agent -name 'prompt_builder*' -path '*__pycache__*' -delete
find ~/.hermes/hermes-agent/agent -name 'system_prompt*' -path '*__pycache__*' -delete
hermes gateway restart

AI on ntzyz's space

Getting Xiaomi MiMo Working with Hermes Agent: Voice, Search & Tool Use

MiMo Web Search: A Low-Hassle Web Search Option

MiMo ASR: Handling Voice Messages

MiMo TTS: Adding Speech Output

Patching Hermes: Getting It to Use Tools