iris

A voice + video AI companion that lives in your browser. Claude as the brain, Live2D as the face, real-time conversation with lip sync, face tracking, and memory. You just talk — iris sees you, hears you, and talks back.

中文

What it does

Continuous voice chat — Silero VAD listens the whole time, no push-to-talk. Start talking over iris mid-reply and she stops mid-sentence.
Live2D avatar — lip-synced to the TTS stream, expressions driven by Claude (<expr:happy> etc.), eye contact follows your face, torso moves while she speaks.
Face tracking — MediaPipe reads your expression and feeds it to Claude as natural language ([The user just shifted from neutral to smiling]), so she reacts to mood changes. Also detects when you leave or come back to the camera.
Persistent memory — editable persona + long-term facts about you. Iris can save facts herself mid-reply via <remember>…</remember> tags.
Ambient check-in — after ~90 s of silence iris gently nudges the conversation.
Multi-model — Sonnet / Opus / Haiku picker in the UI, no API key needed (uses your Claude subscription via the CLI).
Bilingual — Chinese + English, auto-detected per turn. Paraformer-zh for fast Mandarin STT, whisper.cpp for everything else.

Architecture

┌─────────────── browser ───────────────┐         ┌──────── server ────────┐
│ PIXI + pixi-live2d-display            │         │ Fastify + WebSocket    │
│ MediaPipe Face Landmarker             │ ◄─ WS ─►│ Persistent Claude CLI  │
│ Silero VAD (continuous mic)           │         │ whisper.cpp + Paraformer│
│ Web Audio TTS playback + RMS lip sync │         │ edge-tts (Azure neural) │
└───────────────────────────────────────┘         └────────────────────────┘

Brain: Claude Code CLI in persistent --input-format stream-json mode. One long-running subprocess per session handles every turn via stdin JSON — only the first reply pays the cold-spawn tax. Model picker in the UI (sonnet / opus / haiku). Uses your existing Claude subscription, no API key.
STT:
- whisper.cpp (ggml-small-q5_1.bin, ~181 MB, ~3 s/turn on CPU) — multilingual with a strict en/zh script validator that retries on language mis-IDs.
- Paraformer-zh via sherpa-onnx-node (~232 MB ONNX, ~300–500 ms/turn) — state-of-the-art Mandarin, routed automatically when 中文 is picked. In-process native addon, no Python sidecar.
TTS: edge-tts — free, needs internet. Presets pair an English neural voice with a Chinese neural voice (Xiaoxiao / Xiaoyi / HsiaoYu for 她, Yunxi / Yunyang / Yunjian / YunJhe for 他). Speaking speed and model are user-configurable from the compose bar.
Voice activity: Silero VAD via @ricky0123/vad-web in an AudioWorklet. Stays hot during replies for barge-in; a 1 s grace window after TTS start ignores echo leak.
Avatar: Live2D Cubism 4 via pixi-live2d-display. Default models are Haru (女) and the Natori sample / Mark (男), overridable via env. Expression cues from Claude map to per-model mood slots (Haru's f01–f08, Natori's Smile/Angry/Sad/etc.).
Tracking: MediaPipe Face Landmarker runs every ~2 s producing a 52-blendshape snapshot + nose-tip position. The snapshot is prefixed to each user turn as a natural-language hint; the nose tip drives the avatar's model.focus so her eyes follow you. Leave/return transitions fire a short check-in turn.
Lip sync: Web Audio AnalyserNode reads RMS off the TTS stream and writes ParamMouthOpenY + a ParamMouthForm boost every frame so the avatar's mouth opens and shapes with the syllable envelope. Patched at the coreModel.update level because the vertex bake happens inside Cubism's own internal update.

Personality & memory

iris has two user-editable markdown files at the project root, both loaded into every Claude turn:

persona.md — who iris is. Describe her voice, relationship to you, roleplay setting. Edit freely; hardcoded output rules (no emoji / markdown / code blocks, because this is TTS) always stay on top.
memory.md — long-term facts iris knows about you. Iris saves new facts automatically by emitting <remember>fact in one sentence</remember> mid-reply; you can also edit it directly.

Both files are gitignored. The repo ships committed defaults at persona.default.md / memory.default.md as fallbacks. Session history also persists across restarts via .iris/session-meta.json — a SHA-256 hash of the current system prompt plus the voice preset and Claude model keys the resume, so changing identity settings starts a fresh session instead of replaying incompatible history.

Settings panel

Click ⚙︎ settings in the chat header for:

Persona text area (save, restore default)
Memory text area (save, clear all)
Current session id + "new session" button

The language, persona (voice preset + avatar), speaking speed, and model selectors live in the compose bar and are persisted in localStorage.

Running

Supported setup: macOS 12+ (Intel or Apple Silicon) or a recent x86_64/arm64 Linux distribution. Windows users should use WSL2; the supplied installers are POSIX shell scripts and are not tested in native PowerShell. You need:

Node.js 20+
Python 3 and edge-tts (python3 -m pip install --user edge-tts, or pipx install edge-tts)
the claude CLI, installed and logged in (claude must work in the same shell)
Git, CMake, curl, tar, and a C/C++ toolchain
on macOS, Xcode Command Line Tools (xcode-select --install)
a modern browser with microphone and webcam permissions

# 1. Install the exact Node dependencies from package-lock.json
npm ci

# 2. Build pinned whisper.cpp v1.8.6 and verify the model (~181 MB)
./scripts/install-whisper.sh

# 3. Start the loopback-only server
npm run dev
# open http://localhost:3000

The whisper installer verifies the official model's SHA-256. For a controlled mirror, set both IRIS_WHISPER_MODEL_URL and an independently obtained digest in IRIS_WHISPER_MODEL_SHA256.

Paraformer is optional. Its installer pins and verifies the SHA-256 of the fixed official sherpa-onnx-paraformer-zh-2023-09-14 GitHub release asset:

./scripts/install-paraformer.sh

For a controlled mirror, set both IRIS_PARAFORMER_URL and a SHA-256 obtained through a trusted independent channel in IRIS_PARAFORMER_SHA256. If you skip Paraformer, Chinese input falls back to whisper.cpp.

First reply is ~4 s (cold Claude subprocess + model warm-up). Every subsequent reply lands a first audio chunk in roughly 1 s.

Network and privacy boundary

Speech recognition and file storage stay local after model setup. The Claude CLI receives the transcript, expression hint, persona, and memory as prompt context; edge-tts receives reply text for synthesis. Internet access is also required for browser-loaded Live2D, VAD, and MediaPipe packages/models. A CDN outage can therefore disable voice, tracking, or the avatar while the local server is up. The server binds to 127.0.0.1; iris is not a LAN or public deployment as shipped.

Environment variables

Variable	Default	Purpose
`PORT`	`3000`	Loopback HTTP/WebSocket port
`IRIS_CLAUDE_MODEL`	`sonnet`	Default Claude model (UI picker overrides per-connection)
`IRIS_WHISPER_MODEL`	`scripts/whisper.cpp/models/ggml-small-q5_1.bin`	Default/auto whisper model path; relative paths resolve from the process working directory
`IRIS_WHISPER_MODEL_EN` / `_ZH`	`$IRIS_WHISPER_MODEL`	Per-language whisper override
`IRIS_PARAFORMER_MODEL`	`models/paraformer-zh/model.int8.onnx`	Paraformer ONNX path
`IRIS_PARAFORMER_TOKENS`	`models/paraformer-zh/tokens.txt`	Paraformer token file path
`IRIS_VOICE_PRESET`	`female-xiaoxiao`	Startup preset; one of the preset IDs defined in `server/tts.js`
`IRIS_TTS_RATE`	`0`	Default speaking speed offset (−50..+50 %)
`IRIS_TTS_BIN`	auto-detected `edge-tts`	Explicit path to the `edge-tts` executable
`IRIS_AVATAR_FEMALE_URL`	Haru CDN	Custom female Live2D `.model3.json` URL
`IRIS_AVATAR_MALE_URL`	Natori CDN	Custom male Live2D `.model3.json` URL

IRIS_VOICE_EN, IRIS_VOICE_ZH, and their _M variants are not runtime settings. Choose a built-in preset in the UI, or add a preset in server/tts.js. Custom avatar URLs and every texture/motion they reference must be reachable by the browser and permit the required cross-origin requests.

Platform notes

install-whisper.sh automatically enables Metal + Accelerate on Apple Silicon, uses Accelerate without Metal on Intel macOS, and selects a portable CPU build on Linux. Override the build parallelism with IRIS_BUILD_JOBS.

Quality checks

npm test          # deterministic unit tests; no models or network needed
npm run check     # syntax checks plus tests
npm run audit     # production dependency audit; requires npm registry access

CI runs npm ci, npm run check, and npm run audit on Node 20.

Layout

iris/
├── server/
│   ├── index.js         Fastify + WebSocket entry, /api settings endpoints
│   ├── claude.js        Persistent Claude CLI subprocess + persona/memory
│   ├── stt.js           Whisper/Paraformer dispatcher
│   ├── whisper.js       whisper.cpp wrapper with script-validated retry
│   ├── paraformer.js    sherpa-onnx Paraformer-zh wrapper
│   └── tts.js           edge-tts wrapper with voice presets + rate
├── web/public/
│   ├── index.html       Chat + stage markup, settings modal
│   ├── style.css
│   ├── main.js          Browser client (WebSocket, webcam, settings, VAD)
│   ├── stage.js         Live2D avatar with live persona swap + lip sync
│   ├── vad.js           Silero VAD wrapper
│   ├── face.js          MediaPipe Face Landmarker wrapper
│   └── audio.js         WAV helpers
├── persona.default.md   Committed default persona
├── memory.default.md    Committed default memory fallback
├── persona.md           Personal persona (gitignored)
├── memory.md            Personal long-term memory (gitignored)
├── models/              Whisper + Paraformer model files (gitignored)
├── scripts/             install-whisper.sh, install-paraformer.sh
└── package.json

中文

一个运行于浏览器中的语音 + 视频 AI 伙伴。以 Claude 为大脑，Live2D 为面孔，支持实时对话、口型同步、面部追踪与长期记忆。无需按键操作，只需开口说话——她便能看见你、听见你，并以自然的声音作答。

功能概览

持续语音对话——Silero VAD 全程监听，无需按键。你在她回应途中开口，她会立即停顿，让位于你的发言。
Live2D 虚拟形象——口型与 TTS 音频同步，表情由 Claude 通过内嵌的 <expr:happy> 之类标签驱动；视线会跟随你的面部移动，言谈间身体亦有自然的动作。
面部追踪——MediaPipe 将你的表情转化为自然语言提示传给 Claude（如 [用户刚从 neutral 变为 smiling]），使她能够回应情绪的变化。你离开或重新出现在镜头前同样会被察觉，并触发一句简短的招呼。
持久化记忆——人设与长期事实均以本地 markdown 文件存放，可直接编辑。她亦能在回复中通过 <remember>…</remember> 标签自动追加新事实。
静默时主动问候——若超过 90 秒未开口，她会主动说一句轻语。
多模型切换——UI 下拉即可在 Sonnet / Opus / Haiku 之间切换，无需 API key，通过本地已登录的 claude CLI 调用你的订阅。
中英双语——逐轮自动识别。中文由 Paraformer-zh 加速，其他语言由 whisper.cpp 承担。

架构

┌─────────────── 浏览器 ───────────────┐         ┌──────── 服务端 ────────┐
│ PIXI + pixi-live2d-display          │         │ Fastify + WebSocket    │
│ MediaPipe Face Landmarker           │ ◄─ WS ─►│ 常驻 Claude CLI 子进程  │
│ Silero VAD(常开麦克风)              │         │ whisper.cpp + Paraformer│
│ Web Audio TTS 播放 + RMS 口型同步    │         │ edge-tts(Azure 神经音)  │
└─────────────────────────────────────┘         └────────────────────────┘

大脑：Claude Code CLI 以 --input-format stream-json 模式常驻，每个会话对应一个长期存活的子进程，仅首轮承担冷启动开销。UI 可切换 sonnet / opus / haiku，通过本地已登录的 Claude 订阅运行，无需 API key。
STT：
- whisper.cpp（ggml-small-q5_1.bin，约 181 MB，CPU 下每轮约 3 秒）——多语言支持，内置严格的中英文字脚本校验器，识别语种错误时自动重试。
- Paraformer-zh 通过 sherpa-onnx-node（约 232 MB ONNX，每轮约 300–500 ms）——当前最优的中文识别方案，选中"中文"时自动启用。基于原生 Node 插件，无需 Python 辅助进程。
TTS：edge-tts——免费但需要联网。预设以"英 + 中"神经音配对（女声：Xiaoxiao / Xiaoyi / HsiaoYu；男声：Yunxi / Yunyang / Yunjian / YunJhe），语速与模型均可在输入栏的下拉中调节。
语音活动检测：@ricky0123/vad-web 中的 Silero VAD，运行于 AudioWorklet 之中。回复播放期间仍保持监听以支持打断；TTS 开始后设有 1 秒的宽限期，用以屏蔽回音泄漏。
虚拟形象：由 pixi-live2d-display 驱动 Live2D Cubism 4 模型。默认采用 Haru（女）与 Natori / Mark（男），可通过环境变量替换为自定义模型。Claude 发出的表情标签会映射至各模型对应的情绪槽（Haru 的 f01–f08，Natori 的 Smile / Angry / Sad 等）。
追踪：MediaPipe Face Landmarker 每两秒左右进行一次推理，输出 52 项 blendshape 与鼻尖位置。快照会作为自然语言前缀拼接到每轮用户消息中；鼻尖位置同时驱动 Live2D 的 model.focus，令她的视线始终追随你。离开与归来均会触发简短的问候。
口型同步：Web Audio 的 AnalyserNode 从 TTS 音频流中读取 RMS，每帧写入 ParamMouthOpenY 并叠加 ParamMouthForm，使嘴形的开合与轮廓紧贴音节包络。Hook 点置于 Cubism 的 coreModel.update 之上，因为顶点烘焙正发生于此层，写在外层的 internalModel.update 上为时已晚。

人设与记忆

项目根目录下有两份用户可编辑的 markdown 文件，Claude 在每一轮对话中均会读取：

persona.md——定义她的身份：声音、与你的关系、角色扮演的设定等。可自由编辑；硬编码的输出规则（禁用 emoji、markdown、代码块，因输出直送 TTS）始终位居最上层，不可覆盖。
memory.md——关于你的长期事实。她在回复中发出 <remember>一句话</remember> 即会自动追加至此文件，你亦可直接编辑。

二者均 不纳入 git。仓库内置 persona.default.md / memory.default.md 作为回落。会话历史通过 .iris/session-meta.json 跨重启持久化——系统提示词 SHA-256、音色预设与 Claude 模型共同决定是否续接；修改身份相关设置后会自动开启新会话，避免续接不兼容的历史。

设置面板

点击聊天栏顶部的 ⚙︎ settings，可进行以下操作：

编辑人设（保存、恢复默认）
编辑记忆（保存、一键清空）
查看当前会话 id，或创建新会话

语言、音色与形象预设、语速、模型四项下拉位于输入栏，所选值保存在 localStorage 中。

运行

推荐环境为 macOS 12 以上（Intel 或 Apple Silicon），或较新的 x86_64 / arm64 Linux。Windows 建议使用 WSL2；仓库内的安装器是 POSIX shell 脚本，尚未在原生 PowerShell 下验证。前置条件如下：

Node.js 20 以上
Python 3 与 edge-tts（执行 python3 -m pip install --user edge-tts，或 pipx install edge-tts）
已安装并登录的 claude CLI，且同一终端内可直接执行 claude
Git、CMake、curl、tar 与 C/C++ 工具链
macOS 另需 Xcode Command Line Tools（xcode-select --install）
支持麦克风与摄像头权限的现代浏览器

# 1. 严格按 package-lock.json 安装 Node 依赖
npm ci

# 2. 编译固定于 v1.8.6 的 whisper.cpp，并校验默认模型（约 181 MB）
./scripts/install-whisper.sh

# 3. 启动仅监听本机回环地址的服务
npm run dev
# 打开 http://localhost:3000

whisper 安装器会校验官方模型的 SHA-256。若使用受控镜像，须同时设置 IRIS_WHISPER_MODEL_URL，以及从独立可信渠道取得的 IRIS_WHISPER_MODEL_SHA256。

Paraformer 为可选项。安装器已固定并校验官方 GitHub release 中 sherpa-onnx-paraformer-zh-2023-09-14 文件的 SHA-256：

./scripts/install-paraformer.sh

若使用受控镜像，须同时设置 IRIS_PARAFORMER_URL，以及从独立可信渠道取得的 IRIS_PARAFORMER_SHA256。不安装 Paraformer 时，中文识别会回退至 whisper.cpp。

首轮回复约需 4 秒（冷启动 Claude 子进程与模型预热），此后每轮首段音频大致在 1 秒内送达。

网络与隐私边界

模型安装完成后，语音识别与文件存储留在本机。Claude CLI 会收到转写文本、表情提示、人设和记忆作为 prompt 上下文；edge-tts 会收到回复文字用于合成。浏览器还须联网加载 Live2D、VAD、MediaPipe 代码与模型。因此 CDN 故障时，即使本地服务正常，语音、追踪或形象也可能不可用。服务默认仅监听 127.0.0.1，当前形态不用于局域网或公网部署。

环境变量

变量	默认	作用
`PORT`	`3000`	本机 HTTP / WebSocket 端口
`IRIS_CLAUDE_MODEL`	`sonnet`	默认 Claude 模型（UI 选择会覆盖每个连接的值）
`IRIS_WHISPER_MODEL`	`scripts/whisper.cpp/models/ggml-small-q5_1.bin`	默认及自动识别使用的 whisper 模型路径；相对路径按进程工作目录解析
`IRIS_WHISPER_MODEL_EN` / `_ZH`	同上	按语言覆盖 whisper 模型
`IRIS_PARAFORMER_MODEL`	`models/paraformer-zh/model.int8.onnx`	Paraformer ONNX 路径
`IRIS_PARAFORMER_TOKENS`	`models/paraformer-zh/tokens.txt`	Paraformer token 文件路径
`IRIS_VOICE_PRESET`	`female-xiaoxiao`	启动音色预设，值须为 `server/tts.js` 内的预设 ID
`IRIS_TTS_RATE`	`0`	默认语速偏移（−50..+50 %）
`IRIS_TTS_BIN`	自动查找 `edge-tts`	显式指定 `edge-tts` 可执行文件路径
`IRIS_AVATAR_FEMALE_URL`	Haru CDN	自定义女性 Live2D `.model3.json` URL
`IRIS_AVATAR_MALE_URL`	Natori CDN	自定义男性 Live2D `.model3.json` URL

IRIS_VOICE_EN、IRIS_VOICE_ZH 及其 _M 变体并不是有效的运行时配置。请选择 UI 内置预设，或在 server/tts.js 中新增预设。自定义形象 URL 及其引用的纹理、动作资源必须可被浏览器访问，并允许所需的跨域请求。

平台说明

install-whisper.sh 会在 Apple Silicon 上自动启用 Metal + Accelerate，在 Intel macOS 上使用无 Metal 的 Accelerate，并在 Linux 上选择可移植 CPU 构建。可通过 IRIS_BUILD_JOBS 调整并行编译任务数。

质量检查

npm test          # 确定性单元测试，不需要模型或网络
npm run check     # JavaScript 语法检查与测试
npm run audit     # 生产依赖安全审计，需要访问 npm registry

CI 在 Node 20 上依次运行 npm ci、npm run check 与 npm run audit。

Navigation Menu

Repository files navigation

iris

What it does

Architecture

Personality & memory

Settings panel

Running

Network and privacy boundary

Environment variables

Platform notes

Quality checks

Layout

中文

功能概览

架构

人设与记忆

设置面板

运行

网络与隐私边界

环境变量

平台说明

质量检查

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
models		models
scripts		scripts
server		server
web/public		web/public
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
memory.default.md		memory.default.md
package-lock.json		package-lock.json
package.json		package.json
persona.default.md		persona.default.md

Latest commit

History

Folders and files

Repository files navigation

iris

What it does

Architecture

Personality & memory

Settings panel

Running

Network and privacy boundary

Environment variables

Platform notes

Quality checks

Layout

中文

功能概览

架构

人设与记忆

设置面板

运行

网络与隐私边界

环境变量

平台说明

质量检查

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages