I built a teddy bear that lives inside a hologram. It listens to your voice, understands what you say, responds with emotion, and moves accordingly. All in real time.

HoloBear combines a physical hologram display with a full AI conversation pipeline: voice recognition through Whisper, emotional intelligence through Claude API, natural speech through ElevenLabs, and expressive 3D animation through Unity. The bear doesn't just talk. It reacts, dances, thinks, laughs, and sleeps.

This entire system was designed, built, and integrated by one person.

Solo
Designed & built entirely alone
18+
Emotion-driven animations
Real-time
Voice → AI → Speech → Animation

Live demo

Real-time voice conversation with HoloBear through S-GRAM hologram display

How it works

When you speak, Whisper transcribes your voice into text. That text is sent to the Claude API, which generates a response tagged with an emotion. The emotion tag triggers a matching animation on the 3D bear model in Unity, while ElevenLabs converts the response into natural speech. The bear moves its mouth in sync with the audio through a real-time lip-sync system.

The result is a small bear floating inside a hologram that genuinely feels like it's listening and responding to you.

The bear doesn't just respond. It reacts. That's the difference between a chatbot and a companion.

Features

Voice conversation
Speak naturally through your microphone. Whisper STT transcribes in real time, Claude generates emotional responses
Emotion-driven animation
18+ animations including dancing, waving, jumping, thinking, laughing, sleeping, and more. Each triggered by AI emotion tags
Facial expressions
BlendShape-based facial system with happy, sad, surprised, angry, love, sleepy, and wink expressions
Natural speech
ElevenLabs multilingual v2 voice synthesis with real-time lip sync
Idle behaviors
The bear blinks, looks around when idle, yawns after two minutes, and falls asleep after five. It wakes up surprised when you speak
Hologram display
S-GRAM Pepper's Ghost kit projects the bear into physical space. Pure black background makes only the bear visible

Tech stack

Unity
3D engine & animation
Claude API
Conversation & emotion
ElevenLabs
Text-to-speech
Whisper
Speech-to-text

The pipeline

1. Voice input
User holds spacebar to record. Audio captured at 16kHz, converted to WAV, sent to OpenAI Whisper API
2. AI response
Transcribed text sent to Claude API with emotion-tag instructions. Response includes [happy], [dancing], [sad], etc.
3. Animation
Emotion tag parsed and mapped to one of 18+ animations via Unity Animator. Facial BlendShapes set per emotion
4. Voice output
Response text sent to ElevenLabs. PCM audio streamed back, played with real-time lip sync on the bear model

Why this matters

AI companions today live inside screens. They are text on a phone, a voice from a speaker, an avatar on a monitor. HoloBear is an attempt to bring them one step closer to physical presence.

This is not a product yet. It is a working prototype that proves a full voice-to-hologram AI companion pipeline can be built by a single person with consumer hardware. The next steps are a standalone phone app, a custom hologram display, and eventually, a product that anyone can place on their desk.

AI companions shouldn't just live inside screens. They should feel like they're sitting right next to you.

← Emotion Matrix The Hidden Cost of RLHF →