Skip to content
← All builds
openai avatar

Open source

openai/openai-realtime-embedded

Instructions on how to use the Realtime API on Microcontrollers and Embedded Platforms

1.6k starsUpdated 1y agoMIT license
Install
$ git clone https://github.com/openai/openai-realtime-embedded.git# then ask your coding agent> Set up openai-realtime-embedded for "Talk to the OpenAI Realtime API from an ESP32 over WebRTC"  on my ESP32-S3. Follow the README.
No published photo or video yet: repository detailsopenai/openai-realtime-embedded

Talk to the OpenAI Realtime API from an ESP32 over WebRTC

Gadget, Medium

OpenAI's embedded reference points to Espressif's official demo: real-time voice chat on an ESP32-S3-Korvo-2 board, with voice commands that control the device through function calls.

View build on GitHub ↗Share on X

openai-realtime-embedded is a short index repo. The working code is Espressif's openai_demo in esp-webrtc-solution, which connects an ESP32 to the Realtime API over WebRTC. It gets a short-lived token, exchanges SDP, and then streams audio both ways. Voice commands can control the device through function calls.

Compared with the original OpenAI embedded SDK (still on that repo's esp32 branch), Espressif's version adds OPUS encoding, acoustic echo cancellation and a sturdier peer connection. The default board is the ESP32-S3-Korvo-2, and other boards go through esp_board_manager. You set your Wi-Fi in settings.h, export an OpenAI API key and build with ESP-IDF. This is the voice layer you'd put inside a talking robot.

How your agent drives it

Copied from the espressif/esp-webrtc-solution openai_demo README

  1. Shell
    $ pip install --upgrade esp-bmgr-assist
  2. Shell
    $ idf.py gen-bmgr-config -b YOUR_BOARD_NAME
  3. Shell
    $ idf.py -p YOUR_SERIAL_DEVICE flash monitor

Any coding agent that can run a shell can run these. Read the README before you run anything on a robot that moves.

Ideas to try with it

Our ideas, not the builder's claims.

  1. Give an Otto or Stack-chan build a real-time voice with this as the audio layer
  2. Expose your robot's motors as functions and drive it by voice
  3. Run it on a cheaper ESP32-S3 board through esp_board_manager
Source
openai/openai-realtime-embedded on GitHub
Last commit
2025-03-25
Builder
openai
Repo
openai/openai-realtime-embedded, 1,582 stars
Built with
ESP32-S3

Links

Guides: Codex and ChatGPT hardware projects people actually built

What to build next

Builds that share a board, robot or agent with this one.