OP-XY Agent: Browser Replica and AI Companion for Teenage Engineering OP-XY

What changed

The OP-XY Agent is an open-source web application designed to serve as a comprehensive companion for the Teenage Engineering OP-XY hardware. It features a playable, pixel-faithful replica of the OP-XY's interface, meticulously rebuilt from official panel drawings and camera captures to match the original device's layout, including keys, encoders, LEDs, and screen elements at a 480x222 pixel resolution. The replica simulates the OP-XY's four modes, shift layers, engine/filter/LFO selectors, tempo, project settings, and mixer, with sound calibrated against the actual hardware for accurate browser-based playback of synth engines, drum kits, and the sequencer. The application also supports direct connection to a physical OP-XY via Web MIDI (compatible with Chrome and Edge), mirroring device activity and enabling control over notes, transport, track selection, tempo, and sound parameters. A key component is the integrated AI agent, powered by Anthropic's Claude, which draws knowledge from a custom-rewritten manual. This agent can answer user queries, provide step-by-step instructions for operations (including exact key presses and encoder turns), and translate natural language prompts into device settings. It can also program patterns, scenes, and songs directly onto the virtual or physical OP-XY. Furthermore, the project includes a browser-based preset maker that converts user samples into drum kits, multisamples, or synth sampler presets, ready for transfer to the device.

Why it matters for builders

For developers and musicians working with the Teenage Engineering OP-XY, this project offers an unprecedented level of accessibility and control. The browser-based replica eliminates the need for physical hardware to learn and experiment with the OP-XY's functionalities, making it easier for new users to get started and for existing users to prototype ideas. The direct USB MIDI programming capabilities, guided by an AI that understands the device's manual, streamline complex workflows, allowing builders to translate creative concepts into tangible sound designs more efficiently. The ability to create custom presets from user samples directly within the browser further enhances the creative toolkit available to builders.

Practical impact

Builders can explore the OP-XY Agent by accessing the web application, which runs as static files in the browser, requiring no server-side infrastructure. For those interested in contributing or customizing, the project is developed in TypeScript and requires Node.js 24 and pnpm for local development. Developers can install dependencies with pnpm install, run the development server with pnpm dev, and execute checks with pnpm check && pnpm lint && pnpm test:unit --run. The project's architecture is detailed in docs/ARCHITECTURE.md, outlining modules for core logic, device interaction, replica simulation, and agent functionality. The safety of the physical device is a priority, with a strict deny-list for all MIDI messages sent to the hardware and explicit user approval required for any state-changing actions. Builders can also leverage the preset maker to generate custom sound kits from their own audio samples.

Caveats and source limits

This project is described as pre-production and moving fast, with its roadmap detailed in docs/PLAN.md. While the AI agent uses Claude, users are responsible for their own API keys, which are kept client-side and sent only to the provider. The project is an independent effort and not affiliated with or endorsed by Teenage Engineering. The source does not provide specific benchmark results for the AI agent's performance or response times, nor does it detail the exact range of MIDI messages supported beyond general control functions. Pricing for API usage is not specified, as it depends on the user's Anthropic or OpenAI API keys.