Zum Hauptinhalt springen
Research

Intention-Based
Interface

Our research question: can menu structures and keyboard shortcuts be replaced by pure intent — semantically captured, AI-mediated, locally executed — and to what extent is this substitution defensible?

Open the engineering logbookInsights into the ongoing PapoNox research process — methodological notes, discovery findings, crash analyses.
Three research pillars

Research matrix

Chat input, AI vision and voice input — three modalities we investigate in parallel and at different levels of maturity.

Under research

Chat Mode

The Brain

Text-based dialogue between user and DAW. Research question: how reliably can musical intent be extracted from free-form German language — and what systematic limits does this approach face when confronted with domain-specific terminology?

Designed for screen reader and Braille display users
Under research
Under research

AI Vision

The Eyes

Screenshot analysis, OCR and AI-assisted image understanding as a methodological approach to address UI elements beyond official interfaces. Does the three-tier grounding cascade — accessibility API → OCR bounding boxes → vision-language model — work reliably enough for productive use? An open question.

Open question: how precisely can the vision layer reproducibly identify UI elements?
Under research
Long-term question

Voice Control

The Voice

Speech-to-text control as a possible extension of the input modality. Whether and when this modality can be meaningfully investigated depends on funding outcomes and on findings from the chat-mode research.

Future perspective

Not part of the current funding phase. Continued funding and methodological groundwork required.

Proof of Concept

Every command at a glance

50 actions · 18 pre-intent interceptors

What the prototype currently demonstrates in a documented form (Phase 0)

Transport

5 commands
  • "play"Start playback
  • "stop"Stop playback
  • "record"Start recording
  • "rewind"Jump to the start
  • "go to bar 5"Jump to a bar

Tracks

10 commands
  • "select track 3"Select track by number
  • "select bass"Select track by name — fuzzy match
  • "track 2 mute"Mute
  • "track 2 solo"Solo
  • "volume to –6 / louder 3 dB"Volume absolute + relative
  • "pan left"Set panorama
  • "new MIDI track"Create track
  • "rename track 3 bass"Rename track
  • "next track"To the next track
  • "previous track"To the previous track

Tempo & Project

7 commands
  • "tempo"Query tempo
  • "faster"Tempo +1 BPM
  • "slower"Tempo –1 BPM
  • "tempo to 140"Set tempo
  • "save"Save
  • "save as MySong"Save as
  • "metronome / click"Click on/off

Cycle & Loop

3 commands
  • "cycle / loop"Loop on/off
  • "loop from bar 5 to 12"Set loop range
  • "cycle range"Query range

Editing

2 commands
  • "undo"Undo — edit stack
  • "redo"Redo

Vision · Screen

7 commands
  • "what do you see"Screenshot → OCR
  • "describe"OCR → vision model Qwen3-VL
  • "read aloud"Read recognized text aloud
  • "click on {X}"Click — 3-tier cascade
  • "double-click on {X}"Double-click
  • "right-click on {X}"Context click
  • "where is {X}"Find position on screen

Plugins & Sends

9 commands
  • "which plugins"Plugin list of the track
  • "add EQ / compressor"Insert plugin
  • "plugin 3 bypass"Bypass by slot or name
  • "open plugin 3"Open plugin interface
  • "instrument bypass"Control instrument slot
  • "open instrument"Open instrument interface
  • "which sends"Send list of the track
  • "send 1 to –6"Set send level
  • "send 1 mute"Mute send

Plugin presets

2 commands
  • "plugin presets"Preset list
  • "load preset 5"Load preset

Import, Export & Status

5 commands
  • "import audio"Import audio file
  • "export / bounce"Create mixdown
  • "status"Announce transport status
  • "which tracks"Read track list aloud
  • "help"Command overview
Initial proof-of-concept findings documentedAnything beyond this falls within the open validation research
Where it fits

Where PapoNox comes in

How the methodological approach differs from existing tools for blind users — as a documented research status, not a product promise.

What sets it apart

Operating a professional DAW screenlessly — without an official control API

Unlike Reaper or Ableton Live, Logic Pro offers no open control interface. PapoNox investigates whether comparable depth can still be reached: through publicly documented runtime interfaces, a grounding cascade of accessibility API, OCR and a vision model, plus exact, auditorily confirmed values. This combination — German-language, screenless, beyond official interfaces — is matched by no existing tool in this form.

CriterionPapoNoxLogic Pro + VoiceOverOSARA (Reaper)AbletonMCPVoiceOver (generic)
Target DAWLogic ProLogic ProReaperAbleton Liveall apps
Designed for screenless blind operationYesPartialYesNoGeneric
German-language text commandsYesNoNoPartialNo
Exact values (dB/pan) confirmed by audioYesPartialYesNoNo
Plugin parameters controllable semanticallyYesCumbersomePartialNoNo
UI addressable beyond official API (OCR/vision)YesNoNoNoNo
Modal dialogs automatedYesNoPartialNoNo
Works without an open vendor APIYes——No—

This overview reflects the methodological research status (initial feasibility) — not a finished product. Here "Yes" means: demonstrated in the prototype in documented form, not yet independently validated.

Architecture

Methodological principles

How we approach the research question operationally — four principles already empirically implemented in the prototype.

On-device inference

Local & free

We use Apple's FoundationModels as an on-device language model — executed entirely locally. Research premise: no cloud connectivity, no data sharing, no recurring inference costs.

Process injection

Direct access investigated

Direct integration into the DAW process via Apple's documented DYLD_INSERT_LIBRARIES mechanism and the public Objective-C runtime — a methodological approach to investigate to what extent internal data models can be opened up without compromises in usability.

Real-time IPC

Low latency in focus

Bidirectional inter-process communication via Unix Domain Socket using JSON Lines. We investigate how low end-to-end latency can remain when all components operate locally and synchronously.

Privacy by design

No data sharing

Research-ethical principle: all data remain on the user's device. No account, no tracking, no upload. The vision module, the language model and speech output operate exclusively locally.

The Stack

From chat to DAW

Four layers. The prototype establishes that, in principle, the procedure holds. The detailed operationalisation is the subject of ongoing research.

Input

Chat CLI

Text-based commands in German

Processing

PapoNox Core

Intent engine + on-device LLM fallback

Bridge

IPC bridge

Unix Domain Socket + JSON Lines

Output

Logic Pro

Objective-C runtime + CGEvent synthesis

Target latency:< 200ms
Research stages

Milestones

What has been achieved so far — and what still needs to be measured methodologically.

Phase 0Completed
Phase 0

Architectural feasibility

  • Research question precisely formulated
  • Prototype as documented proof of concept
  • On-device LLM and public runtime interfaces empirically validated
  • Open-source research licensing (MIT for code, CC-BY-4.0 for documentation)
Phase 1
Phase 1 — open

DAW control — validation phase

  • Precision measurement of continuous values across the full value range
  • Empirical determination of the methodological limits
  • Validation of the three-tier grounding cascade (AX → OCR → vision)
  • User studies, external accessibility audit, conference submission — funding in preparation
Phase 2
Long-term research perspective

Universal accessibility

  • Transferability of the methodology to other closed applications
  • Classification of addressable application domains
  • Context sensitivity as an open methodological research question
  • No commitment, but a documented perspective

Questions about the research project??

We do our research openly and methodologically transparent. Write to us — we welcome feedback, technical exchange and collaboration interest.

Contact