Machine listening

Machine listening gives visuals and tools a measured signal rather than an approximation derived from the score. This page explains how rendered audio returns to analysis in Mora, how orbit taps measure individual orbits, and how measured signals differ from feedback that follows a pattern’s events.

In ordinary playback, a pattern produces events and a backend such as SuperDirt renders them as sound (see From pattern to audio). Machine listening is a second direction of flow: rendered audio returns to analysis.

The mora/analysis system integrates cl-mltk. Native signal consumers read measured PCM, FFT magnitudes, band envelopes, energy, and related features. These describe actual audio, not a simulation of what the pattern ought to sound like.

Measured analysis is optional. Constructing patterns and sending ordinary sound do not depend on it. The normal Studio launcher tries to start audio analysis along with its other configured services.

The shared input is the captured return signal that signal views read as :tap :input. The captured stereo signal is downmixed to mono for analysis. If the input is unavailable, you can still explore patterns and unmeasured visuals, but analysis is not live.

Loading the analysis system does not change your audio routing. Configure the server device, orbit outputs, and monitoring for your own setup.

A PHALAR mix request also reads the shared input. It captures a contiguous window of the whole input, including any other audio on that return, so the input must already carry the intended mix; a silent input is not a usable query. Like a scope, the request measures what actually sounds; it does not read pattern events or an unevaluated draft.

An analysis worker publishes immutable snapshots. Each snapshot carries capture and sequence identity and timestamps.

A source that is warming, stopped, failed, or stale is reported as such. A missing return is not replaced by a fabricated signal, and silence is not replaced with an invented waveform.

Visual calibration (trim, noise threshold, drive, attack, and release) is separate from the underlying measured waveform and FFT.

Optional SuperDirt orbit taps offer a different observation point from the shared input. They are opt-in:

(orbit-taps :orbits '(0 1)
            :frames 1024
            :rate 30)

Orbit taps are explicitly owned, mono, discontinuous measurement windows taken from dry orbit buses, before orbit-global effects.

• Several stems on one orbit remain mixed. An orbit tap does not isolate individual stems.
• An orbit tap measures one dry orbit, not the final stereo return.
• Enabling orbit taps does not change audio routing and does not start playback.

A Mora channel owns a pattern; an orbit is a SuperDirt routing and mixing destination. The distinction is described in From pattern to audio.

In Studio, scope, frequency-scope, and spectrum read measured signal taps. :tap :input is the shared captured input; orbit taps expose the dry orbits you select.

(d1! (>> (notes "f3 ab3 c4 eb4")
         (s "psin")
         (scope :tap :input
                :samples 1024)
         (frequency-scope :tap :input
                          :min-hz 30
                          :max-hz 16000
                          :mode :line)))

This phrase produces events on its own; the signal views also need a configured, current return signal. The full set of inline views is described in Inline views.

Mora offers two distinct forms of linkage between sound and image:

1. Event linkage. Shared event metadata, event envelopes, and co-emit can preserve a relation through temporal transformation.
2. Measured-audio linkage. cl-mltk analysis supplies measured PCM, frequency content, and derived features from actual audio.

Neither is a substitute for the other. A kick event supplies an exact scheduled moment; its rendered spectrum contains information that was not present in the source event. A composition can use either or both. See Shader composition and Sound, image, and shared time.

The same distinction applies to Studio feedback. A token highlight or event visualization follows the installed pattern; a scope or spectral visual requires captured audio. One can work while the other is unavailable.

If an event view moves but a measured view stays empty, check whether the intended input or orbit tap is carrying a signal. The full checklist is in When something is silent or blank.

Beyond measured analysis, optional models support music similarity, natural-language discovery, constrained pattern generation, and complementary stem ranking:

• Learned search covers MuQ and MuQ-MuLan embeddings for similarity and natural-language queries.
• PHALAR stem pairing ranks library candidates for compatibility, either against an anchor recording or against a measured capture of the audible mix from the shared input.
• Generation and neural instruments covers prompt-to-sample generation, the performer assistant, and neural synthesis.

Each ML feature has its own installation, data, and licensing requirements. None is a prerequisite for writing or hearing a pattern.