Multimodal channel forms

A single channel form can compose audio, visual material, and image processing while keeping their responsibilities separate. This page explains the three facets, what happens when one is omitted, and how a channel relates to audio orbits and to linkage beyond the form.

The multimodal form separates three responsibilities:

(d1! :audio (s "bd ~ sn [hh hh]")
     :viz (>> (osc :freq 36)
              (kaleid 6)
              (rotate :speed .03))
     :post (>> (mora.visual:pixelate :px 128
                                     :py 72)
               (mora.visual:ascii :characters " .:-=+*#%@")
               (mora.visual:edge-glow :amount .6
                                      :radius 1.5)))

Studio supplies the canvas and native GL context. Visual operators and shaders can be described without starting an audio receiver, but measured waveform and spectrum views need an audio input.

An omitted facet is left alone; it is not an instruction to remove that facet. A visual-only or post-only edit therefore need not replace the running audio pattern.

When you re-evaluate a channel form, Mora can retain an unchanged closed audio form. Re-evaluation and source memoization distinguish closed literal construction from dynamic Lisp expressions, and dynamic Lisp expressions are still evaluated according to their assignment rules.

The :post facet has four distinct cases:

• An explicit :post supplies an ordered chain and replaces the renderer’s legacy post treatment.
• Omitting :post preserves the existing treatment.
• nil restores the legacy treatment.
• An empty chain deliberately bypasses it.

Ordered post-processing shows each case.

A Mora channel is an ownership and composition unit. The sounds inside one channel need not share a single SuperDirt output orbit: explicit per-pattern orbit annotations can separate stems for downstream mixing. See From pattern to audio.

Transitions are covered in Arrangement and transitions.

Audio/visual relationships can extend beyond the enclosing form. Mora provides event metadata and query-based linkage as well as measured signal inputs. Event-linked visuals follow the scheduled events of the score; measured-audio visuals follow analysis of the sound itself. The two are distinct; Shader composition describes both.

Measured-audio visuals need an audio input to analyze. Without one, patterns and unmeasured visuals can still be explored, but analysis is not live. See Machine listening and the signal views in Studio.

Start with complete visual examples like the ones in these pages. The example file examples/mora-framework-intro.mora demonstrates shaders and ordered image effects; evaluate its forms individually. See Framework examples.