Shader composition

Mora’s visual DSL builds shader programs from operator objects that you can inspect, transform in time, and link to sound. This page covers how operators compose, how they become GLSL, and the two ways visuals relate to audio.

The visual DSL constructs CLOS operator objects. Generators such as osc, noise, gradient, and solid compose with coordinate, color, blending, and modulation operators through >>. The result is an inspectable program: a retained operator tree.

In this channel form, the :viz facet is a shader chain that starts with the osc generator and continues through kaleid and rotate:

(d1! :audio (s "bd ~ sn [hh hh]")
     :viz (>> (osc :freq 36)
              (kaleid 6)
              (rotate :speed .03))
     :post (>> (mora.visual:pixelate :px 128
                                     :py 72)
               (mora.visual:ascii :characters " .:-=+*#%@")
               (mora.visual:edge-glow :amount .6
                                      :radius 1.5)))

The :audio and :post facets are described in Multimodal channel forms and Ordered post-processing.

Emission produces GLSL with cached compiled programs and retained bindings. The shader language covers retained operator trees, GLSL emission, incoming bindings, cached native programs, temporal composition, and event-linked metadata.

GLSL emission is an interpretation of the visual-operator subset, through the emit-glsl protocol. See Common Lisp and CLOS for how protocols like this one are organized.

A visual operator is also a pattern. Temporal transforms can act on it, and pattern values can drive visual parameters. The shared >> operator supports both temporal pipelines and visual input threading under defined ordering rules.

In a mixed >> pipeline, visual input-threading stages precede the temporal transforms that apply to the resulting visual pattern. Write the shader operators first, then the temporal transforms that act on the whole visual pattern. Pipelines and composition covers >> in general.

Audio patterns and visual operators can use common temporal operations. Sound parameters do not automatically become shader parameters, and backends differ in their capabilities. See Sound, image, and shared time.

Visuals can relate to sound in two distinct ways:

1. Event linkage. Shared event metadata, event envelopes, and co-emit can preserve a relation through temporal transformation.
2. Measured-audio linkage. cl-mltk analysis supplies measured PCM, frequency content, and derived features from actual audio.

Neither is a substitute for the other. A kick event supplies an exact scheduled moment; its rendered spectrum contains information that was not present in the source event.

Measured-audio linkage requires cl-mltk and an audio input; see Machine listening.

code samples Studio’s editor texture into the visual pipeline, where it can be composed with shader operators.

Separately, Studio can fade the background visual intensity around editing activity, including navigation, to maintain legibility.

Neither code nor the background fade changes the sound pattern; both are presentation choices.