Add initial shader module

[TristanCacqueray]

This change adds the loadShader, to be used like this:

let {uniforms} = await loadShader`
  uniform float iColorOffset;
  uniform vec3 iColor;

  void mainImage( out vec4 fragColor, in vec2 fragCoord ) {
    vec2 uv = fragCoord / iResolution.xy;
    vec3 col = 0.5 + 0.5*cos(iColorOffset+iColor+uv.xyx+vec3(0,2,4));
    fragColor = vec4(col, 1);
  }
`
let smooth = (desired) => (value) => value + ((desired - value) / 20)
setcpm(20)
$: s("bd hh [sd -] hh").gain("<1 .5>").onTrigger((_, _hap, now) => {
  // Bump a random color component
  uniforms.iColor.incr(.5, Math.floor(now))
  // Slowly change the palette offset
  uniforms.iColorOffset.set(cur => smooth(cur + 1))
}, false)

… as presented in https://club.tidalcycles.org/t/adding-webgl-shader-uniform-modulation/5481

[TristanCacqueray]

When I run pnpm install, the lockfile is updated to version 9 with a massive diff. I tried to setup pnpm version 6, but pnpm install failed with  WARN  GET https://registry.npmjs.org/acorn error (ERR_INVALID_THIS). May I propose a pnpm bump?

[TristanCacqueray]

This PR looks like this:

[felixroos]

hey, thanks for the PR, being able to directly control shaders is a great feature imo. I am trying to test it, but for some reason, the canvas is not visible, without getting an error.. is there anything else i need to do? I've cloned your branch and tried to test with your first basic example.

[felixroos]

When I run pnpm install, the lockfile is updated to version 9 with a massive diff. I tried to setup pnpm version 6, but pnpm install failed with  WARN  GET https://registry.npmjs.org/acorn error (ERR_INVALID_THIS). May I propose a pnpm bump?

yes the lockfile really needs updating, probably best to do it in a separate PR, i can do that

[TristanCacqueray]

@felixroos That's great to hear! I just pushed a fix that prevented the code to be reloaded (though refreshing the page should have displayed the canvas).

I guess this should be marked as draft until the shader parameters support pattern strings (the values presently need to be single quoted). Also, it would be nice to figure out how to control the modulations, like the decay value which is presently hardcoded. Though I think that PR is already quite usable.

The shader module should also support texture input, for now it's just a plain fragment shader.

[felixroos]

ah turns out the first example did set the alpha channel to 0 so it was always transparent :) I'm still not sure how the pattern interacts with the uniforms defined in the shader function, could you clarify that? The iColor doesn't seem to react to the pattern

[TristanCacqueray]

Interesting, on Firefox the alpha doesn't seem to be enabled. The iColor should be incremented by one each time a note is played, the effect is a bit dim, you can increase it by adding 4.0*iColor in the shader, or there is a gain attribute you can pass to the shader options, like so: {uniform: 'iColor', gain: 4}.

I picked 1 because the haps in onTrigger don't seem to have a velocity, but it would be great to use the sound intensity.

Also note that this PR only reacts to noteOn events, I think we would need to analyze the sound to get a more accurate modulation source, though this simple solution works great for percussion sounds.

[TristanCacqueray]

Lastly, something I haven't done yet but I think would be really nice, is to pass the cpm as uniform to sync to sin values in the shader to better match the composition. Perhaps we could define some useful macros in the shader headers...

[felixroos]

Interesting, on Firefox the alpha doesn't seem to be enabled. The iColor should be incremented by one each time a note is played, the effect is a bit dim, you can increase it by adding 4.0*iColor in the shader, or there is a gain attribute you can pass to the shader options, like so: {uniform: 'iColor', gain: 4}.

I picked 1 because the haps in onTrigger don't seem to have a velocity, but it would be great to use the sound intensity.

Also note that this PR only reacts to noteOn events, I think we would need to analyze the sound to get a more accurate modulation source, though this simple solution works great for percussion sounds.

This idea might be half baked, but wouldn't it be more flexible to allow the user to fill in the mapping between hap values and uniforms? Something like note("c a f e").pan(sine).shader({myUniformName: 'pan'}) . this would move more logic into the pattern realm and less hidden logic into the shader function. One more radical alternative would be to automatically define all hap values as uniforms, e.g note("c a f e").pan(sine).shader() would automatically define note and pan. although i'm not quite sure how to handle string values like notes

[felixroos]

some more ideas:

• it should make sense to move this to a separate package called "shader", as it depends on another package (picogl) + the shader file does not import anything from the draw package anyway
• it seems a bit arbitrary to me to have the canvas on the top right. I'd rather treat it like the hydra canvas and lay it behind the code in full size. There could potentially also be options to customize the size / move the canvas if you want that look.
• we could potentially get rid of picogl ?! i've done some experiments with webgl and it's not a lot of code that's needed to spin up a shader: https://github.com/felixroos/schattenspiel having less dependencies is less to worry about, but also not that big of a deal

what do you think?

[TristanCacqueray]

As explained in the tidal club post, I'm not really familiar with strudel, so I'm looking forward your feedback to improve the implementations.

About the mapping, well, I couldn't find examples for sending events from one track to another, so I just re-used the code from the Midi API. I guess we could also attach the shader destination directly to a signal or pattern like so: pan(sine.shader('uniformName')). The shader destination should also be a pattern to pick different uniforms or array position, e.g. shader("<uniform1 uniform2>").

Your other ideas sound great to me, I can update the PR tomorrow. I picked picogl because it handle the context loss, it's not implemented yet here, but it might be useful to avoid blank screen. Otherwise you are right, we could totally do it by hand.

[felixroos]

I guess we could also attach the shader destination directly to a signal or pattern like so: pan(sine.shader('uniformName')). The shader destination should also be a pattern to pick different uniforms or array position, e.g. shader("").

would that make it possible to call shader multiple times to populate multiple uniforms? If yes, i'd rather call it uniform, so you could do:

$: note("c a f e").pan(sine.uniform("iPan"))
$: s("hh*8").gain(".5 1".uniform("iGain"))

The shader function probably wouldn't be needed, as we're calling loadShader anyway, which could already trigger the rendering. It would only be needed if multiple shaders should be rendered in parallel, which is maybe out of scope. I'm still not sure if I like the separated definitions, so let's compare that to a single uniform mapping:

$: note("c a f e").pan(sine)
$: s("hh*8").gain(".5 1")
all(x=>x.uniforms({iPan: "pan", iGain: "gain"))

In this notation, the uniforms function maps uniform names to hap value keys. The all function is a way to run a function on all patterns (what you probably mean by tracks). Maybe both notations could also coexist..

Your other ideas sound great to me, I can update the PR tomorrow. I picked picogl because it handle the context loss, it's not implemented yet here, but it might be useful to avoid blank screen. Otherwise you are right, we could totally do it by hand.

Maybe let's keep it for now. If it's a separate package the dependency is fine i think

[felixroos]

just formatted the code (using pnpm codeformat) to pass the tests (hopefully) + it's easier to track changes this way

[TristanCacqueray]

With the latest commit, the pretty demo now looks like this:

let truchetFTW = await fetch('https://raw.githubusercontent.com/TristanCacqueray/shaders/refs/heads/main/shaders/Truchet%20%2B%20Kaleidoscope%20FTW.glsl').then((res) => res.text())
await loadShader(truchetFTW)

$: s("bd -".uniform("moveFWD")).bank("RolandTR909")
$: s("- sd".uniform("rotations:seq")).bank("RolandTR909")
$:
 n("<[0 2 3 - 7 8 9 -] [4 2 - 7 3 7 9 -] [9 8 6 3 - 9 4 -]>")
.uniform("modulations:pitch")
.anchor("<C4 A3 C4 G3>/2")
.chord("<Gm7 <C>>/2").voicing()
.transpose(12)
.sound("<sine, z_sine triangle>")
.decay("<.6 1 2 .5 .1>")
.gain("<.2 .2 .3 .4 .5 .1>")
.room("<.6 .5 .4 .3>")
.pan(sine.range(0, 1))
.jux(rev).roomsize(2).delay(.4)
.gain(.2)

In the uniform.mjs module you can find the uniform param documentation. I think this is starting to look great.
Now I'm not sure how to handle signals like sine, in the onTrigger handler, I only get their value at the noteOn event, not the continuous one.

Lastly, I would like to expose the "smoothing" function to pre-process the value, I think we should have:

• direct, e.g. to pass signal value as-is,
• incr(time), e.g. to smooth change and only increment the uniform, as it is presently hardcoded
• adsr, e.g. to let the value go up and down

I though about adding an extra component to the uniform string syntax, e.g. "iColor:decay(100)", but perhaps this should be expressed differently.

PS: Sorry I erased the codeformat, I'll setup the an auto formatter next.

[TristanCacqueray]

I've been studying the pattern and signal module and I'll update the PR with with the following pattern methods:

• uniform("name"): using pat.withValue to pass the current number value
• uniformTrigger("name"): using pat.onTrigger to pass the current note/sound event

It doesn't seem like we can check in advance if a pattern is numerical, so using two methods looks cleaner.

To smooth the value, it looks easier to use an extra argument, e.g. uniform("time", "incr:50"). Though we could also use an object argument, e.g. uniform({name: "time", smooth: "incr:50"}), but that sounds a bit more complicated because we need to evaluate the nested patterns.

Then I'll look into the uniforms method to set multiple uniform values at once, as you suggested with the all functions, but I'd like to figure out the simpler methods beforehand.

[TristanCacqueray]

Here is a new example:

await loadShader`
  uniform float iColorOffset;
  uniform float iColors[3];

  void mainImage( out vec4 fragColor, in vec2 fragCoord ) {
    vec2 uv = fragCoord / iResolution.xy;
    vec3 bcol = vec3(0.+iColors[0],2.+iColors[1],4.+iColors[2]);
    vec3 col = 0.5 + 0.5*cos(iColorOffset+uv.xyx+bcol);
    fragColor = vec4(col, 1);
  }
`
setcpm(20)
$:
 s("bd hh [sd -] hh")
.pan(sine.slow(6).uniform("iColorOffset"))
.uniformTrigger("iColors")

[felixroos]

I am struggling to understand the syntax passed to uniform / uniformTrigger, could you maybe elaborate more on what the different options do? Some functionality could maybe also expressed with patterns, which would add less complexity. Moving logic to pattern land would also allow people to implement their own functions from within the repl.

[TristanCacqueray]

Yeah, I'm not happy with the uniform syntax, but I couldn't figure out a better way to expose the following options:

• gain:N: a number to multiply the value before setting the uniform.
• slow:N: a number to change the value slowly: 1 is instantaneous, 50 is slow.
• index:N: the destination position for uniform's arrays. This can be a number, or seq to pick a different position each time, or random for a random position.

For uniformTrigger, the default index is based on the note or sound.

Here are some example usages:

let truchetFTW = await fetch('https://raw.githubusercontent.com/TristanCacqueray/shaders/refs/heads/main/shaders/Truchet%20%2B%20Kaleidoscope%20FTW.glsl').then((res) => res.text())
await loadShader(truchetFTW)

setcpm(96)

$: s("bd").bank("RolandTR808")
   .gain(2).dist("<1 .7 .7 .7>")
   .mask("<1@30 0@2>")
   // each kick rotate a different ring
   .uniformTrigger("rotations:index:seq:gain:.2")
$: sound("hh*4").bank("RolandTR808")
   .room(.3).gain(".25 .3 .4")
   .mask("<0@8 1@32>")
   // the hats change the color rapidely
   .uniformTrigger("icolor:gain:0.5:slow:5")
$: s("cp/8").bank("RolandTR808")
   .hpf(500).hpa(.8).hpenv("<-3 -2 -3 -2 -1>/8")
   .room(0.5).roomsize(7).rlp(5000).gain(.2)
   // the snare advances the tunnel
   .uniformTrigger("moveFWD:slow:20:gain:4")
$: note("<C D G A Bb D C A G D Bb A>*[2,2.02]")
  .clip(1.1)
  .transpose("<-12 -24 -12 0>/8")
  // .sound("sawtooth")
  .sound("triangle")
  .cutoff(perlin.slow(5).range(20,1200))
  .room(.8).roomsize(.6)
  .gain(.4)
  // each piano note modulate one of the 8 truchet patterns
  .uniformTrigger("modulations:gain:.5:slow:20")

[TristanCacqueray]

I meant I couldn't figure how to use the existing functions like gain or slow, because they get applied to the audio pattern. The uniform pattern is a bit different since it capture the values and send them to the shader. In that situation, how could we re-use the existing functions like gain but for the uniform pattern?

[TristanCacqueray]

I think I get how to handle the uniform's option as pattern. So the API would look like this:
uniform({name: "iColors", gain: "<2 1 1 1>", slow: "10"}). The uniform method can stack the patterns from the options, and by using context I can get their values for each outer hap.

With the last commit 1baa6da, here is how the demo looks like:

let truchetFTW = await fetch('https://raw.githubusercontent.com/TristanCacqueray/shaders/refs/heads/main/shaders/Truchet%20%2B%20Kaleidoscope%20FTW.glsl').then((res) => res.text())
await loadShader(truchetFTW)

setcpm(96)

$: s("bd").bank("RolandTR808")
   .gain(2).dist("<1 .7 .7 .7>")
   .mask("<1@30 0@2>")
   .uniform({dest: "rotations:seq", gain: "<0.1 0.3>", onTrigger: true})
$: sound("hh*4").bank("RolandTR808")
   .room(.3).gain(".25 .3 .4")
   .mask("<0@8 1@32>")
   .uniform({dest: 'icolor', gain: 0.5, slow:"5", onTrigger: true})
$: s("cp/8").bank("RolandTR808")
   .uniform({dest: 'moveFWD', slow: "<40 20>", gain: "<8 4>", onTrigger: true})
   .hpf(500).hpa(.8).hpenv("<-3 -2 -3 -2 -1>/8")
   .room(0.5).roomsize(7).rlp(5000).gain(.2)
$: note("<C D G A Bb D C A G D Bb A>*[2,2.02]")
  .clip(1.1)
  .transpose("<-12 -24 -12 0>/8")
  // .sound("sawtooth")
  .sound("triangle")
  .cutoff(perlin.slow(5).range(20,1200))
  .room(.8).roomsize(.6)
  .gain(.4)
  .uniform({dest: "modulations", gain: 1.5, slow: 50, onTrigger: true})

[felixroos]

looks pretty neat! I think the difficulty for using pattern methods is that the uniform values are stateful, also expected to retain state between evaluations. It is theoretically possible with patterns to do a similar thing:

https://strudel.cc/#bGV0IGNvdW50ID0gMTsKCiQ6IHMoImJkKjw0IDY%2BIikKICAub25UcmlnZ2VyKCgpID0%2BIGNvdW50KyssIGZhbHNlKQogIC5zcGVlZChzaWduYWwoKCkgPT4gY291bnQlOCsyKSkK

I'm not sure if it's a good idea though.. Just playing a bit with the idea, you could maybe also pass uniforms as the second argument of loadShader:

let iColorOffset = 1;

await loadShader(`
  uniform float iColorOffset;

  void mainImage( out vec4 fragColor, in vec2 fragCoord ) {
    vec2 uv = fragCoord / iResolution.xy;
    vec3 bcol = vec3(0.1,2.1,4.2);
    vec3 col = 0.5 + 0.5*cos(iColorOffset+uv.xyx+bcol);
    fragColor = vec4(col, 1);
  }
`, () => ({
  iColorOffset
}))

$: s("bd*<4 6>")
  .onTrigger(() => iColorOffset++, false)

This would allow implement any custom stateful logic in userland. What do you think?

[TristanCacqueray]

Thank you for following up on this feature; I appreciate your feedback!

I think there is room for experimentation as it feels like we are in uncharted territory. I am not sure how many uniform variables could be used in a meaningful way, and perhaps a skillful shader artist could use every pattern to create pleasing visuals. So I hope this feature will be as flexible as possible to avoid limiting explorations.

What I have proposed so far is based on my work with animation-fractal which mostly uses NoteOn events. In this project, I found that it is important to control the rate of change to avoid unpleasing stroboscopic effects and this is why I made this the default behavior with the slow option. I haven't tried using pattern values, like from the pan or the dist, but that is already supported.

Using a regular strudel pattern to activate variables sounds ideal because we can leverage the existing modifiers. For example: .uniform("rotations".mask("<1@3 0@1>")). Though I find the proposed solution for setting the slow and gain to be somewhat clunky. Also it seems that I'm not using stack correctly and the uniform mask gets wrongly applied to the outer pattern.

I like how explicit your uniform argument var example looks like. I guess we can also globalThis the uniforms so that they are available by default. I can update the PR later this week, and I'd be happy to remove the uniform function as it doesn't have to be part of this initial change.

[TristanCacqueray]

I think the difficulty for using pattern methods is that the uniform values are stateful, also expected to retain state between evaluations.

Right, using withValue or withHap doesn't work as I expected: the callback is not synced and we can't use that for setting uniform values. Could we have a withCurrentValue to get the synced value, e.g. the one that is highlighted in the code with a white box? I tried using setTimeout and the strudeltick, but I can't tell if and how that would work...

So I removed the uniform pattern function for now, and changed loadShader to return the uniforms so that they can be manually set on trigger as you suggested. Please find the updated demos in the top comment, let me know what do you think.

[TristanCacqueray]

• we could potentially get rid of picogl ?! i've done some experiments with webgl and it's not a lot of code that's needed to spin up a shader: https://github.com/felixroos/schattenspiel having less dependencies is less to worry about, but also not that big of a deal

I started to remove picogl, that's better indeed, and that should let us try the feature directly on the strudel's version currently deployed.

[TristanCacqueray]

Trying to keep the ball rolling, I made the module standalone and uploaded a demo here: https://strudel.cc/?ErKvnkz3z_3p . Next I'll address the todo list, please let me know if the new design looks good to you.