Feature Request midi 14bit (fine adjustment CC/CC+32 pairs)

[jpnielsen]

Please make it possible to output 14-bit midi over USB.

I don't need NRPN, only the "fine adjustment" allready in the conventional MIDI controller specification, where any of the first 32 controls can be paired with a control offset 32 higher. I found an example here:

fixed CC's are OK for me {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15} paired with {32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47}.
I plan on using it with VCV Rack 2

Cheers,
Jens Peter

Ps: I bought the AtoVproject-Reworked - and asked them too.

[infovore]

Thanks for the request. It is not a trivial one: a large amount of the 16n firmware assumes that 7-bit data is being used everywhere. The internal data is filtered down to 7-bit quite early on in the analog read. To support this, we'd need to port to 14-bit everywhere, and then have a toggle to filter down, as well as update the MIDI output code.

It also would necessitate a patch to the editor, so as to ensure that CCs could be matched with pairing 'MSB' CCs correctly. I'm not interested in offering an alternative mode that has hardwired CCs, given the 'regular' mode is editable.

I'd welcome PRs to both editor and firmware that approach a solution, though, if someone wants to take it on.

[Dewb]

I've been meaning to ask about appetite for a PR on this and somehow didn't see (or more likely, forgot about) this thread! I have been using this 14-bit branch for a while now main...Dewb:16n:14bitCC

It implements fixed CCs (0-31 are always 14-bit, paired according to spec with the Cc number +32 higher), but I can tackle the editor patch as well to make it configurable.

Some questions about that:

1. Do you have any thoughts on how you'd prefer to see the configuration structure expanded? A simple option would be to add two bytes supporting one bit per fader for a "14-bit mode" checkbox on each. But a more extensible approach might be to reserve 4 or 8 bits per fader for a "output format" enum, which would initially just have two options "standard CC" or "14-bit CC", but could eventually allow for other options like RPN, NRPN, pitch bend, etc.

2. I assume with the release of the 16nx, and the fact that the editor is shared between the two, that this feature would need to be added to the 16nx firmware as well (or disabled in the editor), right? I am planning on building a 16nx sometime this winter so I can tackle that too, though I wonder if it's as valuable in the 16nx since the DAC is only 12 bit. (I have some vague plans to try adding a 16-bit external DAC to my 16nx board order.)

[infovore]

So, I appreciate the editor made things a little harder for hacking/futzing, so let me think out loud to document process:

• the memory map of the dataset is quite tight, and the memory map we use corresponds to the data being sent down the pipe over Sysex - ie, it splats the sysex data into EEPROM.
• EEPROM on a 3.2 is 128 bytes; on a 3.2 is 2048 bytes. Currently, on an nx - not the focus of this repo - the virtual EEPROM is set up as (IIRC) 256 bytes, but I could alter that.
• If we implement it with a 16-bit bitmask, a bool for each fader, to go "this fader is high-res", I think we have space. We warn the user if they've picked a 'conflicting" CC in the UI (eg: if you have set 32 to high res, and also picked 64, we need to warn you and stop you uploading config).
• this still changes the memory map, and would require every 16n user to update their firmware in order to continue using the editor. To my mind, that is a big ask and not one I'd take lightly. (Later in this post I might see a way around this)

Here's the other thing: I personally am not convinced 14-bit data is reasonable with the hardware provided. Let me argue with myself:

• the fader is 60mm.
• with 16384 steps, that equates a step to 3 micrometers, assuming there's no jitter/noise.
• the Teensy ADC is 14-bit, yes, but realistically, that means somewhere between 10-12 usable bits of data as a rule of thumb. I genuinely think two of those bits just disappear to noise/mechanical jitter. So whilst I could give you 14-bit data, I think it's meaningless.
• I definitely appreciate that it beats 7 bit data, which is not many steps for smooth musical control, and below the spec of the ADC. But a 14-bit ADC doesn't guarantee 14 bits of usable data. But yes, > 7 bits is definitely desirable and possible

That said: 10 bits of usable data, scaled up to 14 (<<4) is still notable smoother than 7 bits (0-127) of the CC spec. That logic means this still could work with the 12-bit ADC of the RP2040 in the 16nx: read 12 bits of data, bit-shift it up, noise aside, it's still a higher-res improvement on the 7-bit data.

Adding two bytes to the data payload is a reasonable ask there; the editor tweak is a faff but doable. There's also a way I can see to keep some backwards compatibility, but I need to think about that harder.

So: I'm actually inclined to explore this now I've argued myself into the two-bit version. I'm also happy to explore in parallel with 16nx. And it might be an opportunity to port the editor to Svelte 5, replace some ungainly stores with runed variables.

Interesting. This week I'm jam-packed but it could be a thing we explore. @Dewb , what do you think?

[Dewb]

Awesome! Yes, I'm happy to iterate on this on whatever timeline. I've been sitting on this for a while so no urgency, but lately I'm trying to make an effort to finish & upstream dangling projects like this.

I do think we have empirical evidence that the 14-bit data is valuable, despite the limitations of the hardware, from all the Teletype i2c users. I'm one, and I'm pretty happy with it -- you definitely can't reliably hit a specific integer in the 0-16383 range, but it's smoother than 12 bits. (Isn't the Teensy 3.2 DAC 16-bit, with 13 bits usable?) If it's useful for Teletype users I don't see why MIDI users shouldn't also have the option on Teensy hardware. And yes, exactly, even if it's only 10 or 12 bits usable with the RP2040, ten bits is a lot better than seven, and left-shifting to 14 is still appropriate and expected, to map the entire range, even if it jumps in the low bits.

The editor backward compatibility issue seems solvable. You've already got a version field in the config struct/type, and if the editor always requests configuration before sending configuration, you could select the appropriate configuration struct and hide/show UI based on what the firmware supports. This of course makes the editor a little more complicated, but that seems inevitable unless the featureset remains static.

[infovore]

@Dewb You're right, the editor could version features relatively easily - and actually also do stuff per device. We do actually have a config data per device that says what it's capable of, so it's easy to add "supports hi-res output".

I'd like to take this on, I think, it's a nice way to refresh the editor code a little.

I'm also agreed on the idea that 14-bit output is not about 14-bit adcs, or 14 bits of information; it's about more than 7 bits of information.

And I am happy that this can be added with minimal impact to older devices.

It's likely the bitmask will be inverse, simply because 0xFF is the default state of erased memory... making it not necessary to write new bitmasks for every device.

[Dewb]

Great! So you'll handle the editor work? I can update my branch with the proposed config struct change and draft a PR.

[Dewb]

Couple of observations now that I've done a pass over the code:

• it's four additional bytes, not two, because we need the flags for both USB and TRS;
• four bytes isn't enough if the EEPROM layout and the sysex messages are identical, because each byte of the sysex payload has to have its high bit low.

Having said that though, I took a stab at the config support (code here: d7a6d8d), and aside from the high bit problem, this strategy appears to work fine in limited testing (sending sysex from Max to configure.) By adding a check if the sysex message is shorter than expected, I can still use the existing editor to configure everything except the new flags.

[infovore]

@Dewb thank you! This is looking interesting. I agree on four extra bytes, absolutely fine with that, still fits into memory on a Teensy LC.

So this is probably going to take longer to merge in than you'd like, but here's what I propose:

• I work out a way of adding this into the editor elegantly, whilst also possibly patching the editor / giving it a good overhaul. (It feels like a worthwhile time to do this). It'll probably look like a checkbox on each column, and then updating the display to work displaying high-res fader values/debugging, which is a bit harder. I also gate the feature behind a firmware-version on the device (which is not something implemented yet).
• With that done, we review the PR and see if it all plays ball
• Simultaneously, I work on a similar patch for 16nx
• Release the lot together.

I'll be honest and say I'm quite chocka right now, but also quite enthuisastic as a way into this, and it looks like your PR is in a really good place, so it's something nice to frame the editor against.

[Dewb]

Awesome! Again, no rush. If we get confident in the sysex layout, I could release the 14-bit support in VCV Rack before the editor is finalized (Rack could handle setting the 14-bit flags as requested.)

If the editor will keep you busy, I can probably also take a swing at the 16nx firmware changes once I get around to building one.

We do need to solve point 2 above though, I should have called that out more clearly. With a 16 bit field we can’t send that over sysex as two bytes; the high bits have to be zero, so it’s currently impossible to set fader 8 or 16 high in the bit field. I see a couple strategies: the bit field could be split into 3 or 4 bytes instead of 2, or the bitfield could remain densely packed in EEPROM and it could be split across twice as many bytes when going in and out over sysex. The latter would be more space efficient, but it would make the sysex de/serialization code more complex.

[infovore]

Urgh, you're right about the sysex issue - curse the stupid 7-bit spec. I will think on that.

[Dewb]

Another possible solution: there are four three unused bits in each channel config byte, since 0-15 only uses the lower four bits. The unused bits 4-6 could become a "mode" enum for each channel, where 000 is normal CC, 001 is 14-bit CC, and other values could be reserved for future pitch bend/NRPN support, etc. Then this change wouldn't consume any additional space. The only risk is that any software that interprets the 16n sysex that isn't already ignoring those top bits / constraining the channel byte to legal values will need to be updated.

[Dewb]

Ah, it looks like MIDI channel is being stored as 1-16, not 0-15, so that only leaves two unused bits.