Add SmartEEPROM Support #49

TheZoc · 2021-01-31T23:08:25Z

This is a rewrite of #16 to be used with qmk/qmk_firmware#6068, as a cleanup and an easier to read version, after looking up a bunch of stuff in the manual.

I fixed a couple issues while doing this:

I removed the part that read and wrote the same value to DSU_STATUSB as it was redundant.
I modified the bit that reads NVMCTRL_CTRLA, as it seems to be a bug: It was reading 32 bits out of 0x41004000 (i.e. both CTRLA and CTRLB data), and toggling off 0xf0, which would change the CTRLB data.

Also, it's not more verbose, to aid the user pinpoint if anything go wrong - as previously it would silently fail - and it will show a success message if everything works correctly.

As a last addition, I added support to --restart, as it was requested in the other PR.

Please note that all credits of this work should be attributed to @daltona, as he was the one that made this possible :)

sigprof

This looks better than the original PR (in particular, there are less unexplained magic numbers in the code). There still might be some things that could be improved.

mdloader_common.c

sigprof · 2021-02-28T07:52:59Z

mdloader_common.c

+    {
+        if (!write_word(NVMCTRL_USER + i * 4, data[i]))
+        {
+            printf("Warning: Unable to write NVMCTRL_USER page %i\n", i);


This should be a fatal error, otherwise the subsequent code could write some random data to the user row.

Suggested change

printf("Warning: Unable to write NVMCTRL_USER page %i\n", i);

printf("Error: Unable to write NVMCTRL_USER page %i\n", i);

return 0;

Changed code to treat this as a fatal error.

mdloader_common.c

sigprof · 2021-02-28T08:23:59Z

mdloader_common.c

+        puser_row1->bit.SBLK = 0x2; // 2 blocks
+        puser_row1->bit.PSZ = 0x1;  // 8 bytes


According to the Table 25-6 from the datasheet, this configuration is sub-optimal — some part of flash memory which is allocated for SmartEEPROM would always remain unused (apparently the maximum number of SEEP per SEES is limited to 144, even though that fact is not explicitly documented in the datasheet). However, increasing PSZ would mean that small writes to the QMK settings part of EEPROM would result in more flash writes, because every 1→0 bit change would result in allocating a larger page. One option is to decrease SBLK to 1, which apparently would result in the same wear leveling behavior while occupying less flash size.

Okay, so I re-read the code and the manual regarding this. I'm setting it now to:

puser_row1->bit.SBLK = 0x1; // 1 block puser_row1->bit.PSZ = 0x1; // 8 bytes

According to table 25-6, this will set the virtual size to 1024 bytes, in the optimal size.
Looking at qmk/qmk_firmware#6068, it seems it checks to see if either SBLK or PSZ is 0 - and if any of them are, disable the functionality.
Looking further into it, it seems the user configuration just requires 36 bytes for storage, so this could probably be set to:

puser_row1->bit.SBLK = 0x1; // 1 block puser_row1->bit.PSZ = 0x0; // 4 bytes

Yielding a 512 bytes storage, in optimum setting - but this would require an update to the QMK PR.

For the meantime, I'm changing to code to the first option, but if you think we should push this further to the 2nd option, I can do the change - just waiting on your feedback.

One of important users of EEPROM is VIA, and 1024 bytes is a commonly used EEPROM size (e.g., the ATmega32U4 chip (probably the most popular MCU for custom keyboards) has 1024 bytes of EEPROM). 1024 bytes is usually enough to store 4 keymap layers for a TKL keyboard, and going below that would be problematic. As for going to 2048 bytes of EEPROM instead, I'm not really sure — this would allow fitting more VIA keymap layers while occupying the same amount of flash as SBLK=2, PSZ=1, but maybe page rewrites for the settings part would happen more often because of larger page size. However, actually using the extra EEPROM space would require some changes in the QMK code (the current PR does not even try to support VIA, providing support just for a minimal amount of EEPROM required for the QMK core).

And actually the table gives 512 bytes as an optimum size for SBLK=1, PSZ=0 only because there is no way to go any lower than SBLK=1 — some flash space is still wasted in this configuration, so there is probably no point in going that low.

@sigprof Fair enough, looks good for this PR, then.

Still, I didn't know about VIA existence (this one, right?), so might be a good idea to plan ahead for it.
Where can I get more info about VIA and/or how to future proof this PR to support it?

I can happily create a new PR in qmk repo to support it, if you give me some advice on how to proceed :)

From the EEPROM side of things VIA just needs a large enough EEPROM (1024 bytes is enough, although 2048 might be better for a layout larger than TKL) with fast enough read access (every keymap lookup generates several EEPROM reads; there is no RAM cache, because AVR chips might not have enough RAM for that). Apparently SmartEEPROM already provides a RAM cache for the emulated EEPROM, therefore the read access time should not be a problem.

@TheZoc @quaddy I kinda got VIA working on the CTRL thanks to all of the contributors awesome work using the following steps:

(moved complete step-by-step "guide" to #49 (comment) for readability, hadn't figured out the layout for VIA at this point)

Now I remember that I spent a significant amount of time creating the layout file.
You can find the row column in the source code, the keymaps is organized as ROW, COL so it is a matter of creating the correct layout and filling the row, column accordingly

[0] = LAYOUT( COL0, COL1 etc... | | ROW0 => KC_ESC, KC_F1, KC_F2, KC_F3, KC_F4, KC_F5, KC_F6, KC_F7, KC_F8, KC_F9, KC_F10, KC_F11, KC_F12, KC_PSCR, KC_SLCK, KC_PAUS, ROW1 => KC_GRV, KC_1, KC_2, KC_3, KC_4, KC_5, KC_6, KC_7, KC_8, KC_9, KC_0, KC_MINS, KC_EQL, KC_BSPC, KC_INS, KC_HOME, KC_PGUP, ROW2 => KC_TAB, KC_Q, KC_W, KC_E, KC_R, KC_T, KC_Y, KC_U, KC_I, KC_O, KC_P, KC_LBRC, KC_RBRC, KC_BSLS, KC_DEL, KC_END, KC_PGDN, ROW3 => KC_CAPS, KC_A, KC_S, KC_D, KC_F, KC_G, KC_H, KC_J, KC_K, KC_L, KC_SCLN, KC_QUOT, KC_ENT, ROW4 => KC_LSFT, KC_Z, KC_X, KC_C, KC_V, KC_B, KC_N, KC_M, KC_COMM, KC_DOT, KC_SLSH, KC_RSFT, KC_UP, ROW5 => KC_LCTL, KC_LGUI, KC_LALT, KC_SPC, KC_RALT, MO(1), KC_APP, KC_RCTL, KC_LEFT, KC_DOWN, KC_RGHT ),

@daltona thats what I was first going to do as well, however according to https://github.com/qmk/qmk_firmware/blob/master/keyboards/massdrop/ctrl/config.h#L31-L33 we have 11 rows and 8 cols - I guess this means the hardware has it flipped around?

The second problem is, the keymap as it is in the code has a maximum of 6 rows and 17 cols, which doesn't fit the given rows/cols.

I guess https://github.com/qmk/qmk_firmware/blob/master/keyboards/massdrop/ctrl/ctrl.h#L12-L32 is the actual layout of the rows/cols, I'll try using that as a reference and add some trial and error on top.

@kaeltis You're right sorry for the confusion, in fact the 6 rows that I believe existed are in fact split.

to get the matrix layout you should replace all Kxx in the layout view ( the parameters of the macro) with the position in the matrix of the corresponding keycode:

0,0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 6,0 6,1 6,2 6,3 6,4 6,5 6,6 6,7 1,0 1,1 1,2 1,3 1,4 1,5 1,6 1,7 7,0 7,1 7,2 7,3 7,4 7,5 etc...

@daltona thanks! With your help I figured it out :)

Here's the Massdrop CTRL ctrl.json for VIA:
https://gist.github.com/kaeltis/c81ffbe79d7e8a92218dd8a610db39b7

Here's the layout on KLE if anyone needs it:
http://www.keyboard-layout-editor.com/#/gists/df6c13bb95885418bf5c6c8f280e93aa

mdloader_common.c

mdloader_common.h

mdloader_common.c

mdloader_common.h

quaddy · 2021-03-01T08:12:08Z

Sadly I cannot help with this PR because I'm not into C++. But from a users perspective I'm hoping to see this feature soon on my CTRL! 🚀

TheZoc · 2021-03-03T00:40:08Z

@sigprof Went through all your suggestions and added all them. Thank you very much!

I did change the SBLK to 0x1 to get the optimal configuration, and I just updated the SmartEEPROM on my keyboard:

D:\GitHub\mdloader\Debug› .\mdloader.exe --smarteep --forceeep --restart --first
Massdrop Loader 1.05

Massdrop Loader  Copyright (C) 2018-2020 Massdrop Inc.
This program is Free Software and has ABSOLUTELY NO WARRANTY

Scanning for device for 60 seconds
..........
Device port: COM3 (SAMD51J18A)

Opening port 'COM3'... Success!
Found MCU: SAMD51J18A
Bootloader version: v2.18Sep  4 2018 16:48:28
user row: 0xfe9a9239 0xaeecff92 0xffffffff 0xffffffff
SmartEEPROM already configured - SBLK: 0x0002 - PSZ: 0x001.
--forceeep enabled, proceed.
NVMCTRL.CTRLA: 0x0004
        AUTOWS: 0x1
        SUSPEN: 0x0
        WMODE: 0x00
        PRM: 0x00
        RWS: 0x0000
        AHBNS0: 0x0
        AHBNS1: 0x0
        CACHEDIS0: 0x0
        CACHEDIS1: 0x0
Configuring SmartEEPROM... Success!
Booting device... Success!

D:\GitHub\mdloader\Debug› .\mdloader.exe --smarteep --restart --first
Massdrop Loader 1.05

Massdrop Loader  Copyright (C) 2018-2020 Massdrop Inc.
This program is Free Software and has ABSOLUTELY NO WARRANTY

Scanning for device for 60 seconds
........
Device port: COM3 (SAMD51J18A)

Opening port 'COM3'... Success!
Found MCU: SAMD51J18A
Bootloader version: v2.18Sep  4 2018 16:48:28
user row: 0xfe9a9239 0xaeecff91 0xffffffff 0xffffffff
SmartEEPROM already configured - SBLK: 0x0001 - PSZ: 0x001.
Use --forceeep to force to configure it anyway.
Booting device... Success!

If anyone used this PR before and want to improve the memory usage/wear level, just need to run like I did with the parameters --smarteep --forceeep.

(Also, if you're wondering how I did that on Windows, check #50 )

daltona · 2021-03-09T10:28:18Z

Sadly I cannot help with this PR because I'm not into C++. But from a users perspective I'm hoping to see this feature soon on my CTRL! 🚀

I would love creating the VIA config file for CTRL, but i do not have a CTRL keyboard to test with ;)

kaeltis · 2021-08-18T18:26:28Z

I got VIA working on the CTRL thanks to all of the contributors awesome work using the following steps:

Patch & Build mdloader

Clone https://github.com/Massdrop/mdloader
Apply Add SmartEEPROM Support #49 (https://github.com/Massdrop/mdloader/pull/49.patch) - git am 49.patch
Apply Added support for Visual Studio 2019 #50 (https://github.com/Massdrop/mdloader/pull/50.patch) - git am 50.patch
Build patched mdloader with VS2019
Set SmartEEPROM to the current implementation (I was using add option that will update user row to enable smarteeprom #16 before) - mdloader.exe --first --smarteep --forceeep --restart

Patch & Build QMK

Clone https://github.com/qmk/qmk_firmware
Apply massdrop alt/ctrl: support saving into nvm qmk/qmk_firmware#6068 (https://github.com/qmk/qmk_firmware/pull/6068.patch) - git am 6068.patch
Add a keymap for VIA and applied the EEPROM_SIZE 1024 mentioned here Add SmartEEPROM Support #49 (comment) (board would not start with the default calculated value) - you can see my commit at kaeltis/qmk_firmware@c90b2b8 or qmk/qmk_firmware@master...kaeltis:master
Build qmk with qmk setup kaeltis/qmk_firmware (or your fork) and qmk compile -kb massdrop/ctrl -km via
Flash qmk using mdloader.exe --first --download massdrop_ctrl_via.bin --restart

Use VIA

Start VIA
Click on File -> Import Keymap
Import the json file from here: https://gist.github.com/kaeltis/c81ffbe79d7e8a92218dd8a610db39b7

After that you should be able to reassign keys in VIA, lighting currently does not work due to the custom implementation (afaik).

For reference, here's the layout on KLE if anyone needs it:
http://www.keyboard-layout-editor.com/#/gists/df6c13bb95885418bf5c6c8f280e93aa

TheZoc · 2021-10-05T18:01:59Z

Small update:

Fixed conflict
Now extra information about SmartEEP is only displayed with --verbose or -v

TheZoc · 2021-10-13T20:56:56Z

Closed in favor of #62

This was referenced Jan 31, 2021

add option that will update user row to enable smarteeprom #16

Closed

massdrop alt/ctrl: support saving into nvm qmk/qmk_firmware#6068

Merged

TheZoc force-pushed the master branch from d053007 to 554275e Compare February 1, 2021 01:08

sigprof suggested changes Feb 28, 2021

View reviewed changes

TheZoc force-pushed the master branch 2 times, most recently from c70bae9 to f973cea Compare March 3, 2021 00:33

ewersp mentioned this pull request Jun 1, 2021

[Keymap] Updates to personal Drop ALT keymap qmk/qmk_firmware#13058

Merged

14 tasks

zvecr mentioned this pull request Sep 30, 2021

Add SmartEEPROM Support #62

Merged

TheZoc force-pushed the master branch from f973cea to 65ad1f0 Compare October 5, 2021 17:51

Added SmartEEPROM Support

0fca7d2

TheZoc force-pushed the master branch from 65ad1f0 to 0fca7d2 Compare October 5, 2021 18:00

TheZoc closed this Oct 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SmartEEPROM Support #49

Add SmartEEPROM Support #49

TheZoc commented Jan 31, 2021 •

edited

Loading

sigprof left a comment

sigprof Feb 28, 2021

TheZoc Mar 2, 2021

sigprof Feb 28, 2021

TheZoc Mar 3, 2021

sigprof Mar 3, 2021

TheZoc Mar 5, 2021

sigprof Mar 6, 2021

kaeltis Aug 18, 2021 •

edited

Loading

daltona Aug 18, 2021 •

edited

Loading

kaeltis Aug 18, 2021 •

edited

Loading

daltona Aug 18, 2021

kaeltis Aug 18, 2021

quaddy commented Mar 1, 2021

TheZoc commented Mar 3, 2021 •

edited

Loading

daltona commented Mar 9, 2021

kaeltis commented Aug 18, 2021 •

edited

Loading

TheZoc commented Oct 5, 2021

TheZoc commented Oct 13, 2021

	printf("Warning: Unable to write NVMCTRL_USER page %i\n", i);
	printf("Error: Unable to write NVMCTRL_USER page %i\n", i);
	return 0;

		puser_row1->bit.SBLK = 0x2; // 2 blocks
		puser_row1->bit.PSZ = 0x1; // 8 bytes

Add SmartEEPROM Support #49

Add SmartEEPROM Support #49

Conversation

TheZoc commented Jan 31, 2021 • edited Loading

sigprof left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaeltis Aug 18, 2021 • edited Loading

Choose a reason for hiding this comment

daltona Aug 18, 2021 • edited Loading

Choose a reason for hiding this comment

kaeltis Aug 18, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

quaddy commented Mar 1, 2021

TheZoc commented Mar 3, 2021 • edited Loading

daltona commented Mar 9, 2021

kaeltis commented Aug 18, 2021 • edited Loading

Patch & Build mdloader

Patch & Build QMK

Use VIA

TheZoc commented Oct 5, 2021

TheZoc commented Oct 13, 2021

TheZoc commented Jan 31, 2021 •

edited

Loading

kaeltis Aug 18, 2021 •

edited

Loading

daltona Aug 18, 2021 •

edited

Loading

kaeltis Aug 18, 2021 •

edited

Loading

TheZoc commented Mar 3, 2021 •

edited

Loading

kaeltis commented Aug 18, 2021 •

edited

Loading