[CLI] Add c2json

[Erovia]

Description

First of all, I'd like to thank both @lf- for starting to work on this feature in #7218 and @skullydazed for suggesting a Pygments-based approach.

This submodules attemps to parse a keymap.c file and generate a keymap.json for the user.
Sadly this is not as easy as the other way around, so parsing is limited. Sorry drashna. :)

Types of Changes

• Core
• Bugfix
• New feature
• Enhancement/optimization
• Keyboard (addition or update)
• Keymap/layout/userspace (addition or update)
• Documentation

This PR intends to supersede both

• Add support for converting keymap.c -> keymap.json #7218
• CLI: Add basic json keymap support #8498

Checklist

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• I have tested the changes and verified that they work and don't break anything (as well as I can manage).

[Erovia]

CLI CI fails because qmkfm/base_container has old qmk.

[fastjames]

This feature would be a huge benefit to me -- I'm fairly new to QMK and I have trouble keeping my layouts straight in my head, especially when I'm making small changes. @Erovia is there any specific task I could help out with here? I'm not super-familiar with travis / docker but if you have any more insight on what's failing I am happy to dive in as deep as I can.

[Erovia]

Thank you for your interest!
This feature is still on my TODO list, so it's by no means forgotten, just life took priority.

I'll need to rebase and fix the conflicts. Also, I think this feature could benefit greatly from this PR, but I think I'd rather see this merged as is and improve it iteratively.

Ultimately, this just needs testing. (and the rebasing I mentioned)

[fastjames]

Thanks for the fast follow-up -- I definitely understand the priority. I'll try using this branch locally with my own keymap work, to see what results I get.

[Erovia]

That would be great.
I'll try to do the rebase today, no promises though. :)

[Erovia]

@fastjames Rebased and fixed up the code. If you have some time to try it out, I'd appreciate the feedback.

[fastjames]

Thanks very much for freshening this up. I had a few minutes today so here's the test procedure I followed:

1. Check out your branch (cli/c2json)
2. cherry-pick in my commit with my custom crkbd/rev1 keymap
3. bin/qmk c2json -kb crkbd/rev1 --no-cpp keyboards/crkbd/keymaps/fastjames/keymap.c > ~/fj-crkbd.json
4. Visit https://config.qmk.fm/#/crkbd/rev1/LAYOUT
5. Click on Import QMK Keymap JSON file
6. Select the file generated in step 3
7. Observe results

Off the bat, I got a generic alert stating Sorry, this doesn't appear to be a QMK keymap file. Since I didn't have a good starting place for troubleshooting, I exported the default crkbd layout (using the handy nearby button) and I compared that to the c2json generated file. I noted a couple of potentially significant differences:

1. The c2json-generated file had spaces between the key-value separators and the values. I don't know if it's a dealbreaker, but the default crkbd file that I downloaded did not have them.
2. The c2json-generated file was missing the following JSON elements: version, notes, documentation, and keymap. I wasn't sure if any of these were significant so I started adding them to the generated file one at a time. The one that seemed to make a difference was keymap. Adding that got me past the previously described error alert.

I think I hit success after that! I can see the keymap in the configurator, and the layers appear to match what I recall setting up. I hope this testing information helps, and I'm happy to take a shot at making the changes if you're busy.

[elfmimi]

I tried c2json for my keymap for ARM target .
I knew that ARM does not need PROGMEM attribute so I omitted it when I wrote that keymap.
In this case c2json fails with unhelpful message: "Something went wrong" .

[ymotongpoo]

Mentions #6877 here for visibility.

Thank you @Erovia for creating this pull request. This is what I long for in qmk command. @skullydazed can we help anything to move this PR forward?

[Erovia]

Rebased and fixed the missing keymap value. Thanks @fastjames for the awesome comment/bug-report! It helped a great deal!

@elfmimi: For the time being, I'd rather not complicate the parsing with edge-cases like the missing PROGMEM. In the future me or someone else might get back to it.

@ymotongpoo

Sorry for the ping, but thought you guys might like to know that I'm still working on this :)

[Buckwich]

@Erovia Thank you so much for your work, I used this to build a list of all keymaps (https://github.com/qmk-helper/qmk-database) which is used for a keyboard/keymap browser. Because of this I am hoping this PR gets merged soon so I can automate the generation of JSONs.

Another issue besides the keymap field are layer modifiers with integers as identifier MO(1) due to the c processor, which are required for a correct function of the QMK configurator. I am no expert with lex but I think I solved this issue with these lines:

        else:        
            if is_adv_kc:
                layer['keycodes'][-1] += line[1]
            elif line[0] is Token.Literal.Number.Integer and is_keymap:              
                if not layer['name']:
                    layer['name'] = line[1]

One example keymap with this issue is thevankeyboards/minivan:default

keymap.c (simplified):

#define _QW 0
#define _L1 3
const uint16_t PROGMEM keymaps[][MATRIX_ROWS][MATRIX_COLS] = {
  [_QW] = LAYOUT(   
    MO(_L1)
  )
}

keymap.c preprocessed:

const uint16_t PROGMEM keymaps[][MATRIX_ROWS][MATRIX_COLS] = {
  [0] = LAYOUT(   
    MO(3)
  )
}

keymap.json, no-cpp, correct

{"layers": [["MO(_L1)"]]}

keymap.json, with cpp, actual result

{"layers": [["MO()"]]}

keymap.json, with cpp, expected result

{"layers": [["MO(3)"]]}

[Erovia]

@Buckwich thanks for giving this subcommand a proper test-drive and use-case! :D
Double-thanks for the bug report and for even providing a fix.

With ddacab8 those pesky numbers should be handled. Can you give it a spin?

[Buckwich]

Looks good to me, only the position of the keymap name moved in the json (to the correct position). Numbers are working. Thank you for the quick fix.

[ymotongpoo]

woohoo!! Thank you @Erovia, @Buckwich and @skullydazed for merging this into master!