Split Firmware off from Keymaps

[exiva]

I think the main qmk firmware and the keymaps should be put into their own repos and have a 'tap' setup sort of like Homebrew has? The current setup makes it extremely difficult to keep up with any changes to the base firmware since new keymaps and fixes to them pollute the commit log. Would also have the benefit that you could only 'tap' the keymaps for the keyboard you have, or could host on your own repo.

Just a thought I've had for a while now.

[exiva]

This might also help make #1038 easier to deal with.

[jackhumbert]

Having them in a branch of qmk_firmware may work, but this is interesting. What would you propose in terms of the tapping? Do you have a script in mind for this, or would it just be a git submodule thing?

Any sort of fancy thing like this would make everything more complicated for new-comers, though. Having everything in one (albeit large) repo does make it a lot easier to get started.

[exiva]

What would you propose in terms of the tapping? Do you have a script in mind for this, or would it just be a git submodule thing?

I'm not sure what would work the best, my instinct leads me to want to write something in python to manage it all, again, similar to homebrew. But maybe just a submodule would be easiest.

Any sort of fancy thing like this would make everything more complicated for new-comers, though.

I think it may be a safe assumption that if someone is looking for a new keymap or to make their own, they probably can handle an extra repo. At minimum though, keep board configs and a basic qwerty layout for each board in the main firmware repo, then put community keymaps in their own.

...Or, you could leave the main supported qmk boards in the firmware repo, and move 3rd party to external, etc. There's a few ways for something like this to go down, what's best though, idk.

[PureSpider]

My suggestion would be to include the main firmware as a submodule to the keymap repo.
That way users can just checkout the keymap repo and don't need to worry about anything.
If advanced users don't want that clutter, they can just use the firmware repo as a standalone or submodule in an own repo and add their own keymaps on top.

[fredizzimo]

Personally I prefer it the way it is now, at least for maintainers. When everything is in the same repository, it's easy to search where features are used, and even to make global refactoring. But if things were to be changed we should at least at the minimum still support building everything with a single make command to make sure everything compiles, and also make sure that we still build everything as a part of the continuous integration server.

The biggest problem with submodules is that in many cases, you have to make two pull requests. First for the actual QMK feature or fix, and then for the keymap that relies on it. You would also actually have to wait for the QMK to be merged so that you can point the kemap submodule to the right path and version. And if there are several people doing all this is parallel we will have many complicated merge problems. At least that's my experience with submodules, but there might be a better way to use them.

As for the changelog, that could indeed be a problem. The best way currently to get an idea what has been changed is probably to check list of merged pull request rather than the actual commit log. It's also possible to filter out all keymaps when you view the GIT log, at least by using the GIT command line.

The optimal solution would be if someone had time to maintain an easily readable list of changes on the wiki. That would not only include new features and bug fixes, but also code breaking changes that would require changes to existing keymaps, including instructions how to fix that.

[jprMesh]

What if all the core firmware development was on a separate branch that has "releases." The releases would get merged into master, which is where all the keymap stuff would be. So if someone wants to look at the firmware changes, they can just search the firmware branch in the commit log. I'm kinda new to large open source projects, so just let me know your thoughts.

[jackhumbert]

Thanks for bringing this up, and this is a good discussion, but we're going to stick with encouraging users to fork the entire repo for now!

[piotr-dobrogost]

Keymaps kept together with firmware really is a mess. I'm watching this repo and I'm really tired of "Added yet another keymap" commits. If you insist on having some keymaps together with firmware I think there should be one "default" keymap per keyboard type just to be able to quickly try it out and the rest should go into separate repo.
I think this should be reopened and separate repo for keymaps should be created.

[jackhumbert]

I wouldn't mind having a few keymaps supported by the keyboard maintainer, then having the rest in some sort of contributor keymaps repo. I'd like for the make system to be able to support building keymaps/keyboards outside of the qmk_firmware folder - I think that's the only thing that would need to happen to jump-start this.

[jackhumbert]

There are some valid hesitations to using submodules, so I'd like to talk through some options with this. If we assume some will download/clone both repos, they could get a folder structure like this:

qmk_firmware/
 - keyboards/
    - <keyboard>/
        - keymaps/
    - <keyboard>/
qmk_firmware_contributor_keymaps/
 - <keyboard>/
     - keymaps/
 - <keyboard>/

Where Makefiles from the ck repo would reference ../../../qmk_firmware/. Is there a better way to do this?

[algernon]

I used to be on the "lets have one giant repo with a ton of keymaps" side, because:

• It makes it easier to discover interesting keymaps.
• I only need to clone one thing.
• When refactoring, all the keymaps can be easily updated.

But it also has downsides:

• Tons and tons of noise in the git log, which makes it hard to filter out keymap-only changes.
• When refactoring, one also has to update all the keymaps too.

Having worked with Kaleidoscope, where there isn't even a repo that collects keymaps, I changed my mind, and now find the "default keymaps only" approach better, for a number of reasons:

• A lot less noise in the git log.
• Easier for a keymap maintainer to decide between keeping up with QMK, or sticking to an older commit. Some don't want, or need to upgrade QMK.
• Easier to create QMK releases, which in turn makes it easier for keymap maintainers to follow it.
• Keymap maintainers can focus on their keymap, and - apart from breaking changes - ignore the rest of QMK. For most users, this would be a big help, especially combined with proper releases.
• QMK developers need to care less about keymaps, because updating them is the responsibility of their creators. If their creators decide not to keep up with QMK, that is their choice - QMK developers should not bear the burden of having to keep every keymap under the sun up to date.

We lose a bit on the discoverability side (but a WiKi can go a long way there), and one will need to clone two repos, but... that's not such a big deal, I found, and can easily be scripted and documented.

I would not, however, force qmk_firmware and qmk_firmware_contributor_keymaps into the same parent directory. There are people (myself included) who already maintain their keymap without dragging the whole of QMK with it. With the new system, I'd expect to be able to build my keymap by pointing its Makefile to where the rest of QMK is (it could default to ../qmk_firmware, but should be overrideable). I keep code I didn't write checked out under ~/src/ext/, while my own stuff is under ~/src/personal - I'd like to avoid symlinking stuff all over the place to please a picky build system.

Thus, as far as the build system goes, I'd suggest the following that _ck/<keyboard>/keymaps/<keymap>/Makefile would look for QMK in QMK_ROOT, defaulting to ../../../qmk_firmware, but if the user overrides that in, say config.mk, or the make command-line, it should be honored. This makes it easy to build a separate keymap outside of QMK, and outside of the contributor_keymaps repo too.

[nicinabox]

@algernon really hits the nail on the head.

For a long time I kept my keymaps on a separate branch, merging in upstream master. This has always felt awkward and hacky to me. The discoverability of other keymaps has made it easier to learn from other setups, but there's also a ton of noise from them.

Today I split my keymaps off to their own repo so they can be maintained easier. I keep qmk master up to date and I compile out of my keymaps repo. This way it treats QMK more like a dependency of the keymap.

I'm definitely in favor of this workflow and with some core support this could be a viable process.

[jackhumbert]

With the addition of #1609, it might be worth considering organising the contributor keymap repo with layout keymaps, rather than keyboard ones. There's also the argument that there shouldn't be any keymaps in the layout folders in the core repo, but I could see wanting some examples (maybe even default_<layout>) for each.

[exiva]

keeping a default map for each board in core would make sense.

[jackhumbert]

Each keyboard would have a default keymap, on top of the layout's default_<layout>.

I wrote a little bit about using layouts for this:

The layouts/default/ and layouts/community/ are two examples of layout "repositories" - currently default will contain all of the information concerning the layout, and one default keymap named default_<layout>, for users to use as a reference. community contains all of the community keymaps, with the eventual goal of being split-off into a separate repo for users to clone into layouts/. QMK searches through all folders in layouts/, so it's possible to have multiple reposistories here.

[jackhumbert]

#1609 has been merged, so now it's just a matter of abstracting and moving the user keymaps into the layouts/community folder.

It may be worth considering something to restrict layout keymaps to certain keyboards with the KEYBOARD_<keyboard> flags, or even rules.mk settings.

[drashna]

Also, (once it's fully fixed), the version should be incremented for important changes only.

That would mean that docs, keyboards, layouts and user spaces wouldn't increment the version. Anything else would, and that would make it much easier to tell if serious commits have occurred.

Additionally, you can view the history of a folder, such as quantum or tmk_core.
https://github.com/qmk/qmk_firmware/commits/master/tmk_core
https://github.com/qmk/qmk_firmware/commits/master/quantum

[leico]

I support the way to make the layouts/community folder a different repository.

[drashna]

I've been thinking about this a lot, actually.

One of the issues that has cropped up now, is that there is the "userspace" feature. This makes this entire topic much more problematic.

It means that we'd need two submodules (one for keymaps, and one for userspace), or we'd need to reorganize the layouts and userspace into a single folder:
Eg, we'd need to use layouts/default/, layouts/community and layouts/userspace. And then we use the layouts folder entirely for the submodule.

Which isn't a bad idea.

The only downside, is that we'd really need to enable layout support for EVERY keyboard, and have a layout name for every layout.

It would create a lot of work.

In the meanwhile, I've been trying to add Keymap: and Keyboard: to the commits (when squashing), so that it's easier to filter through.

[nicinabox]

I've been running an experiment over the last year keeping my keymaps in their own repo using a custom cli and treating qmk_firmware as a lib only.

It's been awesome. The implementation is a bit hacky due to having to copy keymaps in the lib, instead of including the lib in the keymaps, but it's been a ton easier for me to maintain a growing list of boards.

Prototyping this has made me want qmk to be setup as separate repos:

• firmware - The actual lib and supporting bits (tests, docs, drivers, etc)
• standalone keymaps - Official and contributed keymaps (essentially keybords/) with the same architecture working for keymaps not part of the main repo (like nicinabox/keymaps)
• cli - Glue between the firmware lib and a keymaps repo

Splitting them up reduces noise in the main repo. It's a lot easier to focus on features, bugs, and support related to the firmware itself. Keyboards and layouts are still discoverable, and the CLI can use a real programming language, potentially making it much easier to maintain.

[jackhumbert]

I was thinking about this recently, and put this together as a template we can (eventually, on the qmk org) let people fork and customise: https://github.com/jackhumbert/qmk_firmware_embeded

It requires minimal changes to the QMK core (something we should be able to easily merge), and is mostly just cleaning up the paths (see the embedded branch in qmk/qmk_firmware). The Makefile in the root there is dead simple, and I think it would be pretty easy to support the scenario where someone wants to store all of their keymaps for multiple boards in one repo as well.

[henrebotha]

I'm also doing the separate repo + QMK as submodule + custom CLI thing. It's far more usable and useful than trying to deal with having my layout live in a subdir of QMK. I only have one QMK keyboard, so it's not even like that use case is limited to people with multiple layouts.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged properly or other activity occurs.
For maintainers: Please label with bug, in progress, on hold, discussion or to do to prevent the issue from being re-flagged.

[lowlevl]

Hey,

I'd really want to have a way to use a separate repo for my keymaps, and that's something I partially achieved there, however I needed to supply #19679 to fix misbehaving Makefiles.
I'm now at a roadblock because the qmk CLI that wanting to use the keyboards/ directory from $QMK_HOME for the generate-config-h, generate-keyboard-c, generate-keyboard-h and not the keyboards/ directory from $PWD.

Couldn't we just add a KEYBOARDS_DIR & KEYBOARDS_PATH to the Makefiles and a --keyboard-library-path to the qmk CLI ? I could free some time to do this.

EDIT: typos

[piotr-dobrogost]

How do things stand nowadays?
I turned on github's notifications for this repo and I see the churn from adding/changing various keymaps is enormous.

[keyboard-magpie]

See:

https://docs.qmk.fm/#/ChangeLog/20230827?id=notable-changes

and

https://hackmd.io/@tzarc/Bysm5Vnan

[piotr-dobrogost]

@keyboard-magpie
Do additions like this #22535 still go to this repository after split?

[drashna]

This got missed. KB PRs would still go to the main repo

Also, I think this can probably be closed, now