[WIP] pre-RFC: Better management of SVD to Chip Support Crate process

[jamesmunns]

• Feature Name: Central Management of SVD to CSP process
• Start Date: 2018-05-xx
• RFC PR:
• Rust Issue:

Summary

I need to make this a paragraph, but my main points are:

• SVDs kind of suck, but they are better than starting from scratch
• Having individual owners of SVD files, and "raw" crates generated from these SVD files has negative value IMO
• Having individual owners/maintainers of corrections, patches, and refactorings of Chip Support crates has extreme value IMO
• We should have a central location for all SVD files used by Rust developers, preferrably somewhere "official"
• We should have a central location (for discovery, if not storage) for all Chip Support Crates
• We should reduce focus on the SVD files themselves (and svd2rust), and instead provide better tooling for managing the patches, adjustments, and corrections made to the original generated code
• We should improve the quality of the code generated by svd2rust to reduce duplications which make the generated code hard to compile (with respect to compile time), read, maintain, and modify

Motivation

At the moment, there are a number of blockers to creating a quality, usable Chip Support Crate based on SVD files provided by manufacturers. These issues include:

1. Imperfect SVDs provided by manufacturers, providing incorrect (violating the SVD spec and/or not accurately describing the Chip's functionality) or incomplete information
2. No (official) central location(s) for Chip Support Crates, so that it is apparent what crates already exist, and to what extent they are active/supported
3. Changing semantics of "embedded rust"
4. Changing "abilities" of svd2rust

Additionally:

1. SVD files themselves (as provided by the manufacturer) are rarely updated by the manufacturer
2. SVD files (as XML) are difficult to "grok", as they are very large, and the syntax and semantics of them are not well understood
3. Having individuals "own" crates that are direct outputs of svd2rust is not specifically useful, and in fact is often counter-productive, when semantics of Rust or features of svd2rust change, it is necessary for all maintainers to manually update their crates. However there is extreme value in having maintainers that update, patch, and correct either the SVD files themselves, or the Rust code output by svd2rust
4. Due to the variable quality and consistency of SVD files (both between vendors, as well as between chips of the same vendor), it is currently very difficult to share low level code between chips, or sometimes even between similar portions of the same chip.
5. Despite all of the complaints above, it would be difficult, if not impossible, to programmatically generalize larger subsystems of chips, as they are all uniquely suited to their problem domain
6. The original SVD files are overall a wonderful starting point compared to hand-writing hundreds of thousands of lines of Rust based on a datasheet. We should always support the SVD-to-initial-CSP process, however we should optimize for maintenance of the living CSP crate, rather than the code generation process
7. Chip support crates support two primary use cases: Users that would like to use them directly, or for developers developing/creating -hal crates

Detailed design

// TODO

This is the bulk of the RFC. Explain the design in enough detail for somebody familiar
with the language to understand, and for somebody familiar with the compiler to implement.
This should get into specifics and corner-cases, and include examples of how the feature is used.

How We Teach This

// TODO: Figure out what we want to do first

What names and terminology work best for these concepts and why? 
How is this idea best presented—as a continuation of existing Rust patterns, or as a wholly new one?

Would the acceptance of this proposal change how Rust is taught to new users at any level? 
How should this feature be introduced and taught to existing Rust users?

What additions or changes to the Rust Reference, _The Rust Programming Language_, and/or _Rust by Example_ does it entail?

Drawbacks

Why should we *not* do this?

Alternatives

What other designs have been considered? What is the impact of not doing this?

Unresolved questions

1. How do we do this?

[adamgreig]

I'm very interested in making this better! A few early thoughts..

• I agree with your premise, SVD files do kind of suck, are better than nothing, and lots of individual crates, one per device, is a mess
• Maybe we can have a single crate for an entire family of chips (e.g. STM32), which is either feature-gated or just modules per actual chip, which can then share as many interior definitions (and types!) as possible. I think this would be difficult but possible to autogenerate from SVD sources with something like svd2rust, and as new chips came out you'd want some way to add them to it as well. This one crate could be a good community effort where if a new chip is not supported, it can be added to this crate rather than a whole new crate created.
• I quite like the TockOS style API and it gives much smaller Rust sources than svd2rust, which commonly has >300kloc outputs
• I've spent some time working on this sort of thing for STM32s with stm32-rs, which has a YAML format to represent changes to SVD files: mostly fixing vendor mistakes and adding enumeratedValues for the svd2rust safe API. The patches are shared between all chips that need them, and tooling exists to discover which patches can apply to which chips, so there's minimum duplication. The patched SVDs are then turned into crates per-subfamily (STM32F1, STM32F4, etc) with feature gates on individual chips, which is a sort of compromise between one huge crate (too big for crates.io file size limit) and one crate per device.
• I've also been working out what is shared between chips, with some results here, and a bunch of scripts to help automate this sort of thing. It would probably be an alright first step towards the design of a single STM32 crate with a lot of shared types.

[Emilgardis]

@adamgreig related to your second point: https://blog.japaric.io/brave-new-io/#reduce-implementation-work

[therealprof]

@jamesmunns By and large I agree but:

SVD files themselves (as provided by the manufacturer) are rarely updated by the manufacturer

Not true, while many of them truly suck there're a few notable exceptions to the rule, e.g. NXP updates the files regularly and they also seem to have a process for keeping them equivalent in quality, however it's tedious to obtain them...

Having individuals "own" crates that are direct outputs of svd2rust is not specifically useful, and in fact is often counter-productive, when semantics of Rust or features of svd2rust change, it is necessary for all maintainers to manually update their crates.

Yeah, and there's also a tendency to produce and submit peripheral access crates (whoops, maybe we should normalise the nomenclature here 😉) and then give up and never produce any meaningful crates using them.

Due to the variable quality and consistency of SVD files (both between vendors, as well as between chips of the same vendor), it is currently very difficult to share low level code between chips, or sometimes even between similar portions of the same chip.

Indeed.

[jcreedon]

This is a problem I have been thinking about for some time and so I'm just going to dump a lot of thoughts here.

An Example

So this project https://github.com/adamgreig/stm32-rs seems to try to tackle some of the problems discussed in this issue. The general workflow this project uses is:

1. The SVD file is added as provided by the manufacturer
2. A YAML file for each device covered by the SVD is created. The YAML file defines transformations executed on the original SVD to fix errors and add omissions. Importantly it adds missing enumeratedValues elements.
3. Because peripherals tend to be shared between several families of devices, peripheral-centric YAML files are created. These files are then referenced from the device YAML files for inclusion in the list of transforms.
4. Using the device YAML and any referenced peripheral YAMLs are used to transform the SVD into a corrected SVD.
5. The corrected SVD is then processed through svd2rust.

In general I think this process is a good starting point. I like the idea of transforming an SVD as opposed to editing the original. I also like the idea of splitting up peripherals into smaller chunks. There are a lot of good ideas to glean from this, but I don't think it is a complete solution.

Problems

With that example as context, here are some problems I have seen in the ecosystem as it stands now. A lot of the issues brought up in the issue OP are things I have run into and I agree with. Additionally there are some other opportunities for improvement that I have seen:

CSPs lack the ability to represent that a device comes in multiple packages/form factors and that some peripherals are not available in certain packages and form factors. Someone somewhere had mentioned that maybe CSP crates could use features to toggle these, but we need some kind of descriptor to list them out because the SVD is not providing it.

There is not at present, a way of adding documentation to the CSP without having to do a lot of manual editing after generation via svd2rust. While the rust tooling isn't quite to this maturity yet, one of the nice things in enjoy in my C/C++ tooling is having a tooltip in my IDE showing me the docstring for a function I am calling. Tooling aside, I think it is generally helpful for developers browsing the CSP source to see device specific documentation built into the auto generated code.

To generalize the previous two points, I consider hand editing after running svd2rust to be an anti-pattern, and as a negative for maintainability.

As mentioned by the issue OP, SVDs are incomplete, incorrect, and hardly ever updated by the manufacturer, so they would require editing to be useful. As a statement of fact, collaborative editing of a large monolithic files is hard and managing a community around editing those files is even harder. SVDs are very large monolithic files, and on top of that they are XML which IMO makes the process just that much more annoying (e.g. diffs are more verbose, files are harder to read, lots of whitespace).

The embedded world is more than just Cortex-M, and devices in other architectures don't always have an SVD file. How do we have a cohesive development user story for those devices if our process is SVD-centric?

A Proposal

So with those problems in mind instead I'd like propose an alternative solution.

Instead of making SVD the canonical format for the generation of CSPs, let's define our own format that is geared towards being reusable and community maintained. For the sake of this discussion let's say this is in the TOML format. We would first make a tool we'll call svd2toml to use in bootstrapping this community. svd2toml would generate two types of files: a device TOML and a peripheral TOML. There would be one device file, and a peripheral file for each of the peripherals defined in the SVD.

These generated files would then get added to central repository. This repository would ideally be subdivided by manufacturer, then by device family. From this point on the TOML becomes the canonical format for describing devices. The files are then maintained by community effort. Contributions would follow the same fork, PR, test, and merge workflow we are all used to.

Finally we would have a new tool based on or similar to svd2rust that takes the TOML files and generates CSP crates. Once again, for the sake of discussion, we'll call this tool toml2rust.

File Formats

Functionally the TOML files act as a superset of the information provided by SVDs. It is all the same device and peripheral data provided originally plus additional data that makes generating CSPs easier and more complete.

The device TOML format contains:

• A list of peripherals and references to peripheral TOMLs
• A list of part manufacturer part numbers covered by the CSP
• A memory layout definition for the generation of base linker files (i.e. memory.x)

The peripheral TOML format contains:

• A list of registers
• A list of bitfields within those registers and their allowed values.
• Documentation for the registers and the bitfields.

Depending on the size of the peripheral, we could split the peripheral TOML file up further and make a register TOML format. The overall goal is to make small, easy to read, and maintainable files.

Because of the extra information these files provide, it should be possible to generate everything we need using toml2rust and there for eliminate (or extremely reduce) any hand editing needed after code generation.

We could write a linter that parses the TOML files and lists out what data fields are missing from the TOML files necessary to generate safe rust APIs in the CSP. (e.g. what fields are missing enumeratedValues). This could be useful as a CI tool or as a way of listing areas in need of improvement where contributions would be appreciated.

It should be possible to make the format easy enough to understand that you can take the reference manual for an undocumented device that has no SVD and be able write out all the information necessary to build a CSP crate. This is especially useful for devices/architectures that do not have SVD files.

Also, because the format is a superset of SVD it should also be possible to write a toml2svd utility to generate a SVD based on the community generated data. There is a lot of debug tooling out there that can take SVD files to make it easier to inspect peripherals and I think the embedded community could really benefit from having edited and fixed SVD files.

Other Thoughts

Namespacing. When we generate CSPs and publish them, should we follow a naming convention to make it clear/obvious that the crate is from the community? crates.io does not support namespacing so if we were to do this it would mean some kind of syntax like "csp-{device_family}".

Editorialization. One thing we might need to do from a maintainability perspective is editorialize the grouping of devices and peripherals. This part can be tricky and potential source for subtle errors. Manufacturers generally reuse the same peripheral IP across multiple families of devices. It would be far easier to maintain if these definitions were generic and we just reference them form the device level. However, the manufacturer does not make it explicit which peripherals are identical. A lot of embedded developers know this via familiarity with a particular manufacturers parts (e.g. if you spend a lot of time working with STM32 parts, you tend to figure out what peripherals are the same), however this exists purely as tribal knowledge.

Versioning. How do you version CSPs? There are two source of breaking changes, first in the underlying data (e.g. renaming a register, as a contrived example), and second in the API generated by toml2rust. I think it would be the least confusing if the CSP version matched the API such that all 1.x.y CSP crates all have the same API, however, what do you do if a breaking change is made to the underlying data of a device used to generate the crate? Theoretically you would have to bump the major version of that device's crate, but then it can become confusing.

Conflict resolution. What do you do when a register is named one way in the SVD, but differently in the reference manual?

EDIT

This came out a little longer than I expected. Here's a TL;DR

Goal: Create safe APIs for devices based on device metadata
Problem: Manufacturer provided metadata is incorrect and incomplete
Problem: Metadata file format does not provide all the information we'd like to implement the features we want
Solution?: Define a new metadata format that can represent the information we need and is easy for the community to contribute to.

[therealprof]

@adamgreig

I agree with your premise, SVD files do kind of suck, are better than nothing, and lots of individual crates, one per device, is a mess

It's actually worse: e.g. NXP tends to have one SVD per package/variant, not just per chip.

Maybe we can have a single crate for an entire family of chips (e.g. STM32), which is either feature-gated or just modules per actual chip, which can then share as many interior definitions (and types!) as possible.

That would be nice but there're so many subtle differences that would make managing them a true nightmare IMO.

I quite like the TockOS style API and it gives much smaller Rust sources than svd2rust, which commonly has >300kloc outputs

It certainly is smaller but also far less powerful and safe. I have a feeling that giving up safety properties won't fly well within the WG.

Your stm32-rs work is absolutely fantastic. However that's a heavy weight solution that could use some diet to make it more feasible. Not only are the downloaded crates somewhat big but also the compile times go through the roof (for me, compared to regular svd2rust crates).

[therealprof]

@Emilgardis This may work in a few cases but in general the differences are just too big to make it feasible. Even if a peripherals are 1:1 compatible in a family, maintaining the declarations due to differences in pin mappings are a royal PITA.

[adamgreig]

@jcreedon
Would you have a crate per device TOML file? Or larger meta-crates? With this single repository of every TOML, you could even imagine a crate which built the device you required when used as a dependency perhaps. I'm less convinced at the idea of a hundred crates on crates.io, one for an stm32f405, another for the stm32f407, etc, etc.

I agree up to the point where we have a big repository of well-maintained and up-to-date files describing as many devices as possible, just not necessarily how we use it beyond that.

Editorialization

I would hope that automatic tooling could pick up where peripherals were identical (or identical subsets) between devices (see my comment about this in my earlier post). Ideally that automatic process could be used to ensure no duplication between devices.

Conflict resolution

I'd vote for fixing the SVD to match the reference manual, because the SVDs are so often wrong and the reference manuals feel somewhat more authoritative, plus are what people will be reading. There's another question as to whether you then fix things further to be consistent, where one reference manual calls something one thing and another something slightly different (which of course happens all the time).

@therealprof

NXP tends to have one SVD per package/variant

yikes

That would be nice but there're so many subtle differences that would make managing them a true nightmare IMO.

I think it could be done on some level (with plenty of automatic tooling). ChibiOS and FreeRTOS and other HALs manage it.

However that's a heavy weight solution that could use some diet to make it more feasible. Not only are the downloaded crates somewhat big but also the compile times go through the roof (for me, compared to regular svd2rust crates).

Thank you! The whole design is definitely a work in progress - my main concern has been to ensure that the manual work of going through ref mans to fix bugs and match devices is possible to keep, even if everything else changes, and I expect the crate design will change a lot.

The large source files are a real pain (hence my complaint about the svd2rust output!) but I'm surprised at the longer compile times: I hadn't noticed any difference, and since the only code in each crate is feature-gated for a specific chip, I'd have thought it would be exactly the same as regular svd2rust crates.

[jcreedon]

@adamgreig

just not necessarily how we use it beyond that.

I avoided talking about exactly how the lines are drawn as I don't think I have a strong opinion on that quite yet. Also, depending on the syntax of these files and how we write the generator, there is no reason that either a single set of files couldn't generate multiple crates, or that multiple sets of files couldn't generate a single crate. How metadata is divided out does not necessarily have to be coupled to how crates are divided out. Ideally the rules for dividing out metadata is to optimize for their maintenance and usefulness to downstream tooling, and rules for dividing out crates is to optimize their ease of use for developers using them.

Ultimately there will probably be very close parallels between how metadata is grouped/divided and how crates are grouped/divided, but we need to remember that the goals for the two layers here are very different. The criteria for grouping and dividing metadata files I think is simply to make it easiest to maintain and develop against (keeping in mind that code generation is only one of many potential uses of this metadata). The criteria for grouping and dividing CSP crates I think is a little more nuanced. There are multiple constraints at play. To illustrate a few:

Starting from one extreme, I think from a user/developer standpoint it would be easiest if there was an individual crate for every variant. If I have a STM32F415RGT6 on my board, then extern crate STM32F415RGT6 is the least ambiguous way of using that chip and requires the least mucking around in configuration. It also makes writing documentation easier, it is simply "use the crate that matches your part number". It doesn't matter from a maintainer's perspective if there are 100s or 1000s of crates because we should be able to write tooling so that they are all automatically generated. This does however, put an additional (and perhaps unnecessary) strain on other parts of the ecosystem. For example if we rev an API and push changes, crates.io and docs.rs might not appreciate 1000s of crates being updated simultaneously.

On the other extreme if I have one crate for all STM32 parts, that makes it easier from a number of crates perspective, but it makes the crate so much bigger and longer to download. It would make documentation easy, as it is mostly something along the lines of "import the crate for your manufacturer". But another downside would be that it consolidates your breaking change risk into a single bucket, so if you have to implement a breaking change for just one family or one variant, that would mean revving the major version for all of them.

Somewhere in the middle would be to follow whatever the manufacturers grouping is. This give more compactness with regards to both number of crates and size of the crates, but it can be more confusing for users. Manufacturers use different methods for grouping families of products and if we adhere to the manufacturer divisions, that means across the entire embedded rust ecosystem there are different rules for how devices are grouped. This makes documentation more complicated or verbose (e.g. "for ST it works like this... but for NXP it works like this... etc.)

I don't think grouping by large families is quite the right solution though. For example, using ST, let's say if you were to make a single crate for all F4 devices, and you build them, test them, they are stable, you make a 1.x release crate. Then lets say ST comes out with a new sub-family the F48x. Theoretically that would belong in the F4 crate, but ideally this new feature set would start it out at 0.1 per semver and rust versioning guidelines. There could be errors in the reference manual or SVDs, there could be typos in metadata, we don't want to misrepresent stability until it has been proven out for a while.

With that in mind it would seem that sub-families seem to be the appropriate division. The naming of these sub-families can be tricky. Once again, to pick on ST, the F405, F415, F407, F417 would constitute one sub-family, except you can't name it F4xx, because that describes all of F4, and you can't call it F4[01][57] because you are limited to [a-z-] for crate names. It's not an insurmountable problem, but it is annoying to be sure.

Bottom line, the goals for grouping and dividing crates is probably a healthy balance of the following:

• Compactness of crate and generated code
• Least confusing for the user/developer
• Similarity to manufacturer families
• Segmented stability guarantees
• Segmented breaking change risk

In the end it will probably end up looking something like manufacturer sub-families. That does come with it's own set of challenges, that I think can be solved in other areas of the ecosystem. For example, I think one thing we can do to greatly improve the onboarding process for new users is create a tool and/or website that basically allows the user to select the Manufacturer and the Part number, and it generates an example Cargo.toml file that has the proper dependent crates with the appropriate feature toggles set. That way regardless of the complexities of whatever crate strategy we decide upon, we can have an easy way for user/developers to get up and running.

[jcreedon]

@adamgreig

reference manuals feel somewhat more authoritative

👍 Also on that note, while manufacturers are inconsistent about updating SVDs, they are pretty consistent about updating reference manuals across the board, and some even do a good job of outlining the errata from previous revisions.

[jcreedon]

@adamgreig

I would hope that automatic tooling could pick up where peripherals were identical

Automatic tooling could work, but that still requires having accurate metadata, which still comes back to having someone manually review it.

[jcreedon]

@adamgreig @therealprof

I quite like the TockOS style API and it gives much smaller Rust sources than svd2rust, which commonly has >300kloc outputs

It certainly is smaller but also far less powerful and safe. I have a feeling that giving up safety properties won't fly well within the WG.

I definitely can relate to both of these sentiments. The raw lib.rs output that svd2rust generated was way too big and very unwieldy. It made it very hard to take a quick look at structs, and lot's of tooling would fail on it due to its size (for example the intellij-rust plugin for idea would not provide any autocomplete for it). Using form to split it out did improve some things (it was easier to take a peek at things, and autocomplete did start working) but it also came with other tradeoffs. For example the split out crate now has hundreds of files, which has a performance impact on tooling that has a per file cost (e.g. git).

The reduced code size of svd2reg is tempting, however, I am trying to ditch C/C++ because I want leverage the safety features Rust has to offer for developing firmware. If the libraries don't utilize those safety features, why am I going through all the effort to switch to rust? Reducing crate LoC is definitely a worthwhile goal, as long as it is not at the cost of safety.

[jcreedon]

@therealprof

maintaining the declarations due to differences in pin mappings are a royal PITA.

I'm not aware of SVD files representing pin mappings of peripherals, or any CSPs providing abstractions for pins. While pins do have abstractions in HAL crates, I don't know if HAL crates fit into the scope of this RFC.

That being said, I think it would nice to be have a nice set of metadata representing the pinmux capabilities of devices, and I can imagine a few different tooling possibilities to soothe some of the painful parts of keeping track of pins.

[ithinuel]

Hi all,

I don't think that code generation is a good thing anyway.
Manufacturers all have the same scheme with Family, subfamily etc down to the chip specific datasheet.
In https://ithinuel.me/embedded-rust-framework-structure/ I expose the hierarchy that could work.
It maximises the reuse of code and thus minimise maintenance effort (DRY for the win).

The leaf crates (aka the board crate) would there only provide board specific functions such as pin mapping setups etc.

This architecture also makes it possible and easy for chip manufacturers to maintain their own set of crates to provide anything they want/need from core specific feature to demo/example crates that can run on different chips.

[therealprof]

@jcreedon

I'm not aware of SVD files representing pin mappings of peripherals, or any CSPs providing abstractions for pins.

I'm also not aware of any directly expressed mappings, however implicitly they are all there and hence need to be addressed. There are various combinations of behaviours what will happen if you simply gloss over the fact that you're trying to work with data which does not exist in the particular hardware you have at hand. E.g. for STM32 the registers for pins which are not available in a smaller package will be there and can be used but will not have any effect on the system. If you try to do the same on a different chip, even from the same "family", it might either give you the ignorance treatment or it might cause faults. Alternate settings are again chip specific (but not mentioned in the SVD at all) but same here: Using a wrong setting will cause unexpected behaviour or, worst case, damage the hardware. Fun fact: Some chips do have a special one-off remapping functionality to use peripherals which would otherwise be not accessible.

NB: NXP goes to the extreme of doing one SVD file per package, do avoid those kind of situations.

While pins do have abstractions in HAL crates, I don't know if HAL crates fit into the scope of this RFC.

Well, the bigger picture is: A register description alone is pretty much worthless in the long run because no one wants to write MCU specific applications and reinvent the wheel all the times by manually programming all the required peripherals. The two important points from my POV here are:

• identical registers need to have identical names (often thwarted by typos and other errors in the definitions), while functionally different registers may not (to allow the compiler to spot that something is different and a slightly different HAL implementation is needed)
• those crates need to provide enough information to simplify the implementation of fairly generic HAL crates without putting the onus on the user to manually code all possible pin mappings using the manual. Of course Nordic chips are at a slight advantage here due to the ability to use any function on any pin. ;)

[rudihorn]

I guess my view of this is that it may be worth working horizontally (consolidating peripherals) rather than vertically (generating crates for individual devices).

I'm quite a big advocate for separating implementations for different modules and using proof of setup rather than having individual modules fiddling with other modules. So ideally, a HAL implementation for USART shouldn't touch a register of the AFIO module, because the USART hardware doesn't know about this. And developing a crate that only knows about USART registers, but with cfg's modifying for specific devices, will help to have the HAL respect this as well. This would allows us to focus on a small things at a time and allows us to do it properly. Honestly I'd rather have all the rust stuff only support 7 modules (that covers 99% of Adafruit stuff), but well across the entire STM32 range, than half baked stuff annoying everyone for decades...

A specific device crate would on the other hand import family peripheral crates with desired configurations and specify what peripherals are located where, just as the specific device datasheet would specify:(http://www.st.com/content/ccc/resource/technical/document/datasheet/33/d4/6f/1d/df/0b/4c/6d/CD00161566.pdf/files/CD00161566.pdf/jcr:content/translations/en.CD00161566.pdf, figure 11). A HAL crate for USART, would only (or mainly) depend the USART register crate.

So to sum it up, you may have crates such as:

• stm32-usart
• stm32-usart-hal (depends on stm32-usart, possibly requiring the same configs)
• stm32f103xx-hal (depends on stm32-usart-hal, stm32-i2c-hal, …) and declares which modules are located where in memory, maybe specifies which pin combinations are allowed for what).

As many have mentioned, tooling will probably be key to any ideas here, and I think looking into how we can also visualize differences generated from svd files would be super helpful. The work @adamgreig has done so far seems fantastic, and I think visualizing and understanding it further will help lots (may I post that other link you mentioned in IRC?). I'd like to perform some experimentation of my own (especially comparing registers with different memory offsets as well), but I have other deadlines at the moment so will have to see if I can find time.

An alternative idea I was thinking of was looking considering is a "reverse approach", of using svd files to validate what we have and potentially point out errors, used-but-not-allowed configs etc., so that if we do start performing weird modifications, we can get some feedback as to where things have gone wrong.

[hannobraun]

@therealprof

SVD files themselves (as provided by the manufacturer) are rarely updated by the manufacturer

Not true, while many of them truly suck there're a few notable exceptions to the rule, e.g. NXP updates the files regularly and they also seem to have a process for keeping them equivalent in quality, however it's tedious to obtain them...

I've been in contact with someone at NXP and can confirm this. From what I gather, they take their SVD files seriously, but don't generally release them to the public, as they're intended for internal use and use by tool partners. According to my contact, they provide SVD files to anyone on request though.

It's actually worse: e.g. NXP tends to have one SVD per package/variant, not just per chip.

Are you talking about Kinetis? All the LPC SVDs I've seen have been per-family.

@jcreedon

So with those problems in mind instead I'd like propose an alternative solution.
[...]

I like that idea. Having a Rust-centric format could make the whole process easier, maybe. How would we handle SVD updates by the manufacturer? Re-generate the TOML, merge it with the maintained TOML version by hand? Seems acceptable, if SVD updates are infrequent.

[therealprof]

@hannobraun

According to my contact, they provide SVD files to anyone on request though.

Yes, you can download them yourself by going to the "MCUXpresso SDK Builder" site and selecting what you need. They'll only distribute full bundles with documentation and everything. Takes a long while to build and is quite a package.

Are you talking about Kinetis? All the LPC SVDs I've seen have been per-family.

Yeah, that'd be Kinetis.

[jcsoo]

I have been using a s-expression based DSL that is a superset of SVD to solve some of these problems for my own projects:

https://github.com/bobbin-rs/bobbin-sdk/tree/master/dsl

It's been put together specifically to support composing MCUs from multiple crates - generally arch (Cortex-M), vendor (STM32, NXP, etc.) and model (SAMD21, K64, STM32F30x), with multiple variants supported within each file.

The general process is to start with a SVD file, then use bobbin-svd to convert it to this DSL. It would be possible to do codegen at that point, but what's better is to examine the generated DSL and then replace common peripherals with imports from the vendor peripheral crate (moving new peripherals to the vendor crate if necessary).

Along the way, there is the opportunity to clean up the peripheral definitions by consolidating register arrays and defining indexed fields, gaining back structure that might have been thrown out during vendor's original SVD generation process. It's also good to fix up identifiers by removing redundant prefixes and making sure they match up with the documentation and/or existing libraries (I have seen many examples where these are in conflict!).

This particular DSL goes a lot further than SVD in that it supports concepts such as channels (ADC, DMA, etc.), pins and signals (which allow automatic trait-based compile-time checked pin configuration), and clocks and gates (which allow defining clock trees and automatic trait-based compile-time checked clock gate control). These are optional, but produce a much more complete modeling of most MCUs. They require some combination of data entry / automated tooling to create from the datasheets, but the process is not too difficult and only needs to be done once.

I have a handful of MCU definitions that are up to date with the current version of the DSL, from a couple of vendors: https://github.com/bobbin-rs/bobbin-sdk/tree/master/mcu-src

And here are the generated crates: https://github.com/bobbin-rs/bobbin-sdk/tree/master/mcu

I have a dozen or so other MCUs (TI, SiLabs, Ambiq, etc) that just need to be updated to the current DSL specification.

The overall structure is similar to what's been discussed, though I have grouped peripherals into a single vendor-common crate rather than breaking them out individually. The codegen is done in a different style than what svd2rust currently produces (and which I prefer) but that's something that can easily be changed.

There are a lot of good ideas in this discussion (I really want to add tools for visualizing the MCUs as well as generating nice peripheral and register diagrams that can be embedded in the Rust documentation).

[hannobraun]

@therealprof

Yes, you can download them yourself by going to the "MCUXpresso SDK Builder" site and selecting what you need. They'll only distribute full bundles with documentation and everything. Takes a long while to build and is quite a package.

Good to know. Thank you!

[adamgreig]

@jcreedon
It's clear you've thought about this a lot, thanks for the insightful comments!

[how many crates? one per manufacturer/family/subfamily]

It is a tricky balance. Another reason to favour a single crate for all STM32s is to make it easier to share types between the devices, so that you could write HAL implementations once which apply to STM32s even across families. But @rudihorn's suggestion also solves this, if we had an stm32-usart crate and stm32f405 depended on it, etc. Maybe that's a nice way around the versioning issue, though I'm still not totally convinced on one crate per device (or per variant even) living on crates.io.

If the libraries don't utilize those safety features, why am I going through all the effort to switch to rust?

Personally I'm using Rust on embedded for a lot of reasons besides extreme type safety in peripheral registers, like the build system, other libraries, language features, safety for the rest of the code, etc. They all make it a lot nicer than C/C++ irregardless of the register API. I'd be perfectly happy with a lightweight and extremely simple API that was basically like using C. Adding safety and convenience is great (especially having all the field variants available as functions), but I worry svd2rust ends up being way too heavyweight. From my point of view it's just as unsafe to write the wrong logical value to a field as it is to write a nonsensical variant, so I have to check just as carefully.

@jcsoo

I have been using a s-expression based DSL [...]

I really like this! I feel like TOML would not cope as well with the level of nested structures we'd probably want and something like this could be a lot nicer to read/write both by hand and by software. Including channels and pins and clocks is nice and we'll need that information present somehow or other eventually.

I really want to add tools for visualizing the MCUs as well as generating nice peripheral and register diagrams that can be embedded in the Rust documentation

One of the nice things that came out of stm32-rs was this page which made going through to find what was missing easy, but is also nice as an alternative to the reference manuals. Combining it with details of the Rust API and making the UI a bit nicer (instead of a multi-MB single page..) and I think we could build something people would use even if they weren't using Rust at all.

[chrysn]

@rqou, you have been working on this for EFM32; did that yield results that might be valuable in this discussion?

[ryankurte]

I definitely agree with your summary @jamesmunns, and would love to see a community maintained repo for board support packages, though I am still in favour of SVDs as the source of truth with some form of patches and analysis applied to them.

I didn't see them linked after a quick scroll, so, y'all might also be interested in rust-lang-nursery/embedded-wg#33 and japaric/svd2rust#96 for previous discussion of some of the sharing issues, also japaric/svd2rust#96 working to allow svd exporting to solve some of these.

[brainstorm]

We should have a central location for all SVD files used by Rust developers, preferrably somewhere "official"

What about just forking https://github.com/posborne/cmsis-svd (or any other comprehensive multi-MCU SVD repo) into this org (rust-embedded) and make that fork the authoritative source to iteratively patch against until some "SVD nirvana steady state" is reached? ;)

Relatively low immediate effort, potentially high reward over time?

[brainstorm]

We should have a central location for all SVD files used by Rust developers, preferrably somewhere "official"

What about just forking https://github.com/posborne/cmsis-svd (or any other comprehensive multi-MCU SVD repo) into this org (rust-embedded) and make that fork the authoritative source to iteratively patch against until some "SVD nirvana steady state" is reached? ;)

Relatively low immediate effort, potentially high reward over time?

This is a perfect example of what I mean from my previous "SVD steady state" comment:

https://github.com/rust-embedded/svd2rust/blob/3b2e8a535f1e4e6b1db7cd1ff05f6fb7f0c89d03/ci/script.sh#L82

This huge list of exceptions/blacklisting could be fixed right away on our own fork and pullrequest upstream to posborne for all embedded devs from other langs/frameworks to enjoy well built, curated, SVDs (curation being as automated as possible or reasonable, of course).

/cc @burrbull

[therealprof]

I think this has been overtaken by events. There're now multiple alternative solutions to generating chip support crates from SVD files and managing SVD files, including the WGs https://github.com/rust-embedded/svd2rust/ as well as non-WG projects like https://github.com/embassy-rs/chiptool and https://github.com/adamgreig/ral-registers.

I think more elaborate support of individual MCU families is out of scope of the Embedded WG so I'm going to call this done for now.