GPIO features #129

teskje · 2020-07-31T20:56:08Z

This PR is an attempt to replace the mess that is our current GPIO pin mapping code with auto-generated mappings extracted from the STM32CubeMX database. It's inspired by a similar effort done for the stm32l0xx-hal: stm32-rs/stm32l0xx-hal#82.

Parsing of that database is done by executing the new codegen crate and pointing it at the database path, e.g.:

$ cargo run --target x86_64-unknown-linux-gnu -- gpio /opt/stm32cubemx/db/mcu

This command yields feature-gated gpio! macro invocations that can be copy-pasted at the end of gpio.rs.

Based on this parsing of the CubeMX database, this PR introduces 5 new features for the 5 different GPIO peripheral versions used between the F3 MCUs. Each MCU feature selects the correct GPIO version, so users don't have to do anything. I've added two new MCU features too, for stm32f302x6 and stm32f302x8, which we previously missed.

Apart from the GPIO code itself, I had to touch a few other modules in cases where code used feature-gates that were too broad. For example, code would gate the definition of a UART pin behind a stm32f302 feature gate, but the pin isn't available on all F302 MCUs. For the I2C and serial modules I was able to switch to the new gpio-* features and I'm reasonably confident that this code is correct (or at least more correct than before). For PWM I only did the minimal changes required to make it compile again because, honestly, this code is a mess and I don't understand what it's doing. We need to refactor that too at some point.

In the long term, the idea is to only use peripheral version features like the gpio-* ones as feature gates in the code. MCU features (stm32f3*) should only select the correct peripheral versions and nothing more. I think to achieve this we'll have to extend the codegen to more peripherals first.

Currently, the code doesn't compile for most MCUs. It looks like that is because of bugs in the PAC. The PAC contains different GPIO modules (gpio{a,b,c}), which seems to be due to different reset values between the different GPIO ports (port A and B have debug pins that are set to AF0 by default). As far as I can tell, all GPIO versions have the same debug pins and therefore the same reset values, so they should have the same GPIO modules in the PAC too. This is not the case for some reason. The stm32f303 PAC is lacking gpioc, for example. I plan to open an issue on the stm32-rs repo and hopefully get that fixed.

In summary, what's still to do here:

Fix the PAC to get this code to compile
Check the auto-generated alternate function mappings against the datasheets
Add high-level documentation for codegen
Wait for new PAC release

teskje · 2020-07-31T22:11:44Z

Btw, the PAC modules matter only because we have functions that give out access to the shared GPIO registers and we need to name their return types. That wouldn't be the case if we used the CS approach (discussed in at length #37) to access those registers, but I didn't want to make this a full refactor of the GPIO code, to keep the changes manageable. And the PAC should be fixed anyway.

src/pwm.rs

teskje · 2020-07-31T22:18:56Z

src/serial.rs

+        use crate::dma;
+        use cortex_m::interrupt;
+    }
+}


I am not completely happy with using cfg_if to apply feature gates to multiple items. I guess it beats repeating yourself, but there is quite a lot of added indentation now.

teskje · 2020-07-31T22:22:25Z

src/serial.rs


 unsafe impl TxPin<USART2> for gpioa::PA2<AF7> {}
-// unsafe impl TxPin<USART2> for gpioa::PA14<AF7> {}
-// unsafe impl TxPin<USART2> for gpiob::PB3<AF7> {}


I don't know why these pins were commented out. I just added them in again, with the features derived from the available AF mappings according to the CubeMX DB.

teskje · 2020-08-07T17:32:12Z

All checks pass but this is only because the CI is actually broken. I created a PR to fix it: #130

In reality, the compilation still does not work for the stm32f398 feature. The reason is that the PAC is missing the enable bits for GPIOG and GPIOH (iopgen and iophen). I opened another issue on stm32-rs for this: stm32-rs/stm32-rs#410.

teskje · 2020-08-08T10:00:44Z

That second issue with the PAC has been fixed and this PR compiles not for all MCU sub-families.

teskje · 2020-08-08T13:19:40Z

I've looked at most of the relevant datasheets and it looks like all pin and AF mappings that are present in the datasheets are also present in the auto-generated mappings. Which means the CubeMX database is likely complete.

Some MCU families omit some of the theoretically available AFs in their datasheet. That is because these AFs relate to peripherals not available on the respective MCUs. For example, on the F301 and the F318, the CAN-related AFs are missing. Once we have feature gates for all peripherals, we might be able to use those to decide which AFs are not actually useful on which MCU and exculde them. However, this sounds like a lot of complexity for little gain. There is not much harm in allowing the user to set pins into an AF mode that is useless on the particular MCU. I think it's fine to just ignore those superfluous AFs.

In some MCUs (e.g. F318, F378) the pin PB2 is not available according to the datasheet. That's probably because these MCUs have a smaller package that cannot expose all the pins. Like with the superfluous AFs I don't think it's an issue to still leave them accessible in the HAL. Users will be able to configure the non-existing pin but they won't be able to do anything harmful this way, as far as I can tell.

Finally, the F318 omits whole GPIO ports. According to the alternate functions documentation in the datasheet, ports C and D are missing. I'm not sure if this is quite correct since other sections in the data sheet mention a GPIOC, but GPIOD is definitely not there. According to the RM this MCU family still has the respective RCC enable and reset bits for both ports, so again no harm should be done if we just ignore the fact that those ports are missing.

TLDR: We are not missing any mappings but for MCUs with fewer peripherals or smaller packages we sometimes have too many. We could avoid that with more feature gates but I'd expect that to make the code significantly more complex and that is probably not worth it. It should be fine to have a few too many mappings on some MCUs since the worst thing that can arise from users making use of those is useless code that does nothing.

strom-und-spiele · 2020-08-08T15:51:17Z

Thanks for looking into it.
While I share your point on an abstract level (there is no need to keep the user from configuring something useless) I'm not sure wrong configs are without side effects all the time. e.g. if there is no GPIO D bank, but I still configure it, the configuration needs to go somewhere. Now if the memory address of the (non existent) GPIO D config registers is actually some mem address that configures something else this could lead to bugs that will be hard to identify.
If you are ahead of me and you already checked this, I'm impressed. Let's just note that we talked about it ;)

teskje · 2020-08-09T12:31:20Z

I'm currently operating under the assumption that the F318 has the exact same GPIO implementation as, say, the F302x6, just with no usable pins on the GPIOD port. This assumption is purely based on what the CubeMX DB says, I don't see any way to actually verify it, unfortunately. But if it is true that means writing to the memory-mapped registers of this port on the F318 still configures the GPIO normally, it just doesn't have any effect. So that should be totally safe.

What we can easily do is have a look at the datasheets to see what is actually mapped at the missing port's memory. For the F318 GPIOD would be mapped at 0x4800_0C00 - 0x4800_0FFF (directly above GPIOC). This region is marked as "Reserved" in the datasheet, so there is nothing else mapped there. If you check out any other datasheet in the F3 family you will actually notice that the GPIO ports are always mapped to the same address, independent of which GPIO implementation is actually used. E.g. GPIOA is always at 0x4800_0000, GPIOD is always at 0x4800_0C00. And if a port is missing, the respective region is always marked as "Reserved". This makes me quite sure that we are not in danger of accidentially configuring some unrelated peripheral.

Then again I know nothing about the underlying hardware details and my mental image of how this stuff works might be totally wrong. If you have any idea how we could verify or falsify this more, that would help :)

Cargo.toml

strom-und-spiele · 2020-08-09T13:23:16Z

If you check out any other datasheet in the F3 family you will actually notice that the GPIO ports are always mapped to the same address, independent of which GPIO implementation is actually used.

I haven't (really) checked any datasheets but the one related to the f303xc before - so thanks for doing that. While my mental image of how this stuff works might be totally wrong, that pretty much resolves my concern.

Cargo.toml

strom-und-spiele

Wow, this will take a while to read through. Thanks for all that work.

I'll try to go through it file by file. Hope that works.

A note on testing (disclosure: I'm a big fan of TDD):
I see why there is not a single test for the HAL, as a test would most of the time simply reproduce the code written (or at least I don't get how to test a proper register setup). Also property based testing (like idempotency tests) doesn't really feel in place here. If there is something else that makes sense on hardware on the low level of a HAL, I'd like to learn about it and even use it here, but I don't know of any.

However this seems not to be the case for this codegen subproject. If I grasp the scope of this codegen project correctly, it provides a very good starting point to replace a lot of repetetive tasks in a lot of the modules already written and to be wirtten. So establishing an easy to refactor codebase seems like a good idea.
however I'm just starting to read all of this, so stop me if I'm ahead of myself ;)

teskje · 2020-08-17T16:58:30Z

@jamesmunns actually made a blog post for testing on embedded, might be interesting to you: https://jamesmunns.com/blog/hardware-ci-overview/.

I don't know too much about testing embedded code myself. I think it is a good idea to try to separate low-level hardware-specific code (register accesses) from hardware-independent algorithmic code. Put the algorithmic code into a separate (no_std) crate and unit test it like you are used to. Because it doesn't do any hardware-specific stuff, you can just run any tests for this kind of code directly on your host system. The hardware-specific code is more problematic. There are some ideas out there for test frameworks that transparently execute your tests (or just the register accesses) on an actual MCU, but nothing production-ready AFAIK. The interest in something like that doesn't seem huge, which is partly because Rust already provides good robustness through its type system, so there is generally less need for writing tests than there would be in weakly or dynamically typed languages. What's usually provided as a replacement for tests in hardware-specific code are examples. They can be run semi-automatically (you sometimes need a very specific hardware setup) and act like integration tests.

A HAL is by definition hardware-specific code. I can't think of anything we could decouple enough from the hardware to write unit tests for. So we are left with making good use of Rust's type checking and writing examples (we could definitely use more of those). It would be interesting to explore options of having unit tests that run directly on the MCU.

You are right that codegen doesn't have any of these issues. It is a normal std crate and can be tested using Rust's normal test facilities. I haven't written tests for it because I find it hard to come up with things worth testing here. The parsing step is implemented in serde and we mostly define structs to deserialize into, the actual logic resides in third-party crates which, I assume, are well tested already. The rest is the code generation, which seems hard to verify. How should we automatically test that the code we generate is actually correct? We can try compiling it, but since it depends on other HAL code, we'd have to do that together with the rest of the HAL. This would be possible, but a lot of effort for, IMO, too little value. It is much easier to look at the generated code and let our existing CI ensure the HAL still compiles with it. Also, keep in mind that codegen is an internal crate that is used during development only. It won't be used by HAL users, so there are no high robustness requirements. IMO not having tests for this kind of code is fine, same as you wouldn't test your CI scripts.

That all said, I'm not against adding tests to codegen if they make sense (i.e. they relatively easy to implement and have the potential to catch bugs that cannot be avoided through the type system). If you find opportunities for adding such tests, let me know!

strom-und-spiele

This review is related to the general structure and modifications to the existing files. I started to look into the sub-crate as well but noticed that it makes more sense to do this in one sitting.

codegen/README.md

src/lib.rs

strom-und-spiele · 2020-08-21T11:10:08Z

.github/workflows/ci.yml

@@ -18,6 +18,8 @@ jobs:
          - stm32f302xc
          - stm32f302xd
          - stm32f302xe
+          - stm32f302x6
+          - stm32f302x8


I don't know how travis works. I was under the impression that we're only building for stm32f303xc (it also says so in the include part)
If that's the case, why list all the mcus here?
If that's not the case (and hey - thats better), we should extend the note in the README to also update this file.

There is no Travis involved, we are using Github Actions now :)
Here is the relevant documentation for matrices: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategy

Basically, it runs the job for each combination of all specified features. In our case it means, every MCU is checked with the "rt" feature. The "include" field is special and allows you to add new entries to the matrix or overwrite existing ones. We use that to specify that the Check for the stm32f303xc MCU should use the features "rt,stm32-usbd" instead of "rt". But all other MCUs are still tested. You can actually see that in the names of the checks executed for a PR.

Not sure which note in the README you mean? There is one in the Cargo.toml, that already mentions the ci.yml file:

# Any Changes here should be mirrored in README.md, src/lib.rs, and # .github/workflows/ci.yml.

There is no Travis involved, we are using Github Actions now :)
Here is the relevant documentation for matrices: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategy

Basically, it runs the job for each combination of all specified features. In our case it means, every MCU is checked with the "rt" feature. The "include" field is special and allows you to add new entries to the matrix or overwrite existing ones. We use that to specify that the Check for the stm32f303xc MCU should use the features "rt,stm32-usbd" instead of "rt". But all other MCUs are still tested. You can actually see that in the names of the checks executed for a PR.

Thanks for taking the time to explain this. I like it.

Not sure which note in the README you mean? There is one in the Cargo.toml, that already mentions the ci.yml file:

# Any Changes here should be mirrored in README.md, src/lib.rs, and # .github/workflows/ci.yml.

In the root README there is a comment on line 50 I was talking about:
[comment]: # (Any changes here should be mirrored in src/lib.rs)
I'm not sure if adding the instructions to all places where one can possibly think about makes sense or if it makes more sense to add a central note in the README on "contributing" and reference that.

Ha, good catch. Somehow I totally missed this comment in the README.

I'm not sure if adding the instructions to all places where one can possibly think about makes sense or if it makes more sense to add a central note in the README on "contributing" and reference that.

Probably a good idea. I'm also not a huge fan of having to repeat all the features so often. I think it would be better to just tell the user to pick one feature from the Cargo.toml. But that discussion doesn't really fit here I guess.

How about simply removing this comment in the README? IMO having it only in the Cargo.toml is enough, because if you want to add a new feature you will obviously do it there first. Nobody (I hope) will think that just adding a new line to the README will magically make a new feature appear in the code. Plus the README is targeted to users anyway, not developers.

src/dma.rs

This also adds two MCU versions (stm32f302x{6,8}) that we missed previously.

This commit switches to the current nightly version of the stm32f3 PAC to make use of the fixes regarding the F3 GPIO peripherals. We also need to adjust the codegen of the GPIO pin mappings a bit to account for the fact that the F373 GPIO is still special in that the PAC contains an additional `gpiod` module for it.

Setting the memory and peripheral addresses of a DMA channel has been marked unsafe in the newest version of the stm32f3 HAL. This commit adjusts the DMA abstraction code accordingly.

teskje · 2020-09-25T21:00:58Z

Updated the stm32f3 crate to version 0.12. This is ready for merging now.

dfrankland · 2020-09-25T21:57:50Z

codegen/src/main.rs

+fn parse_args() -> ArgMatches<'static> {
+    App::new("codegen")
+        .about("Code generation for the stm32f3xx-hal crate")
+        .setting(AppSettings::SubcommandRequiredElseHelp)
+        .subcommand(
+            SubCommand::with_name("gpio")
+                .about("Generate GPIO mappings from an STM32CubeMX database")
+                .arg(
+                    Arg::with_name("db_path")
+                        .help("Path of the STM32CubeMX MCU database (.../db/mcs/)")
+                        .required(true),
+                ),
+        )
+        .get_matches()
+}


It might be work it to just use structopt

You're right, that makes the code a bit nicer :)

dfrankland

Wow, this looks really great!

strom-und-spiele · 2020-09-26T09:57:01Z

Sorry, I lack the time for a throughout code review. If @dfrankland approves it I'm fine with merging it (and possibly building on top of it)

teskje commented Jul 31, 2020

View reviewed changes

teskje mentioned this pull request Aug 1, 2020

GPIO reset values for STM32F3 devices are wrong stm32-rs/stm32-rs#405

Closed

teskje force-pushed the gpio-features branch from c40f269 to d55c2e4 Compare August 7, 2020 17:06

teskje force-pushed the gpio-features branch from 798b0ea to c593a0c Compare August 8, 2020 09:57

teskje commented Aug 9, 2020

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

teskje commented Aug 14, 2020

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

teskje marked this pull request as ready for review August 16, 2020 15:01

teskje requested review from Sh3Rm4n and strom-und-spiele August 16, 2020 15:01

strom-und-spiele reviewed Aug 16, 2020

View reviewed changes

strom-und-spiele reviewed Aug 21, 2020

View reviewed changes

teskje mentioned this pull request Aug 28, 2020

added stm32f302x6, stm32f302x8 #132

Merged

teskje force-pushed the gpio-features branch 2 times, most recently from 4b4cb76 to 4dd3e83 Compare September 4, 2020 13:33

teskje requested a review from strom-und-spiele September 4, 2020 13:34

teskje added 7 commits September 25, 2020 22:32

Add codegen for GPIO mappings

c4e5a72

Add new features for GPIO peripheral versions

8376c90

This also adds two MCU versions (stm32f302x{6,8}) that we missed previously.

Use GPIO pin mappings based on GPIO version

f484f6a

Adjust to the new unsafe DMA register access methods

99cdd2f

Setting the memory and peripheral addresses of a DMA channel has been marked unsafe in the newest version of the stm32f3 HAL. This commit adjusts the DMA abstraction code accordingly.

Revert previous find-replace accidents

9c3b0c3

Revert accidential Cargo.toml autoformat

4686000

teskje added 5 commits September 25, 2020 22:37

Add codegen README

127d876

Update changelog

2cfe3f1

Build codegen with x86_64-unknown-linux-gnu by default

938bb36

codegen: Emit 'auto-generated' comment

4e9236d

codegen: Describe how 'gpio-' features are generated

add481d

teskje force-pushed the gpio-features branch 2 times, most recently from b90e56f to 209eef3 Compare September 25, 2020 20:39

Update PAC to version 0.12

d3dfa2e

teskje force-pushed the gpio-features branch from 209eef3 to d3dfa2e Compare September 25, 2020 20:41

teskje added 2 commits September 25, 2020 22:44

Fix serial module broken during merging

4d8ea24

rtc: Remove unsafes not needed after PAC update

c9a0c7e

dfrankland reviewed Sep 25, 2020

View reviewed changes

dfrankland approved these changes Sep 25, 2020

View reviewed changes

codegen: Replace clap with structopt

9b89122

teskje merged commit 54e8261 into stm32-rs:master Sep 27, 2020

teskje mentioned this pull request Sep 27, 2020

Wrong alternate function mappings for GPIO banks #118

Closed

This was referenced Dec 9, 2020

Increase readability and consistency in src/gpio.rs #126

Closed

Tracking issue for refactoring #122

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GPIO features #129

GPIO features #129

teskje commented Jul 31, 2020 •

edited

Loading

teskje commented Jul 31, 2020

teskje Jul 31, 2020

teskje Jul 31, 2020

teskje commented Aug 7, 2020 •

edited

Loading

teskje commented Aug 8, 2020

teskje commented Aug 8, 2020

strom-und-spiele commented Aug 8, 2020

teskje commented Aug 9, 2020

strom-und-spiele commented Aug 9, 2020

strom-und-spiele left a comment

teskje commented Aug 17, 2020

strom-und-spiele left a comment •

edited

Loading

strom-und-spiele Aug 21, 2020

teskje Sep 4, 2020

strom-und-spiele Sep 6, 2020

teskje Sep 9, 2020

teskje commented Sep 25, 2020

dfrankland Sep 25, 2020

teskje Sep 27, 2020

dfrankland left a comment

strom-und-spiele commented Sep 26, 2020

GPIO features #129

GPIO features #129

Conversation

teskje commented Jul 31, 2020 • edited Loading

teskje commented Jul 31, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teskje commented Aug 7, 2020 • edited Loading

teskje commented Aug 8, 2020

teskje commented Aug 8, 2020

strom-und-spiele commented Aug 8, 2020

teskje commented Aug 9, 2020

strom-und-spiele commented Aug 9, 2020

strom-und-spiele left a comment

Choose a reason for hiding this comment

teskje commented Aug 17, 2020

strom-und-spiele left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teskje commented Sep 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dfrankland left a comment

Choose a reason for hiding this comment

strom-und-spiele commented Sep 26, 2020

teskje commented Jul 31, 2020 •

edited

Loading

teskje commented Aug 7, 2020 •

edited

Loading

strom-und-spiele left a comment •

edited

Loading