[Exporters] Refactor Uvision and IAR and initial support of CMSIS projects #2708

sarahmarshy · 2016-09-14T16:11:13Z

Description

Exporters
This PR removes the dependencies on progen and progen definitions. Instead, target information necessary for uvision project files is retrieved from CMSIS packs using the device name (which has been added to targets.json) and @theotherjimmy's arm pack manager. Arm pack manager has been moved to mbed tools, and you can view the history in the commits of this PR.

IAR continues to require magic strings, but they are now stored in iar_definitions.json in tools/export/iar. The definitions are indexed by the same device name found in targets.json.

Preferred debugger is set per vendor, as found in the CMSIS pack.

Uvision4 support has been removed.

In addition, the ability to export to a .cpdsc CMSIS project has been added (http://arm-software.github.io/CMSIS_5/Pack/html/cpdsc_pg.html). You can open this file in Uvision to see a project. but it will not build as Keil has not implemented specification of macros and scatter files. This is under active development.

Build tests
Export-build test now exports projects in parallel. It is also capable of testing the export of mbed-os tests. They can be found in the mbed-os/TESTS. The targets and tests are by default those from the latest release, though you can specify previous releases.

Status

READY

Todos

Tests
Documentation

Steps to test or reproduce

To export
mbed export -m k64f -i uvision5
mbed export -m k64f -i iar
mbed export -m k64f -i cmsis

To run build tests
Please add UV4.exe and IarBuild.exe to your path.
On Windows:
UV4 is usually found in:
C:\Keil_v5\UV4\UV4.exe

IarBuild is usually found in:
C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.x\common\bin\IarBuild.exe

python tools/test/export/build_test.py --release 2 -n MBED_BLINKY (blinky is default for release 2)
python tools/test/export/build_test.py --release 5 -os-tests tests-mbedmicro-rtos-mbed-basic (though 5 is default release, and this basic test is the default test for release 5)

@sg- @theotherjimmy @0xc0170 @screamerbg

bridadan · 2016-09-14T16:21:11Z

hal/targets.json

+        "release_versions": ["2", "5"],
+        "device_name": "NUC472HI8AE"
+    },
+    "NCS36510": {


It looks like there was an issue rebasing, there are two entries of NCS36510 now.

bridadan · 2016-09-14T16:27:15Z

Awesome work, just a few comments. It should be clarified that this PR completely removes uVision 4 exporting, not just deprecates it, correct?

sarahmarshy · 2016-09-14T17:19:33Z

/morph export-build

mbed-bot · 2016-09-14T17:19:55Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph export-build

0xc0170 · 2016-09-15T10:33:24Z

Preferred debugger is set per vendor, as found in the CMSIS pack.

Cmsis pack is for MCU, not a target/platform?

This PR removes the dependencies on progen and progen definitions. Instead, target information necessary for uvision project files is retrieved from CMSIS packs using the device name (which has been added to targets.json) and @theotherjimmy's arm pack manager.

I like this idea to get this info from the packs, not to scrape the project files. I would imagine to have a separate arm pack manage python module, that can be reused. For instance, progen would use it to eliminate progen def, as it's in here, or any other python module that requires some info from cmsis packs. Have you guys considered it? If not, someone else will :)

What I don't like it's a duplication effort. If there can be a generic python module handling cmsis-packs, also another module for generating projects (in this case it would be progen or anything else you guys can write if you don't like what's out there), why to put all in here and not be able to reuse? I believe many of us would benefit from it (users, projects). That's my 5 cents. This is fine as it is ,it has been in this codebase.

You eliminated custom beetle uvision project, was this tested it works with generic one? Because there were some settings for that target.

Once reviewed, please clean the log - for instance "arm_pack_manager - ya gotta start somewhere" :-)

You are missing .ewd files for IAR that is important to have debugger settings.

sarahmarshy · 2016-09-15T14:11:42Z

Sorry, as per the pack meaning that the vendor comes from the pack, and we use that to set the debugger.

sarahmarshy · 2016-09-15T14:21:29Z

@theotherjimmy and I made the decision to move armpackmanager into mbed, as it would eliminate the volley of version bumps when the module changes.

The build system of mbed provides a lot of vital information. I think it is best to have exporters integrated with mbed tools, rather than existing as a separate, generic entity. I think this will make exporters more reliable, rather than trying to cater to cases outside of mbed. I appreciate that this is an opinion, and if mbed users somehow demand it, we should consider it. As it is, I think this will potentially provide a better experience for both users and developers.

bridadan · 2016-09-15T16:18:25Z

An interesting point that @sarahmarshy brings up, it may be good to keep the pack manager in mbed for now as changes are made to it. When it becomes more stable, it could always be brought out as a separate package.

theotherjimmy · 2016-09-15T16:20:37Z

@bridadan that is exactly what I was thinking when I made the decision to include it in this PR. After watching the number of changes that happened within arm_pack_manager caused by the needs of this PR, I was unwilling to let it live on it's own just yet.

theotherjimmy · 2016-09-15T16:24:35Z

@0xc0170

Once reviewed, please clean the log - for instance "arm_pack_manager - ya gotta start somewhere" :-)

I prefer to do it before review, so that review is done on the final commit series.

When you say clean the log, do you mean to squash the arm_pack_manager commits, or do you want the commit messages edited to be more informative?

0xc0170 · 2016-09-16T07:57:37Z

When you say clean the log, do you mean to squash the arm_pack_manager commits, or do you want the commit messages edited to be more informative?

@theotherjimmy : Mainly latter, but also squashing for some that should not be there. I would propose to send package manager separately, and this changeset on top of it. We are mixing again various logical changes here.

Sorry, as per the pack meaning that the vendor comes from the pack, and we use that to set the debugger.

@sarahmarshy : Please check if a debugger is correct for some platforms (jlink - efm32 platforms, stlink, cmsis-dap). Otherwise this will cause a regression.

@bridadan that is exactly what I was thinking when I made the decision to include it in this PR. After watching the number of changes that happened within arm_pack_manager caused by the needs of this PR, I was unwilling to let it live on it's own just yet.

@theotherjimmy : That would be nice if that happens in the future, however I would consider it would be much better to start right from the beginning, as a separate module brings other challenges (proper versioning, self contained module - not using anything from this codebase, clean API as its user facing, this is not as tools are magic to a user).
If lot of changes are planned, will be the same if it's in this module or outside (the main thing there would be to use proper semantic versioning, and state that the module is under heavy development). The only thing I understand that having it as a separate module brings bigger challenge as stated above - provide user facing API. But again, this is a plus! And providing tests !

The build system of mbed provides a lot of vital information. I think it is best to have exporters integrated with mbed tools, rather than existing as a separate, generic entity. I think this will make exporters more reliable, rather than trying to cater to cases outside of mbed. I appreciate that this is an opinion, and if mbed users somehow demand it, we should consider it. As it is, I think this will potentially provide a better experience for both users and developers.

@sarahmarshy :
If anyone thinks this way, there won't be any reusable module out there to be used ;) What can be made generic, should be made reusable. I would accept this as to have a solution quickly in place, but let's think about the long run. Think that tools might once be completely replaced by something else, you will need to refactor the entire exporters to a new codebase. This already happened once. From my experience, I would always vote for having something generic that can be used outside of mbed, you might get much more traction (contributions -> user base). Yes, there are some costs, to consider some generic implications as you noted, handling much more user cases. I find this as a benefit ! Because you never know when one of those come to your way.
People tried to pull out exporters from mbed previously, and it was almost impossible job. me or @sg- can tell you :-)

Regarding progen:
I haven't discussed with you guys why progen is being pulled out, where it failed, and why we did not try to make it work for mbed. As this is already here as a patch, I'll find time to talk to you guys, create tracking issues for progen, and find time to fix those problems, so other projects can use it.

To summarize:
Please send it as a separate patch for adding package manager (this changeset should be just for exporters), and describe what it does -> provide a documentation, and how is it used.
Having own exporters - not completely on board, but if the rest of our team agrees, fine with me. I just wanted to highlight my concerns from the experience. I still believe there can be generic exporter module, but that's for another discussion.

sarahmarshy · 2016-09-16T17:07:13Z

Please send it as a separate patch for adding package manager (this changeset should be just for exporters),

@0xc0170 package manager support in mbed is inextricably linked to this PR. It is necessary for using CMSIS packs as device information within Uvision and providing support for CMSIS projects. It's only function within mbed (for now) is as an exporter aid, so I think is this changeset is just for exporters

sarahmarshy · 2016-09-21T20:09:36Z

You eliminated custom beetle uvision project, was this tested it works with generic one? Because there were some settings for that target.

@0xc0170 Can you clarify what those settings are? I don't have a beetle board to test, unfortunately. Looking at the diff starting here, I see that a generic ARM cortex target is used. This PR defaults to the generic CPU target if the device name is not present or is invalid. I've exported to Beetle, successfully set the target to Cortex-M3 and built, but I can't try flashing. I'll have to try to try to get ahold of a board. If you have one, could you try using this branch to export/flash in Uvision5?

sg- · 2016-09-22T05:39:16Z

tools/test/export/build_test.py

-    targetnames = TARGET_NAMES
-    targetnames.sort()
+
+    ide_list = ["iar", "uvision5"]


seems like a list of tests and ide should be in a json file for easy extension. Maybe another pr though

sg- · 2016-09-22T05:40:13Z

tools/test/export/build_test.py

+        test: string name to index all_os_tests
+    returns: tuple of test_name and source location of test,
+        as given by find_tests"""
+    all_os_tests = find_tests(ROOT, "K64F", "ARM")


Is this the only combination that is run?

No, it will run the specified tests for all specified targets. This function unfortunately requires a target to return a dictionary of possible tests. However, most tests in the TESTS folder at the root of mbed-is should also support all other v5 boards (in addition to K64F). That is my understanding of that function thanks to @bridadan

@sarahmarshy is right. She brings up a valid point thought that being able to supply None or something to the the target/toolchain could maybe bring up tests that apply to all target toolchain/combinations. Then you wouldn't have to pick just one combination of target and toolchain to get a list of tests. This would be a separate PR though

sg- · 2016-09-22T05:44:36Z

/morph export-build

mbed-bot · 2016-09-22T05:45:22Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph export-build

sg- · 2016-09-22T06:16:32Z

@sarahmarshy needs some conflicts resolved.

bridadan · 2016-10-05T01:18:22Z

targets/targets.json

-            "S130",
-            "TARGET_MCU_NRF51822"
-        ],
+        "macros": ["NRF51", "TARGET_NRF51822", "BLE_STACK_SUPPORT_REQD", "SOFTDEVICE_PRESENT", "S130", "TARGET_MCU_NRF51822"],


Could revert this formatting change?

bridadan · 2016-10-05T01:19:13Z

targets/targets.json

            "lf_clock_src": {
                "value": "NRF_LF_SRC_XTAL",
-                "macro_name": "MBED_CONF_NORDIC_NRF_LF_CLOCK_SRC"
-            },
+                "macro_name": "MBED_CONF_NORDIC_NRF_LF_CLOCK_SRC"},


Could the ending } go on the next line?

bridadan · 2016-10-05T01:19:32Z

targets/targets.json

            "lf_clock_src": {
                "value": "NRF_LF_SRC_XTAL",
-                "macro_name": "MBED_CONF_NORDIC_NRF_LF_CLOCK_SRC"
-            },
+                "macro_name": "MBED_CONF_NORDIC_NRF_LF_CLOCK_SRC"},


Same here about the }

bridadan · 2016-10-05T01:21:20Z

targets/targets.json

-            "NRF52_PAN_62",
-            "NRF52_PAN_63"
-        ],
+        "macros_add": ["BOARD_PCA10040", "NRF52_PAN_12", "NRF52_PAN_15", "NRF52_PAN_58", "NRF52_PAN_55", "NRF52_PAN_54", "NRF52_PAN_31", "NRF52_PAN_30", "NRF52_PAN_51", "NRF52_PAN_36", "NRF52_PAN_53", "S132", "CONFIG_GPIO_AS_PINRESET", "BLE_STACK_SUPPORT_REQD", "SWI_DISABLE0", "NRF52_PAN_20", "NRF52_PAN_64", "NRF52_PAN_62", "NRF52_PAN_63"],


~~Can you change the formatting here?~~

Nevermind, seems to be inconsistent across a lot of the targets. Looks fine!

sarahmarshy · 2016-10-05T19:02:51Z

/morph export-build

bridadan

Thanks @sarahmarshy for putting up with all of my complaints! Changes LGTM, let's see what CI says after going through a final run

mbed-bot · 2016-10-05T19:51:00Z

Result: ABORTED

Your command has finished executing! Here's what you wrote!

/morph export-build

bridadan · 2016-10-05T20:21:02Z

/morph export-build

mbed-bot · 2016-10-05T20:40:45Z

Result: ABORTED

Your command has finished executing! Here's what you wrote!

/morph export-build

bridadan · 2016-10-05T21:17:49Z

/morph export-build

mbed-bot · 2016-10-05T22:45:49Z

Result: SUCCESS

Your command has finished executing! Here's what you wrote!

/morph export-build

0xc0170 · 2016-10-07T14:50:09Z

@sarahmarshy Is there a documentation for iar how to add a new MCU? and how it works for uvision (cmsis packs)? I could not locate any readme in the exporters. Is it somewhere else?

bridadan · 2016-10-07T15:52:51Z

@0xc0170 Wouldn't it be better to leave that documentation to the IDEs?

bridadan reviewed Sep 14, 2016
View reviewed changes

sarahmarshy changed the title ~~Refactor Uvision and IAR~~ Refactor Uvision and IAR and initial support of CMSIS projects Sep 14, 2016

sarahmarshy changed the title ~~Refactor Uvision and IAR and initial support of CMSIS projects~~ [Exporters] Refactor Uvision and IAR and initial support of CMSIS projects Sep 14, 2016

sg- added needs: work needs: CI and removed needs: CI labels Sep 16, 2016

sarahmarshy mentioned this pull request Sep 20, 2016

New exporter build tests #2754

Merged

sarahmarshy force-pushed the uvision-and-iar branch 3 times, most recently from 9de3b87 to b5bd0cf Compare September 21, 2016 20:04

sg- reviewed Sep 22, 2016

View reviewed changes

sarahmarshy mentioned this pull request Sep 22, 2016

Relocates IAR --no_static_destruction to compile command only #2787

Closed

sarahmarshy force-pushed the uvision-and-iar branch from 0ee5b2a to b99996d Compare September 23, 2016 16:03

bridadan reviewed Oct 5, 2016

View reviewed changes

bridadan mentioned this pull request Oct 5, 2016

restructure - Restructure cmsis directory #2896

Merged

7 tasks

sarahmarshy force-pushed the uvision-and-iar branch from 0b06f5e to d764d7d Compare October 5, 2016 01:54

More formatting changes in targets.json

2a8c9ef

sarahmarshy force-pushed the uvision-and-iar branch from d764d7d to 2a8c9ef Compare October 5, 2016 19:00

bridadan added needs: CI and removed do not merge needs: work labels Oct 5, 2016

bridadan approved these changes Oct 5, 2016

View reviewed changes

sarahmarshy mentioned this pull request Oct 5, 2016

No export options for nRF52-DK yet #2925

Closed

bridadan added ready for merge and removed needs: CI labels Oct 5, 2016

sg- merged commit 76fb1f8 into ARMmbed:master Oct 6, 2016

sg- added release-version: 5.2.0-rc2 and removed ready for merge labels Oct 6, 2016

This was referenced Oct 17, 2016

[Exporters] Uvision5 project configuration missing Program Algorithm List #3049

Closed

[Exporters] Download of CMSIS packs #3051

Closed

This was referenced Oct 31, 2016

[Exporters] Add a device_name to microbit entry in targets.json #3173

Merged

Confusing "Target not supported" errors from absent device_name field in targets.json #3174

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Exporters] Refactor Uvision and IAR and initial support of CMSIS projects #2708

[Exporters] Refactor Uvision and IAR and initial support of CMSIS projects #2708

sarahmarshy commented Sep 14, 2016 •

edited

Loading

bridadan Sep 14, 2016 •

edited

Loading

bridadan commented Sep 14, 2016

sarahmarshy commented Sep 14, 2016

mbed-bot commented Sep 14, 2016

0xc0170 commented Sep 15, 2016 •

edited

Loading

sarahmarshy commented Sep 15, 2016

sarahmarshy commented Sep 15, 2016 •

edited

Loading

bridadan commented Sep 15, 2016

theotherjimmy commented Sep 15, 2016

theotherjimmy commented Sep 15, 2016 •

edited

Loading

0xc0170 commented Sep 16, 2016 •

edited

Loading

sarahmarshy commented Sep 16, 2016 •

edited

Loading

sarahmarshy commented Sep 21, 2016 •

edited

Loading

sg- Sep 22, 2016

sg- Sep 22, 2016

sarahmarshy Sep 22, 2016

bridadan Sep 22, 2016 •

edited

Loading

sg- commented Sep 22, 2016

mbed-bot commented Sep 22, 2016

sg- commented Sep 22, 2016

bridadan Oct 5, 2016

bridadan Oct 5, 2016

bridadan Oct 5, 2016

bridadan Oct 5, 2016 •

edited

Loading

sarahmarshy commented Oct 5, 2016

bridadan left a comment

mbed-bot commented Oct 5, 2016

bridadan commented Oct 5, 2016

mbed-bot commented Oct 5, 2016

bridadan commented Oct 5, 2016

mbed-bot commented Oct 5, 2016

0xc0170 commented Oct 7, 2016

bridadan commented Oct 7, 2016

[Exporters] Refactor Uvision and IAR and initial support of CMSIS projects #2708

[Exporters] Refactor Uvision and IAR and initial support of CMSIS projects #2708

Conversation

sarahmarshy commented Sep 14, 2016 • edited Loading

Description

Status

Todos

Steps to test or reproduce

bridadan Sep 14, 2016 • edited Loading

Choose a reason for hiding this comment

bridadan commented Sep 14, 2016

sarahmarshy commented Sep 14, 2016

mbed-bot commented Sep 14, 2016

0xc0170 commented Sep 15, 2016 • edited Loading

sarahmarshy commented Sep 15, 2016

sarahmarshy commented Sep 15, 2016 • edited Loading

bridadan commented Sep 15, 2016

theotherjimmy commented Sep 15, 2016

theotherjimmy commented Sep 15, 2016 • edited Loading

0xc0170 commented Sep 16, 2016 • edited Loading

sarahmarshy commented Sep 16, 2016 • edited Loading

sarahmarshy commented Sep 21, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bridadan Sep 22, 2016 • edited Loading

Choose a reason for hiding this comment

sg- commented Sep 22, 2016

mbed-bot commented Sep 22, 2016

sg- commented Sep 22, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bridadan Oct 5, 2016 • edited Loading

Choose a reason for hiding this comment

sarahmarshy commented Oct 5, 2016

bridadan left a comment

Choose a reason for hiding this comment

mbed-bot commented Oct 5, 2016

bridadan commented Oct 5, 2016

mbed-bot commented Oct 5, 2016

bridadan commented Oct 5, 2016

mbed-bot commented Oct 5, 2016

0xc0170 commented Oct 7, 2016

bridadan commented Oct 7, 2016

sarahmarshy commented Sep 14, 2016 •

edited

Loading

bridadan Sep 14, 2016 •

edited

Loading

0xc0170 commented Sep 15, 2016 •

edited

Loading

sarahmarshy commented Sep 15, 2016 •

edited

Loading

theotherjimmy commented Sep 15, 2016 •

edited

Loading

0xc0170 commented Sep 16, 2016 •

edited

Loading

sarahmarshy commented Sep 16, 2016 •

edited

Loading

sarahmarshy commented Sep 21, 2016 •

edited

Loading

bridadan Sep 22, 2016 •

edited

Loading

bridadan Oct 5, 2016 •

edited

Loading