Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge MicroPython V1.14 #4280

Closed
wants to merge 2,723 commits into from
Closed

Merge MicroPython V1.14 #4280

wants to merge 2,723 commits into from

Conversation

microdev1
Copy link
Collaborator

This PR merges in upstream MicroPython changes till release V1.14.

Notes :-

  • mpy-cross builds successfully. CI currently fails at buid unix port.
  • I am doing a draft PR to get the changes reviewed before adapting the rest of the codebase to it.

Changes :-

  • Upstream ran a code formatting script... so a lot of the changes are formatting related.
    We can run the formatting script later in another PR after this is merged for consistency.
  • Okay there's just too much to document here so I'd suggest to just dive write in... :)
  • To be fair... I have only taken a look at the conflicting changes... can we split this up? I am thinking based on a directory basis.
    I'll update this section once there is more info for future reference.

dpgeorge and others added 30 commits October 1, 2020 12:57
Signed-off-by: Damien George <damien@micropython.org>
This provides microsecond accuracy.

Signed-off-by: Damien George <damien@micropython.org>
It requires mp_hal_time_ns() to be provided by a port.  This function
allows very accurate absolute timestamps.

Enabled on unix, windows, stm32, esp8266 and esp32.

Signed-off-by: Damien George <damien@micropython.org>
So it can be enabled without modifying the source.

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
DMA2 clock and registers should be left in their current state in the H7
build.
Changes are:
- Fix missing IRQ handler when SDMMC2 is used instead of SDMMC1 with H7
  MCUs.
- Removed outdated H7 series compatibility macros.
- Defined common IRQ handler macro for F4 series.
The new functions provide FUS/WS status, version and SYS HCI commands:
- stm.rfcore_status()
- stm.rfcore_fw_version(fw_id)
- stm.rfcore_sys_hci(ogf, ocf, cmd)
This commit adds a script that can be run on-device to install FUS and WS
binaries from the filesystem.  Instructions for use are provided in
the rfcore_firmware.py file.

The commit also removes unneeded functionality from the existing rfcore.py
debug script (and renames it rfcore_debug.py).
This WS update to 1.9.0.0.4 broke the workaround used in rfcore for
OCF_CB_SET_EVENT_MASK2, so fix it to support WS 1.8 and 1.9.
The flash can sometimes be in an already-unlocked state, and attempting to
unlock it again will cause an immediate reset.  So make _Flash.unlock()
check FLASH_CR_LOCK to get the current state.

Also fix some magic numbers for FLASH_CR_LOCK AND FLASH_CR_STRT.

The machine.reset() could be removed because it no longer crashes now that
the flash unlock is fixed.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When installing WS firmware, the very first GET_STATE can take several
seconds to respond (especially with the larger binaries like
BLE_stack_full).

Allows stm.rfcore_sys_hci to take an optional timeout, defaulting to
SYS_ACK_TIMEOUT_MS (which is 250ms).

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The last argument of TUD_CDC_DESCRIPTOR() is the endpoint size (or
wMaxPacketSize), not the CDC RX buffer size (which can be larger than the
endpoint size).

Signed-off-by: Damien George <damien@micropython.org>
Adding a port number other then 443 to a PyPI URL may be needed if a local
server like devpi is used.
This is a generally useful feature and because it's part of the object
model it cannot be added at runtime by some loadable Python code, so enable
it on the standard unix build.
If the device is not connected over USB CDC to a host then all output to
the CDC (eg initial boot messages) is written to the CDC TX buffer with
wrapping, so that the most recent data is retained when the USB CDC is
eventually connected (eg so the REPL banner is displayed upon connection).

This commit fixes a bug in this behaviour, which was likely introduced in
e4fcd21, where the initial data in the CDC
TX buffer is repeated multiple times on first connection of the device to
the host.

Signed-off-by: Damien George <damien@micropython.org>
The function scope_find_or_add_id used to take a scope_kind_t enum and
save it in an uint8_t. Saving an enum in a uint8_t is fine, but
everywhere this function is called it is not actually given a
scope_kind_t but an anonymous enum instead. Let's give this enum a name
and use that as the argument type.

This doesn't change the generated code, but is a C type mismatch that
unfortunately doesn't show up unless you enable -Wenum-conversion.
mp_emergency_exception_buf_size is signed, so let's make sure we compare
it as such.
titouanc and others added 17 commits February 1, 2021 11:20
With a check for reproducible build date.  Invocation of the test suite is
not needed because it's already run in another job.

Signed-off-by: iTitou <moiandme@gmail.com>
Some devices have lower precision than 1ms for time_ns() (eg PYBv1.x has
3.9ms resolution of the RTC) so make the test more lenient for them.

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
This fixes machine_pin.c to build against the new pico-sdk coming down the
pipeline, whilst still working with the existing version.
In particular it fixes GPIO19 so that it can be used as an output.

Signed-off-by: Damien George <damien@micropython.org>
PIO state machines can make a conditional jump on the state of a pin: the
`JMP PIN` command.  This requires the pin to be configured with
`sm_config_set_jmp_pin`, but until now we didn't have a way of doing that
in MicroPython.

This commit adds a new `jmp_pin=None` argument to `StateMachine`.  If it is
not `None` then we try to interpret it as a Pin, and pass its value to
`sm_config_set_jmp_pin`.

Signed-off-by: Tim Radvan <tim@tjvr.org>
This was adapted from the `pio/uart_rx` example from the `pico-examples`
repository:
https://github.com/raspberrypi/pico-examples/blob/master/pio/uart_rx/uart_rx.pio

It demonstrates the `jmp_pin` feature in action.

Signed-off-by: Tim Radvan <tim@tjvr.org>
MCUs with device-only USB peripherals (eg L0, WB) do not implement (at
least not in the ST HAL) the HAL_PCD_DisconnectCallback event.  So if a USB
cable is disconnected the USB driver does not deinitialise itself
(usbd_cdc_deinit is not called) and the CDC driver can stay in the
USBD_CDC_CONNECT_STATE_CONNECTED state.  Then if the USB was attached to
the REPL, output can become very slow waiting in usbd_cdc_tx_always for
500ms for each character.

The disconnect event is not implemented on these MCUs but the suspend event
is.  And in the situation where the USB cable is disconnected the suspend
event is raised because SOF packets are no longer received.

The issue of very slow output on these MCUs is fixed in this commit (really
worked around) by adding a check in usbd_cdc_tx_always to see if the USB
device state is suspended, and, if so, breaking out of the 500ms wait loop.
This should also help all MCUs for a real USB suspend.

A proper fix for MCUs with device-only USB would be to implement or somehow
synthesise the HAL_PCD_DisconnectCallback event.

See issue #6672.

Signed-off-by: Damien George <damien@micropython.org>
With mboot encrpytion and fsload enabled, the DEBUG build -O0 compiler
settings result in mboot no longer fitting in the 32k sector.  This commit
changes this to -Og which also brings it into line with the regular stm32
build.
Default to just calling python since that is most commonly available: the
official installer or zipfiles from python.org, anaconda, nupkg all result
in python being available but not python3.  In other words: the default
used so far is wrong.  Note that os.name is 'posix' when running the python
version which comes with Cygwin or MSys2 so they are not affected by this.
However of all possible ways to get Python on Windows, only Cygwin provides
no python command so update the default way for running tests in the
README.
Added functions in the machine module are:
- unique_id (returns 8 bytes)
- soft_reset
- idle
- lightsleep, deepsleep (not power saving at the moment)
- disable_irq, enable_irq
- time_pulse_us

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
@dhalbert
Copy link
Collaborator

dhalbert commented Feb 27, 2021

Thank you for starting to work on this! The last time I did it was #1068, in 2018.

I would really like to understand the changes rather than just depend on automatic merging. To that end, I think it would be good to make some of the changes that MicroPython has made, by hand, first, so as to minimize the diffs.

For instance:

  • Code formatting: If we reformat the code in advance according to the same style rules, that will greatly reduce the number of changes.
  • Some macros were renamed or made lower-case. We can make those changes in advance.
  • Some casting styles might have been changed, and we can make those changes too.
  • We can remove things in advance that are still leftovers, or remove them in the merge. For example, we don't use drivers/ much if at all, and if we can just discard the incoming files, that would be good.
  • Many of the new things in extmod we don't need or we should bring in selectively by hand to shared-bindings, shared-module, and common-hal. So we can discard those in the merge.
  • Discard all the code for new MicroPython ports (maybe you did this already).

The worst merge issues I had were in extmod/, for the modules that we still used, like the vfat code. The py/ files also merit very careful inspection.

We changed the signatures for Python API routines in shared-bindings and the leftovers in extmod, and we need to keep those changes. Again, maybe you did that already.

[note that I've edited the above several times]

@microdev1
Copy link
Collaborator Author

@dhalbert Thanks! for the suggestions.

Code formatting: If we reformat the code in advance according to the same style rules...

I ran the same script (i.e. tools/codeformat.py) on the whole CPY codebase.
I had to modify it a bit as I got an IndexError. That reduced about 150 files from the diff.
I can make a PR for it... I think I should wait for the open PRs to come down a bit before doing that.

Some macros were renamed... casting styles might have been changed...

These can have another PR too...

We can remove things in advance that are still leftovers... we don't use drivers much if at all... Many of the new things in extmod we don't need or we should bring in selectively by hand...

Sure, I plan to rebase this a few times. These can be handled based on the reviews.

Discard all the code for new MicroPython ports... We changed the signatures for Python API routines in shared-bindings and the leftovers in extmod, and we need to keep those changes...

Ya... I have only kept changes in drivers extmod py lib mpy-cross tests tools and some files in root.
I used git blame a lot to resolve conflicts so CPY specific changes are still present.

@microdev1
Copy link
Collaborator Author

V1.15 is around the corner. Should we wait?

@tannewt
Copy link
Member

tannewt commented Mar 22, 2021

I don't think so. I'd expect merging 1.15 in to be easier after 1.14 has already been. I suspect 1.14 merge will be much more difficult. We shouldn't merge 1.14 until we branch off for 6.2.0 though because it may lead to instability.

@dhalbert
Copy link
Collaborator

I don't think so. I'd expect merging 1.15 in to be easier after 1.14 has already been. I suspect 1.14 merge will be much more difficult. We shouldn't merge 1.14 until we branch off for 6.2.0 though because it may lead to instability.

Though if 1.15 comes out before 6.2.0 is branched, I'm not sure it makes much difference if we start with 1.15.

@tannewt
Copy link
Member

tannewt commented Apr 8, 2021

We've marked the main branch as 7.x so now is the time to do it. In particular, 1.14 will bring a new mpy version. The sooner we switch to it, the simpler it'll be to match 7.x mpys to 7.x CP builds.

@tannewt
Copy link
Member

tannewt commented Apr 19, 2021

@tannewt
Copy link
Member

tannewt commented Apr 21, 2021

I've started this merge work but am taking a different approach. I'm merging each MP release from 1.10 up to 1.15. This will make the merges a bit more tractable and easier to hunt errors/bugs as we find them. (Started in #4646 )

@tannewt tannewt closed this Apr 21, 2021
@microdev1
Copy link
Collaborator Author

I'm merging each MP release from 1.10 up to 1.15. This will make the merges a bit more tractable and easier to hunt errors/bugs as we find them.

I agree, merging all of it at once is kinda overwhelming.

@microdev1 microdev1 deleted the merge-in-micropython branch October 31, 2022 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.