Add device init/de-init functionality

[gmarull]

This RFC is about adding a new core feature to the device model: device
de-initialization.

Why ?

One may think that this is an unnecessary feature since Zephyr primary goal is
to allocate resources at compile time. However, cases arise occasionally asking
for more dynamic features, see e.g. #20012, #40385.

The following scenarios would benefit from being able to de-initialize a device:

(1) Multi-functional IP (e.g. USART/SPI/I2C)
(2) Re-allocation of device resources (e.g. pin re-assignment)
(3) Runtime interpreters like Micropython (where some devices may be dynamically loaded based on user input)
(4) Bootloaders: to leave some devices in their reset state
(5) See concrete case: "Allow multiple ICE40 on the same SPI bus" #77980, #80010

One could argue that all these cases could be solved by exposing custom device
level APIs, however, since this requirement applies to a few vendors, solving
the problem in a more generic way seems like a better approach. Some may also
argue that PM can partly solve some of these problems, however, we should
acknowledge that Zephyr PM subsystem has not been widely adopted, and still
today, many of its design points are discussed #84049.

How... ?

To support de-initialization, device_deinit() is proposed. However, while all devices now have an init function, they do
not have any de-init function, so the following is introduced:

struct device_ops {
    int (*init)(const struct device *);
    int (*deinit)(const struct device *);
}

And introduce additional device definition macros that allow passing a de-init function, e.g. DEVICE_DT_DEINIT_DEFINE(node_id, init_fn, deinit_fn, ...).

/* A device that supports de-init */
static int mydev_init(const struct device *dev)
{
    ...
}

static int mydev_deinit(const struct device *dev)
{
    ...
}

DEVICE_DT_INST_DEINIT_DEFINE(0, mydev_init, mydev_deinit, ...);

Open questions

All have been discussed. The Agreement was to implement the simplest option: a new deinit API with no guarantees, no usage tracking, etc.

Samples

• Last commits, tagged with [DNM] show required driver changes or sample de-init implementations
• A generic sample that allows switching UART pins at runtime is provided. It works on both nRF5340DK and ST Nucleo H743Z, see 7d73702.

[pillo79]

Arduino is extremely interested in this functionality as well, so thanks for providing a quite comprehensive starting point for discussion! 🙇

[teburd]

Would this allow for in some manner dynamically loaded drivers using llext?

[gmarull]

Would this allow for in some manner dynamically loaded drivers using llext?

I'm not very familiar with llext, but I assume that would require more work to have dynamically loadable drivers.

[ceolin]

#44564 (comment)

[ceolin]

What is expected in the de-init() ? Cleanup all device context ?

We are clearing overlapping things with power management and we need to have better integration get() and put() here can easily hook in the device pm triggering resume and suspend actions respectively. Then we drop all the reference count logic there.

Now, how to handle ON and OFF actions ? If a device receives the action PM_DEVICE_ACTION_TURN_OFF it calls de-init if it is implemented ? PM_DEVICE_ACTION_TURN_ON could invoke the init.

Obviously pm_device_runtime_get and pm_device_runtime_put could be replaced with device_get and device_put but the async option does not, we need to come up with something to have it consistent.

@bjarki-andreasen ^

[tannewt]

This would be great for CircuitPython as well! My expectation is that de-init would de-init the device and any dependencies that are no longer in use, such as clocks.

We do "in use" tracking in CircuitPython and it is very helpful when a vendor SDK handles it too (ESP-IDF 5 does).

[aescolar]

Thanks for the proposal @gmarull. Some thoughts:
I can see the use cases in which this would be helpful, but I think any change to support something like this should be done, wherever possible, without breaking the current drivers and APIs.
Meaning, I think it would be much nicer to not change DEVICE_DT_DEFINE() at all, but maybe introduce a new macro for devices that support such a deinit.
In a similar line of thought, I would avoid deprecating device_init() & device_is_ready().
Would that not be enough to not break anything else?

Note that I expect many users will not need this feature (they will continue with just init'ing once at boot), but will dislike the churn of yet another API change, or increased RAM/code use.
Which also makes me wonder if it would make sense for this new feature to be optional, either per device or for the whole build (kconfig).

Initialization: is it a good idea to switch to a model where init is performed based on device_get() calls? Maybe not?

In general no. I expect most users will want to continuing init'ing at boot.

Anyhow, from the use cases you list, I'm not specially clear about what one should expect from such a device driver deinit:

• is it to allow that device to go to a lower power mode? (while not being immediately usable)
• is it to have that driver release taking control of other associated HW resources its needs?
• is it really to bring it (and its associated HW) back to the state after HW reset?

Should device_put() de-init the device, or just put it to "idle"? Maybe an extra device_deinit

I guess a user of the driver may want to imply "I really need this device to release the resources for something else", or "I don't need this by now", or "I don't need this ever again".
Note that this may compound with a deinit doing practically nothing, or wasting a lot of time clearing state that would require that state to be reinitialiazed in the next device_init/get().
So maybe that function needs a parameter?

About naming: device_reserve(), device_release() could sound a bit better to me.

[gmarull]

Thanks for the proposal @gmarull. Some thoughts: I can see the use cases in which this would be helpful, but I think any change to support something like this should be done, wherever possible, without breaking the current drivers and APIs. Meaning, I think it would be much nicer to not change DEVICE_DT_DEFINE() at all, but maybe introduce a new macro for devices that support such a deinit. In a similar line of thought, I would avoid deprecating device_init() & device_is_ready(). Would that not be enough to not break anything else?

I'd probably explore the options to keep macros untouched, but device_init or device_is_ready should really become equivalent to device_get(); otherwise we can't implement the solution. Maybe they can be kept changed (ie, use device_get underneath) but not deprecated for a while, to make the migration to device_get smoother.

Note that I expect many users will not need this feature (they will continue with just init'ing once at boot), but will dislike the churn of yet another API change, or increased RAM/code use. Which also makes me wonder if it would make sense for this new feature to be optional, either per device or for the whole build (kconfig).

I see this is a core feature, so we need to be careful if made optional. If we e.g. make device_get behave differently depending on a Kconfig option, some images may end up not working as expected in all scenarios.

Initialization: is it a good idea to switch to a model where init is performed based on device_get() calls? Maybe not?

In general no. I expect most users will want to continuing init'ing at boot.

I think it's not even doable right now. But once Zephyr transitions to device_get() everywhere, we could give it a try. This would actually solve some long-standing problems like the init ordering. But this has many implications IMHO and should not be our current goal.

Anyhow, from the use cases you list, I'm not specially clear about what one should expect from such a device driver deinit:

• is it to allow that device to go to a lower power mode? (while not being immediately usable)

No, de-init is not directly related to low-power states. A device may just enter idle/lp state by turning off its clock. It's true that by de-initing a device, system will likely consume less power, but that is not the goal of this API.

• is it to have that driver release taking control of other associated HW resources its needs?

Yes, it must release any obtained resources (e.g. a clock, a DMA channel, etc.)

• is it really to bring it (and its associated HW) back to the state after HW reset?

Yes, leave peripheral as in reset state. Some MCUs can easily do that (STM32).

Should device_put() de-init the device, or just put it to "idle"? Maybe an extra device_deinit

I guess a user of the driver may want to imply "I really need this device to release the resources for something else", or "I don't need this by now", or "I don't need this ever again". Note that this may compound with a deinit doing practically nothing, or wasting a lot of time clearing state that would require that state to be reinitialiazed in the next device_init/get(). So maybe that function needs a parameter?

A parameter or extra APIs could be options. I thought about this, if we integrate PM functionality into the device model itself.

About naming: device_reserve(), device_release() could sound a bit better to me.

[gmarull]

What is expected in the de-init() ? Cleanup all device context ?

Leave device in its reset state, also releasing any obtained resources.

We are clearing overlapping things with power management and we need to have better integration get() and put() here can easily hook in the device pm triggering resume and suspend actions respectively. Then we drop all the reference count logic there.

I agree, this opens the door to de-duplicate some efforts.

Now, how to handle ON and OFF actions ? If a device receives the action PM_DEVICE_ACTION_TURN_OFF it calls de-init if it is implemented ? PM_DEVICE_ACTION_TURN_ON could invoke the init.

Maybe for ON we can re-use init, but for OFF perhaps de-init is not always necessary.

Obviously pm_device_runtime_get and pm_device_runtime_put could be replaced with device_get and device_put but the async option does not, we need to come up with something to have it consistent.

Note that here, device_put would end up de-initializing a device, not suspending it. As noted in the RFC, we could change this by making device_put suspend, and device_deinit do an actual de-init (in general a heavier operation).

@bjarki-andreasen ^

[benediktibk]

Regarding the naming: device_open/device_close seems to be more obvious than device_get/device_put.

[bjarki-andreasen]

Notes for arch meeting:

• device_deinit() is a simple and major improvement to jumping between apps, to properly "reset" hardware. With a new DEVICE_DT_ macro to define it it is non-breaking as well. Note PM is not suited for deinit, OFF or SUSPEND is not reset/unload.
• device_get(), if implemented everywhere, potentially solves the device driver init prio problem entirely. Its breaking but we will likely win back the time by not having to chase down init order related bugs :)

[carlescufi]

Architecture WG:

• @gmarull presents the proposal
• @fabiobaltieri finds it a bit strange that device_get() can, in some cases, call init() internally. Perhaps we should change the auto-initialization to be manual?
• @henrikbrixandersen replies that this may not be possible, because many device requires being available early on in the kernel boot process
• @henrikbrixandersen asks what happens with devices that do not support natively an init/deinit cycle. @gmarull replies that they put() implementation may be empty
• @cfriedt worries about reference counting, in case for example a thread dies. Perhaps using a mutex instead? @gmarull will think about it a bit
• @bjarki-andreasen would like a very simple approach where you only introduce a deinit() function in struct device
• @pillo79 mentions that Arduino is very interested in this functionality. He suggests perhaps having support for device_get/put() in devices that are set to use deferred initialization
• @jfischer-no asks what state should deinit() set the hardware to. @gmarull replies that all resources should be freed, and the hardware should be set to its boot state
• @fabiobaltieri why do we need deinit() if we have a power domain off? @gmarull states that power management does not currently offer a real way to implement the equivalent of deinit(). We need to work together
• @bjarki-andreasen says that power management talks about power domains being shut down and the peripherals losing power, which is quite different from the purpose of deinit() here
• @henrikbrixandersen likes the proposal from Luca to limit it to devices whose initialization has been deferred, making the whole concept effectively optional

[fabiobaltieri]

Note that here, device_put would end up de-initializing a device, not suspending it. As noted in the RFC, we could change this by making device_put suspend, and device_deinit do an actual de-init (in general a heavier operation).

It'd be very tempting to suggest the this could actually replace PM runtime entirely, the init - get - put - deinit looks very much like the whole resume - on - suspend - off in pm runtime/power domains. I understand that there's some differences now but this may be a chance to revisit the whole device states story and come back with a coherent story, and since it'd be a breaking change that'd be a chance of rewriting current pm enabled drivers to stick to the new structure and have some better samples in the code base that then all other driver developers will undoubtedly use as references for all new drivers and one year from now the whole code base is full wonderful perfectly implemented drivers with suspend/resume/on/off support.

[swift-tk]

For runtime multi-target switching or runtime pinctrl assignment, aren't we supposed to improve pinctrl?
All I did to workaround current pinctrl for multi-target switching in Ambiq mspi controller is to redefine some of the pinctrl macro for declaration.

[bjarki-andreasen]

For runtime multi-target switching or runtime pinctrl assignment, aren't we supposed to improve pinctrl? All I did to workaround current pinctrl for multi-target switching in Ambiq mspi controller is to redefine some of the pinctrl macro for declaration.

deinit is a bit more flexible as it also "releases" resources like clocks, dma channels, shared buffers, theoretically interrupts? etc. With deinit, we have one simple mechanism for "freeing" all resources from the applications perspective, compared to enforcing the application/subsystems to know of and interact with individual resources like pinctrl states directly.

[gmarull]

Last push:

• Add new functionality as a non-breaking change

What I think should be discussed (next Arch WG):

• How to de-init:
  • When calling device_put and refcnt hits 0
  • Explicit device_deinit, only allowed if refcnt is at 0 (I'm leaning towards that)
  • ...?
• Is refcnt actually needed?
  • In general, it is unsafe to access devices concurrently. But... not always. So, refcnt is in practice only useful in some cases... Should applications manage this entirely? Could this lead into a can of worms?
  • If we ever have true device classes, class-level APIs would likely replace the de-init API with something more specific.

[swift-tk]

deinit is a bit more flexible as it also "releases" resources like clocks, dma channels, shared buffers, theoretically interrupts? etc. With deinit, we have one simple mechanism for "freeing" all resources from the applications perspective, compared to enforcing the application/subsystems to know of and interact with individual resources like pinctrl states directly.

I feel like we are going in the direction where the devices are hotpluggable like in Linux while the devicetree is still compile time not run time. Although you can free up only some of the resources maybe even very limited but not ram usages.

[swift-tk]

I see that usage_lock have the potential to replace all the individual k_sem in each device drivers. Does this mean we can safely remove all the acquire and release in each API implementation?

[swift-tk]

I don't see how device_get/device_put can guarantee the safety of switching pinctrl... Should be checking usage count as zero and then switch pin? Only the device driver will know its usage, particularly with async methods, even with a usage count zero does not guarantee the async operation had completed, the device_put should involve checking the device status with the device driver instead of making assumptions.

[gmarull]

Last push: should be the final version.

Last 4 commits will be dropped and moved somewhere else, they're still here to illustrate a usecase.

[swift-tk]

what is wrong with using section iterable… thought it was neater than having this additional device flag?

[cfriedt]

@gmarull - would be great to ensure bisectability of tests/drivers/console_switching with twister. My intuition says that commits 1, 2, 3, 4, and 10 might need to be squashed.

Otherwise, LGTM.

[JordanYates]

Jumping in late here since I only just found out about this addition.
Shouldn't this new feature for the core device API have included some level of documentation? @kartben

[bjarki-andreasen]

I'm actually working on it :) ATM, its the primary reason for #90275 since we need an actual way to deinit which can then be documented and added to samples like #89927

[JordanYates]

ATM, its the primary reason for #90275 since we need an actual way to deinit which can then be documented and added to samples like #89927

So we have a core API function, other subsystems adding functionality to support it and samples being added before we document what the functions are supposed to do, how drivers should implement them, and how applications should use them? That seems backwards to me, how can we review whether code implements an API if the API isn't documented? If it isn't clear from the start we will just end up with every driver doing its own thing.

[bjarki-andreasen]

ATM, its the primary reason for #90275 since we need an actual way to deinit which can then be documented and added to samples like #89927

So we have a core API function, other subsystems adding functionality to support it and samples being added before we document what the functions are supposed to do, how drivers should implement them, and how applications should use them? That seems backwards to me, how can we review whether code implements an API if the API isn't documented? If it isn't clear from the start we will just end up with every driver doing its own thing.

It was merged after multiple review rounds and arch meetings with clear use cases in the description of this PR. This is what its supposed to do:

zephyr/include/zephyr/device.h

Lines 877 to 894 in 11c543e

	/**
	* @brief De-initialize a device.
	*
	* When a device is de-initialized, it will release any resources it has
	* acquired (e.g. pins, memory, clocks, DMA channels, etc.) and its status will
	* be left as in its reset state.
	*
	* @warning It is the responsibility of the caller to ensure that the device is
	* ready to be de-initialized.
	*
	* @param dev device to be de-initialized.
	*
	* @retval 0 If successful
	* @retval -EPERM If device has not been initialized.
	* @retval -ENOTSUP If device does not support de-initialization.
	* @retval -errno For any other errors.
	*/
	__syscall int device_deinit(const struct device *dev);

[gmarull]

A quick note here: device_deinit() is heavily unsafe (just see the scription saying @warning It is the responsibility of the caller to ensure that the device is ready to be de-initialized.). This means in practice its usage is easily application-specific, so building anything on top of this API should be considered dangerous and equally unsafe.

[JordanYates]

It was merged after multiple review rounds and arch meetings with clear use cases in the description of this PR

I'm not suggesting it wasn't discussed, I'm suggesting it is not well documented.

What does "release any resources" mean in practice? Does it only mean getting the software back to a state such that the hardware can be re-used for a different purpose (nRF I2C vs SPI for example)? Does it also include hardware state that doesn't have any access control (Should every GPIO be put back into the default state as it was after a SoC reboot)?

What does "its status will be left as in its reset state" mean in practice? Does this mean anything beyond the "release any resources" requirement previously listed? Are we explicitly trying to put devices back into the state they were on a power up (even if that is a worse idling state)?

This means in practice its usage is easily application-specific, so building anything on top of this API should be considered dangerous and equally unsafe.

Usage being application-specific is fine, but if the API is hand-wavy, there is a question as to what exactly PM for example should be doing in response. This is mostly in response to the above points.

And if the answer is that what "de-init" means is specific to individual drivers, where can drivers document what their particular implementation does (and the API wrapper should point users to that)?

[gmarull]

It was merged after multiple review rounds and arch meetings with clear use cases in the description of this PR

I'm not suggesting it wasn't discussed, I'm suggesting it is not well documented.

What does "release any resources" mean in practice? Does it only mean getting the software back to a state such that the hardware can be re-used for a different purpose (nRF I2C vs SPI for example)? Does it also include hardware state that doesn't have any access control (Should every GPIO be put back into the default state as it was after a SoC reboot)?

What does "its status will be left as in its reset state" mean in practice? Does this mean anything beyond the "release any resources" requirement previously listed? Are we explicitly trying to put devices back into the state they were on a power up (even if that is a worse idling state)?

This means in practice its usage is easily application-specific, so building anything on top of this API should be considered dangerous and equally unsafe.

Usage being application-specific is fine, but if the API is hand-wavy, there is a question as to what exactly PM for example should be doing in response. This is mostly in response to the above points.

And if the answer is that what "de-init" means is specific to individual drivers, where can drivers document what their particular implementation does (and the API wrapper should point users to that)?

I think the spec is clear enough: release any taken resources and leave peripheral in its reset state. In some platforms, this may have some unforeseen side effects, but well, that's what happens when trying to abstract something that is difficult (or impossible) to abstract. I'd say it's not a problem of this specific interface, though, some other Zephyr APIs suffer from the same (just see the reboot API).