uVisor prevents LwIP/Hal from properly using ENET hardware when enabled

[sherrellbc]

EDIT: I changed the issue's name to more appropriately track the purpose of this thread after relevant information has been uncovered surrounding the original problem.

---

I have been trying to determine the appropriate uVisor ACL such that I can use the Ethernet hardware on the FRDM K64F development board. After a few hours of debugging and tracing the problem I managed to determine two points of failure when using EthernetInterface.cpp and (a dependency thereof) the LWIP implementation; the root of the corresponding source tree can be found here.

The first problem I found was that the LWIP implementation was trying to disable the MPU, which I thought might be problematic for uVisor.

MPU->CESR &= ~MPU_CESR_VLD_MASK;

https://github.com/ARMmbed/mbed-os/blob/master/features/net/FEATURE_IPV4/lwip-interface/lwip-eth/arch/TARGET_Freescale/hardware_init_MK64F12.c#L41

I have defined the MPU as an enabled memory range in my ACL, but that does not prevent the system from stopping on that instruction.

const UvisorBoxAclItem g_main_acl[] = {     
         {SIM,    sizeof(*SIM),    UVISOR_TACLDEF_PERIPH}, 
         {OSC,    sizeof(*OSC),    UVISOR_TACLDEF_PERIPH}, 
         {MCG,    sizeof(*MCG),    UVISOR_TACLDEF_PERIPH}, 
         {PORTA,  sizeof(*PORTA),  UVISOR_TACLDEF_PERIPH}, 
         {PORTB,  sizeof(*PORTB),  UVISOR_TACLDEF_PERIPH}, 
         {PORTC,  sizeof(*PORTC),  UVISOR_TACLDEF_PERIPH}, 
         {PORTD,  sizeof(*PORTD),  UVISOR_TACLDEF_PERIPH}, 
         {PORTE,  sizeof(*PORTE),  UVISOR_TACLDEF_PERIPH}, 
         {RTC,    sizeof(*RTC),    UVISOR_TACLDEF_PERIPH}, 
         {LPTMR0, sizeof(*LPTMR0), UVISOR_TACLDEF_PERIPH}, 
         {PIT,    sizeof(*PIT),    UVISOR_TACLDEF_PERIPH}, 
         {SMC,    sizeof(*SMC),    UVISOR_TACLDEF_PERIPH}, 
         {UART0,  sizeof(*UART0),  UVISOR_TACLDEF_PERIPH}, 
         {I2C0,   sizeof(*I2C0),   UVISOR_TACLDEF_PERIPH},
         {SPI0,   sizeof(*SPI0),   UVISOR_TACLDEF_PERIPH},
         {MPU,  sizeof(*MPU), UVISOR_TACLDEF_PERIPH}, 

         /* EthernetInterface */
         {ENET,   sizeof(*ENET),   UVISOR_TACLDEF_PERIPH}, 
         {EWM,    sizeof(*EWM),   UVISOR_TACLDEF_PERIPH}, 
};

---

The second problem was related to a call to EnableIRQ in fsl_enet.c. At the highest level, a call to EthernetInterface.connect() will lead to execution of the following.

/* Enables Ethernet interrupt and NVIC. */
    ENET_EnableInterrupts(base, config->interrupt);
    if (config->interrupt & (kENET_RxByteInterrupt | kENET_RxFrameInterrupt))
    {
        EnableIRQ(s_enetRxIrqId[instance]);
    }
    if (config->interrupt & (kENET_TxByteInterrupt | kENET_TxFrameInterrupt))
    {
        EnableIRQ(s_enetTxIrqId[instance]);
    }
    if (config->interrupt & (kENET_BabrInterrupt | kENET_BabtInterrupt | kENET_GraceStopInterrupt | kENET_MiiInterrupt |
                             kENET_EBusERInterrupt | kENET_LateCollisionInterrupt | kENET_RetryLimitInterrupt |
                             kENET_UnderrunInterrupt | kENET_PayloadRxInterrupt | kENET_WakeupInterrupt))
    {
        EnableIRQ(s_enetErrIrqId[instance]);
    }

https://github.com/ARMmbed/mbed-os/blob/master/hal/targets/hal/TARGET_Freescale/TARGET_KSDK2_MCUS/TARGET_MCU_K64F/drivers/fsl_enet.c#L453

As it turns out, EnableIRQ() is an inline function defined in fsl_common.h as a link to an NVIC_EnableIRQ call:

static inline void EnableIRQ(IRQn_Type interrupt)
{
#if defined(FSL_FEATURE_SOC_INTMUX_COUNT) && (FSL_FEATURE_SOC_INTMUX_COUNT > 0)
    if (interrupt < FSL_FEATURE_INTMUX_IRQ_START_INDEX)
#endif
    {
        NVIC_EnableIRQ(interrupt);
    }
}

https://github.com/ARMmbed/mbed-os/blob/master/hal/targets/hal/TARGET_Freescale/TARGET_KSDK2_MCUS/TARGET_MCU_K64F/drivers/fsl_common.h#L182

I could not trace the NVIC definition down any further as there were a very large number of them. However, I did check out the generated source code in a disassembler and it seems that NVIC_EnableIRQ is being translated to an SVC 3 call.

Are there a special configuration required for this? Does the default SVC_Handler function not handled this case for enabling interrupts? The behavior I am observing is that a call to this function never returns, so I can only guess (since I do not have a debugger) that the uVisor is trapping on this call.

---

As a side note, I tried using UVISOR_PERMISSIVE for my main ACL, but the problem still persists.

Are there any obvious problems with using NVIC_EnableIRQ or writing to the MPU? Are there additional configuration requirements to support either?

If I comment out the offending MPU and NVIC_EnableIRQ lines then everything executes without issue -- except the Ethernet hardware does not work, of course.

[sherrellbc]

I hooked SVC_Handler and confirmed that this function is not returning when called from the NVIC_EnableIRQ proxy. What might be keeping uVisor from returning, at the very least, an error in this case?

[sherrellbc]

I was think I was able to solve the problem related to NVIC_EnableIRQ, but I am still unable to read the MPU.

A simple program that only reads from the MPU and has the MPU ACL does not work.

[meriac]

@sherrellbc : Please don't access the MPU from unprivileged code. We need to figure why lwip insists on disabling the MPU - which is clearly a dirty hack that needs to be solved differently.

[sherrellbc]

@meriac,
@geky,

I agree. Is there a way to read or write to this memory without a uVisor fault or is it strictly disallowed in unprivileged mode?

I do not have access to a debugger at the moment, but when I try to even read MPU (or FB) registers I get some sort of error and trap into the uVisor, or so I can only assume. Execute will halt on the read instruction. This behavior is observed regardless of whether the MPU ACL is included.

I observed that LWIP will break if the MPU is not disabled. That is, if the line is commented out and uVisor is disabled LWIP will not work. It all works as expected, of course, if LWIP is allowed to disable the MPU.

[meriac]

@sherrellbc writing to the MPU is out of question. We can't allow that for security reasons while uVisor is enabled. We need to find out why lwIP believes disabling the MPU is a good idea and fix the underlying problem without touching the MPU configuration from user mode.

[sherrellbc]

Perhaps it is related to the necessary permissions for the ENET hardware.

https://github.com/ARMmbed/uvisor/blob/master/core/system/inc/mpu/vmpu_freescale_k64_mem.h#L30

As I am sure you have seen already, I submitted an issue over at mbed since they own the LWIP implementation.

[geky]

It looks like on reset, the K64F MPU defaults to a single, full-access memory-region covering the entire address space.

That is, except for bus master 3, which is dedicated to the ENET hardware. Whoever implemented the original lwip port must have just disabled the MPU as a quick solution, since as far as I'm aware is has no impact on vanilla mbed.

Granting access to bus master 3 seems to work as an alternative to disabling the MPU completely:

MPU->RGDAAC[0] |= 0x007c0000;

If we don't want lwip to make system wide changes, disabling the MPU could be moved to the k64f's startup sequence. In this case, does disabling the MPU need to be conditionally defined on the presence of uVisor?

I'm unfamiliar with how uVisor handles DMA, but is there a method to grant access to bus master 3? This will be needed for the ethernet to operate.

k64f mpu datasheet:
https://developer.mbed.org/media/uploads/GregC/k64f_rm_rev2.pdf#d204e3a1310_d819e16

[sherrellbc]

This is a fix that I have also been working on with various modes of permissions. I (temporarily) patched the uVisor code to give the above permission and it still does not quite work (with uVisor enabled). It does work, however, when uVisor is disabled.

As far as changing the permission, I think the default background will be modified with the inclusion of ACLs. In particular:

static const UvisorBoxAclItem g_main_box_acls[] = 
{
    {ENET,  sizeof(*ENET),  UVISOR_TACLDEF_PERIPH}
}

If the above ACL is not included then LWIP/Hal will immediately trigger a memory access exception. Still, the below is valid even if the ENET ACL is provided.

---

The observation:
The ENET IRQs are not being triggered unless the MPU is entirely disabled (or if 0x007c0000 permissions are set in RGDAAC[0] and with uVisor disabled .. as pointed out by @geky), so the implementation is still a bit broken. If the below patches are included then LWIP/Hal just times out rather than faulting.

The (a) reason:
The HAL layer does not register for ENET interrupts -- it just tries to enable them (which will cause a uVisor fault). The default ENET interrupt vectors are set properly in startup_MK64F12.S, but uVisor relocates (and presumably clears) the vector table into SRAM. To verify, I checked using

vIRQ_GetVector() on interrupts 83/84 (ENET Transmit/Receieve) and found they were set to 0x00. I have read that it is necessary to register your handlers. As such, I think the fix for that is to first register as so:

Put this before line 457 of fsl_enet.c:
NVIC_SetVector(s_enetRxIrqId[instance], ENET_Receive_IRQHandler);

Put this before line 461 of fsl_enet.c:
NVIC_SetVector(s_enetTxIrqId[instance], ENET_Transmit_IRQHandler);

However, we are still not actually getting any interrupts regardless if the vectors are registered or not.

Comments
I can confirm that without these patches uVisor will halt with an exception since a box is trying to enable interrupts for which it has not first registered. The interrupts will properly fire if uVisor is disable and the MPU is disabled (with or without the above patches, but I think they are necessary when uVisor is enabled).

That said, it seems like the permission problem is related to the ENET hardware being allowed to trigger interrupts. Can this be tied back to MPU permissions?

I was able to falsely trigger an ENET interrupt by using vIRQ_SetPendingIRQ, so they appear to now be at least configured properly.

To Summarize

Working:
1) MPU and uVisor disabled
2) MPU->RGDAAC[0] |= 0x007c0000 with uVisor disabled

Partial:
3) uVisor Enabled, ENET ACL included, Above IRQ patch -> No interrupts generated

I cannot find any other failures in the implementation, but uVisor is either preventing interrupts from being routed to the box that enabled them or is keeping data from going over the wire entirely. I will try to find a managed switch so I can observe network traffic to see if anything is getting out.