Fix stm32f723disco host/cdc_msc_hid HIL: UART RX starvation + DWC2 DMA split-IN NAK storm (#3677)

Fix stm32f723disco host HIL: UART RX starvation + DWC2 split bulk NAK/XactErr handling (#3677)
stm32f7 BSP — UART RX starvation
- The host console USART shared interrupt priority with the USB OTG ISR, so a long
  OTG interrupt could starve RXNE and drop received bytes. Raise the USART RX IRQ
  above OTG_FS/OTG_HS in both the bare-metal and FreeRTOS init paths, guarded by
  #ifdef UART_ID so boards without a UART console keep the default OTG priority.

dwc2 host — split NAK/XactErr handling
- Slave mode: a persistently-NAKing split bulk/control IN poll re-armed the
  start-split immediately, storming the ISR and starving task context. Throttle by
  disabling the channel and re-arming on the resulting halt (no frame deferral).
- Buffer-DMA mode: a pure split bulk-OUT NAK was unhandled, leaving the channel
  halted and stalling the transfer — the dominant cause of CDC echo truncation.
  Handle it by rewinding the buffer pointers and retrying the start-split
  (Programming Guide v4.20a 5.1.4.2).
- Buffer-DMA mode: a split bulk-OUT XactErr was retried immediately, exhausting
  HCD_XFER_ERROR_MAX before the transient cleared. Throttle via channel_disable +
  re-arm to give the hub TT a recovery gap, mirroring slave mode.
- All three are scoped to split transfers (hcsplt.split_en); non-split NAK/XactErr
  keep the core-handled / immediate-retry behavior. The OUT XactErr throttle also
  excludes periodic split, where channel_disable() is a no-op and would wedge the
  channel. The nak_disabled flag is generalized to retry_disabled and honors
  xfer->closing so an endpoint close during a throttled retry tears down cleanly.

Verified on stm32f723disco HIL (slave + CFG_TUH_DWC2_DMA_ENABLE): host/cdc_msc_hid,
msc_file_explorer, and device_info all pass on both variants; DMA CDC echo went
from ~15-25% raw failure to 10/10 clean.

1 files changed, 16 insertions, 5 deletions

diff --git a/hw/bsp/stm32f7/family.c b/hw/bsp/stm32f7/family.c
index 7a322591b..9427ac4a6 100644
--- a/hw/bsp/stm32f7/family.c
+++ b/hw/bsp/stm32f7/family.c
@@ -82,8 +82,9 @@ static UART_HandleTypeDef UartHandle = {.Instance = USARTn,
                                               .OverSampling = UART_OVERSAMPLING_16,
                                         }};
 
-// RX ring buffer via RXNE interrupt — no HAL IT functions used (avoid HAL state conflicts)
-static uint8_t   uart_rx_ff_buf[32];
+// RX ring buffer via RXNE interrupt — no HAL IT functions used (avoid HAL state conflicts).
+// Sized to absorb a full host-forwarding burst (>64B) when the main loop briefly stalls.
+static uint8_t   uart_rx_ff_buf[256];
 static tu_fifo_t uart_rx_ff;
 
 void USARTn_IRQHandler(void) {
@@ -142,13 +143,24 @@ void board_init(void) {
   // 1ms tick timer
   SysTick_Config(SystemCoreClock / 1000);
 
+  // Set UART interrupt higher priority than USB OTG since the F7 USART has no hardware RX FIFO, so a host example's
+  // UART RX must not be starved by the frequent USB host interrupts or incoming bytes overrun (ORE) and dropped.
+  NVIC_SetPriority(OTG_FS_IRQn, 1);
+  NVIC_SetPriority(OTG_HS_IRQn, 1);
+  #ifdef UART_ID
+  NVIC_SetPriority(USARTn_IRQn, 0);
+  #endif
+
 #elif CFG_TUSB_OS == OPT_OS_FREERTOS
   // Explicitly disable systick to prevent its ISR from running before scheduler start
   SysTick->CTRL &= ~1U;
 
   // If freeRTOS is used, IRQ priority is limit by max syscall ( smaller is higher )
-  NVIC_SetPriority(OTG_FS_IRQn, configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY);
-  NVIC_SetPriority(OTG_HS_IRQn, configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY);
+  NVIC_SetPriority(OTG_FS_IRQn, configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY + 1);
+  NVIC_SetPriority(OTG_HS_IRQn, configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY + 1);
+  #ifdef UART_ID
+  NVIC_SetPriority(USARTn_IRQn, configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY);
+  #endif
 #endif
 
 #ifdef UART_ID
@@ -156,7 +168,6 @@ void board_init(void) {
   HAL_UART_Init(&UartHandle);
   tu_fifo_config(&uart_rx_ff, uart_rx_ff_buf, sizeof(uart_rx_ff_buf), false);
   USARTn->CR1 |= USART_CR1_RXNEIE;
-  NVIC_SetPriority(USARTn_IRQn, (1 << __NVIC_PRIO_BITS) - 1);
   NVIC_EnableIRQ(USARTn_IRQn);
 #endif