[Dialog] Support non-nested modal dialogs

[aarongarciah]

Bug report

Current behavior

Rendering several non-nested modal dialogs simultaneously causes issues. See the attached video and demo.

1. When clicking inside a dialog, all opened dialogs close except the one you clicked on. This only happens when using a pointer; it doesn't happen when using the keyboard.
2. Clicking outside a dialog closes them all.

Kapture.2025-01-09.at.15.12.38.mp4

In order to have two or more well-functioning modal dialogs opened at the same time, users need to nest them:

// first dialog
<Dialog.Root>
  <Dialog.Portal>
    <Dialog.Popup>

      // second dialog
      <Dialog.Root>...</Dialog.Root>
    </Dialog.Popup>
</Dialog.Root>

This is very limiting and users will expect to be able to open several dialogs at the same time without nesting them in the component tree:

// first dialog
<Dialog.Root>...</Dialog.Root>

// second dialog
<Dialog.Root>...</Dialog.Root>

Expected behavior

Like other libraries, users should be able to render several dialogs simultaneously without needing to nest them. Radix and other libraries support this: https://codesandbox.io/p/sandbox/radix-dialog-nested-without-nesting-jvvlrn?file=/src/App.tsx&workspaceId=ws_HE9JRzFW36Y2kpBoSwdvZ5

Reproducible example

• Base UI example: https://codesandbox.io/p/sandbox/base-ui-dialog-nested-without-nesting-nwpvnv?workspaceId=ws_HE9JRzFW36Y2kpBoSwdvZ5
• Radix example: https://codesandbox.io/p/sandbox/radix-dialog-nested-without-nesting-jvvlrn?file=/src/App.tsx&workspaceId=ws_HE9JRzFW36Y2kpBoSwdvZ5

Base UI version

v1.0.0-alpha.5

Additional context

Currently, data-nested, data-has-nested-dialogs, and --nested-dialogs are only functional when nesting dialogs in JSX. Having to nest dialogs in JSX is a big constraint for users (e.g., programmatically opened dialogs). While working on this issue, the team could think about allowing users to style dialogs using data-nested, data-has-nested-dialogs, and --nested-dialogs without actually nesting them in JSX.

[oliviertassinari]

Material UI / MUI Base supports this too: https://codesandbox.io/p/sandbox/unruffled-greider-hmdxgx?file=%2Fsrc%2FDemo.tsx%3A16%2C1.

I recall fixing a couple of bugs around this, it's likely going to prevent some teams from migrating. My guess is that people would hit this between 20–49% of real-life projects.

Could the root problem be that we need to bring back more test cases from Material UI: https://github.com/mui/material-ui/blob/60106b31d0b4e5e304fdb4d612cc62c1c21a20f6/packages/mui-material/src/Modal/Modal.test.js#L613 (not sure this reproduces this bug though) and Radix Primitives (didn't find it, but I don't know the codebase enough)?