Open the modal. Then, open the popover inside. Hit escape. The popover should close, not the modal.
Open Modal
Popover menu
Choose one of these
Menu item 1
Menu item 2
Menu item 3