Annoyingly there's no easy way to do this without doing some math on the projection matrix. Perhaps select events for "screen" target ray modes should also come with (x, y) positions?

I wonder: Are standard pointer events still being processed during an AR session? If so developers could choose to listen for those instead in a "screen-space" interaction space. Not sure if that's the pattern we want to promote, but there's no way I can see to completely normalize these input modes anyway so if the logic needs to branch it may be better to have it switch to a more robust and proven input mechanism than for us to try and offer a partial substitute.

I feel like they won't be? cc @klausw

But if they are, then yes, that makes sense. Perhaps we should spec more about how handheld AR should work vis a vis inputs in this spec, then.

In the current ARCore-based implementation in Chrome, there are no DOM input events while a regular immersive-ar session is active.

Enabling DOM Overlay mode adds DOM events for the selected overlay element, including pointer events, along with the beforexrselect logic to make it easier for applications to deduplicate DOM events and XR input events.

FYI, in current Chrome Canary, the transient WebXR input source for screen touch already reports x/y axis values in its gamepad attribute. These are normalized -1..1 range values, not DOM coordinates.

Example:

> session.inputSources[0].gamepad
Gamepad
id: ""
index: -1
connected: true
timestamp: 15682728.100000007
mapping: ""
axes: (2) [-0.3270833492279053, 0.5138888359069824]
buttons: Array(3)
0: GamepadButton {pressed: false, touched: false, value: 0}
1: GamepadButton {pressed: false, touched: false, value: 0}
2: GamepadButton {pressed: false, touched: true, value: 1}

Hmm. It might be important to expose the size of the screen, then? This got discussed a bit in #9: it may be useful to expose the offsets of the screen rect from the viewer space or something.

I guess one question is: given that it's handheld AR, do we expect people to draw things in viewer space with an epsilon offset from the screen, or do we expect people to draw things directly to the output buffer? In the former case we need to know the near rect of the screen to know where the "edges" are so we can draw within them, in the latter case we need to know the near rect and framebuffer size (which you have already) so that we can map the input events to screen-space coordinates. Either way you need to know the near rect.

It can potentially be exposed as an offset of the viewer space, plus a width and height. Then you need to intersect the target ray with the resultant yz plane and see where on the screen it falls.

I wonder: Are standard pointer events still being processed during an AR session? If so developers could choose to listen for those instead in a "screen-space" interaction space. Not sure if that's the pattern we want to promote, but there's no way I can see to completely normalize these input modes anyway so if the logic needs to branch it may be better to have it switch to a more robust and proven input mechanism than for us to try and offer a partial substitute.

In the current ARCore-based implementation in Chrome, there are no DOM input events while a regular immersive-ar session is active.

Enabling DOM Overlay mode adds DOM events for the selected overlay element, including pointer events, along with the beforexrselect logic to make it easier for applications to deduplicate DOM events and XR input events.

FYI, in current Chrome Canary, the transient WebXR input source for screen touch already reports x/y axis values in its gamepad attribute. These are normalized -1..1 range values, not DOM coordinates.

Example:

> session.inputSources[0].gamepad
Gamepad
id: ""
index: -1
connected: true
timestamp: 15682728.100000007
mapping: ""
axes: (2) [-0.3270833492279053, 0.5138888359069824]
buttons: Array(3)
0: GamepadButton {pressed: false, touched: false, value: 0}
1: GamepadButton {pressed: false, touched: false, value: 0}
2: GamepadButton {pressed: false, touched: true, value: 1}

I think there's some ambiguity about what it means to draw directly on the screen. For example, a stereoscopic display may have separate left/right eye views, so applications would still need to respect the viewport split when drawing to the screen, but for projection purposes it would act as screen-space.

Maybe explicitly highlight that applications still need to respect the views array and viewports for rendering?

Will do!

Though, for a stereoscopic handheld display I suspect you'd actually want to draw directly to the canvas anyway? You don't want to have multiple versions of the same buttons.

(This is precisely the kind of distinction this API lets us make, that an API based on exposing form factor directly does not)

@Manishearth: It looks like the enum itself did not get renamed here.