CARVIEW |
Navigation Menu
-
Notifications
You must be signed in to change notification settings - Fork 20
Add XRInteractionSpace #50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
// intersect the target ray with the UI elements in world space | ||
// to determine which UI element, if any, was selected | ||
} else { | ||
// determine the screen-space coordinates of the select event and |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Annoyingly there's no easy way to do this without doing some math on the projection matrix. Perhaps select events for "screen"
target ray modes should also come with (x, y)
positions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder: Are standard pointer events still being processed during an AR session? If so developers could choose to listen for those instead in a "screen-space" interaction space. Not sure if that's the pattern we want to promote, but there's no way I can see to completely normalize these input modes anyway so if the logic needs to branch it may be better to have it switch to a more robust and proven input mechanism than for us to try and offer a partial substitute.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel like they won't be? cc @klausw
But if they are, then yes, that makes sense. Perhaps we should spec more about how handheld AR should work vis a vis inputs in this spec, then.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the current ARCore-based implementation in Chrome, there are no DOM input events while a regular immersive-ar session is active.
Enabling DOM Overlay mode adds DOM events for the selected overlay element, including pointer events, along with the beforexrselect
logic to make it easier for applications to deduplicate DOM events and XR input events.
FYI, in current Chrome Canary, the transient WebXR input source for screen touch already reports x/y axis values in its gamepad attribute. These are normalized -1..1 range values, not DOM coordinates.
Example:
> session.inputSources[0].gamepad
Gamepad
id: ""
index: -1
connected: true
timestamp: 15682728.100000007
mapping: ""
axes: (2) [-0.3270833492279053, 0.5138888359069824]
buttons: Array(3)
0: GamepadButton {pressed: false, touched: false, value: 0}
1: GamepadButton {pressed: false, touched: false, value: 0}
2: GamepadButton {pressed: false, touched: true, value: 1}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm. It might be important to expose the size of the screen, then? This got discussed a bit in #9: it may be useful to expose the offsets of the screen rect from the viewer space or something.
I guess one question is: given that it's handheld AR, do we expect people to draw things in viewer space with an epsilon offset from the screen, or do we expect people to draw things directly to the output buffer? In the former case we need to know the near rect of the screen to know where the "edges" are so we can draw within them, in the latter case we need to know the near rect and framebuffer size (which you have already) so that we can map the input events to screen-space coordinates. Either way you need to know the near rect.
It can potentially be exposed as an offset of the viewer space, plus a width and height. Then you need to intersect the target ray with the resultant yz plane and see where on the screen it falls.
I think something like this is necessary, I'm not sure if this is the best option. It seemed to be the best after all the discussion on #9, though perhaps we may want a companion enum on the DOM overlay module that mentions if the overlay is screen-space or emulated world-space. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems like a reasonable approach to the problem, so I support it. I'd like to hear from more WG members regarding this pattern before we commit to it, though.
// intersect the target ray with the UI elements in world space | ||
// to determine which UI element, if any, was selected | ||
} else { | ||
// determine the screen-space coordinates of the select event and |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder: Are standard pointer events still being processed during an AR session? If so developers could choose to listen for those instead in a "screen-space" interaction space. Not sure if that's the pattern we want to promote, but there's no way I can see to completely normalize these input modes anyway so if the logic needs to branch it may be better to have it switch to a more robust and proven input mechanism than for us to try and offer a partial substitute.
9c18894
to
7edb211
Compare
Now that we have DOM Layers, is this feature needed? |
Yes , both can have such a UI, but there's a clear preference. The feature is still needed: DOM overlays are specifically for DOM UI only. Furthermore, the plan for headsets is to have the UA to pick some rect to place it in, which does not help if for headsets you want to design your own 3D UI. Also, not everyone may implement DOM overlays, and this is a relatively lightweight boolean that enables frameworks and apps to know how best to do their own UIs. A possible outcome of this is that in screen-space situations people will just use DOM overlay, whereas in world-space situations they may build their own. |
What I think you're trying to achieve is that the author can distinguish between immersive and handheld AR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this looks good overall.
// intersect the target ray with the UI elements in world space | ||
// to determine which UI element, if any, was selected | ||
} else { | ||
// determine the screen-space coordinates of the select event and |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the current ARCore-based implementation in Chrome, there are no DOM input events while a regular immersive-ar session is active.
Enabling DOM Overlay mode adds DOM events for the selected overlay element, including pointer events, along with the beforexrselect
logic to make it easier for applications to deduplicate DOM events and XR input events.
FYI, in current Chrome Canary, the transient WebXR input source for screen touch already reports x/y axis values in its gamepad attribute. These are normalized -1..1 range values, not DOM coordinates.
Example:
> session.inputSources[0].gamepad
Gamepad
id: ""
index: -1
connected: true
timestamp: 15682728.100000007
mapping: ""
axes: (2) [-0.3270833492279053, 0.5138888359069824]
buttons: Array(3)
0: GamepadButton {pressed: false, touched: false, value: 0}
1: GamepadButton {pressed: false, touched: false, value: 0}
2: GamepadButton {pressed: false, touched: true, value: 1}
XRInteractionSpace {#xrinteractionspace-enum} | ||
---------------------- | ||
|
||
Sometimes the application will wish to draw UI that the user may interact with. WebXR allows for a variety of form factors, including both handheld phone AR and head-worn AR. For different form factors, the UIs will belong in different spaces to facilitate smooth interaction, for example the UI for handheld phone AR will likely be drawn directly on the screen without projection, but the UI for headworn AR will likely be drawn a small distance from the head so that users may use their controllers to interact with it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think there's some ambiguity about what it means to draw directly on the screen. For example, a stereoscopic display may have separate left/right eye views, so applications would still need to respect the viewport split when drawing to the screen, but for projection purposes it would act as screen-space.
Maybe explicitly highlight that applications still need to respect the views array and viewports for rendering?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will do!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Though, for a stereoscopic handheld display I suspect you'd actually want to draw directly to the canvas anyway? You don't want to have multiple versions of the same buttons.
(This is precisely the kind of distinction this API lets us make, that an API based on exposing form factor directly does not)
Yes. This has already been discussed a couple times (conclusions from last meeting it got discussed) The UI hint is intended to be a safer abstraction. We don't necessarily want to directly expose the UI form factor (especially since new form factors may crop up), but instead we want to help answer the question of "where should I draw my UI?". In general it seems to be a pattern that if a piece of information is only going to be used for a single purpose, we should expose it tailored for that purpose instead. In practice, this enum is going to be roughly equivalent to a headworn/handheld enum. |
/agenda to get wider input and maybe make a decision |
Note to self fix the bot |
(discussed in the call, nobody seems allergic to this. we'll make the space->mode change though) |
Can you reiterate what cases you see where "screen-space" is not a handheld device and "world-space" is not a headset? |
@cabanier alternate devices, for example a CAVE system may have UI on the surface. Or even some form of transparent screen device that's not necessarily handheld. |
Does WebXR work on CAVE systems? |
Yeah, basically. It's not necessarily "handheld". |
An Such a UA could support the AR module without allowing the creation of |
Merging since we believe that all outstanding feedback was addressed on the last call. |
@@ -87,7 +120,13 @@ enum XREnvironmentBlendMode { | |||
"alpha-blend", | |||
}; | |||
|
|||
enum XRInteractionSpace { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Manishearth: It looks like the enum itself did not get renamed here.
Fixes #9
cc @klausw @bialpio @cabanier
r? @toji