Input events dispatch to top-level frame

[sadym-chromium]

As discussed in w3c/webdriver-bidi#795, the actions should be dispatched from the top-level browsing context.

---

Preview | Diff

[sadym-chromium]

@whimboo WDYT?

[OrKoN]

The following two things seem to be still missing:

1. steps to convert the coordinates to the top-level browsing context coordinate space if they are frame scoped. Edit: unless this is covered by the implementation-defined bits?
2. apply the same fix to other actions (I assume we do actually want to dispatch keyboard events through the top-level frame as well)

[whimboo]

The following two things seem to be still missing:
1. steps to convert the coordinates to the top-level browsing context coordinate space if they are frame scoped. Edit: unless this is covered by the implementation-defined bits?

Yes, absolutely. By changing the way where we dispatch the events the coordinates should be exactly the same. Not taking care of offsets would cause quite a lot of regressions for those users who make use of actions a lot.

Therefore see:

web-platform-tests/wpt#48147
web-platform-tests/wpt#48123

Given that Chrome seems to already dispatch actions in the top-level browsing context the referenced tests are failing.

2. apply the same fix to other actions (I assume we do actually want to dispatch keyboard events through the top-level frame as well)

Yes, we would have to do it for all the pointer and wheel input sources.

Keyboard actions are more challenging because they could allow users to trigger shortcuts that access restricted browser features. This would require additional checks to determine what actions are permitted and which ones should be blocked. Maybe we should have a follow-up issue for it?

[sadym-chromium]

2. apply the same fix to other actions (I assume we do actually want to dispatch keyboard events through the top-level frame as well)

Aren't all the actions end up in perform implementation-specific action dispatch steps?

[sadym-chromium]

@OrKoN @whimboo I added calculation of the offset relative to the parent browsing context, but IDK how well it will work

[sadym-chromium]

Alternatively we can "hand-wave" hear like "These steps must be equivalent to user trying to perform the given input device manipulations on context through the top-level browsing context."

[sadym-chromium]

#1850

[sadym-chromium]

@OrKoN @whimboo we can keep specification implementation-specific, just clarify the events are dispatched via top-level browsing context. I'm not sure if that approach would work with the events intercepted by the top-level frame.

[whimboo]

@OrKoN @whimboo I added calculation of the offset relative to the parent browsing context, but IDK how well it will work

I think that we should have more wdspec tests (similar to mine for mouse) so we cover at least each input source (including sub sources) and verify implementations against. Did you get Chrome working for my example?

[OrKoN]

@sadym-chromium it looks like each action type dispatches on the context parameter https://w3c.github.io/webdriver/#dfn-dispatch-a-keydown-action

2. apply the same fix to other actions (I assume we do actually want to dispatch keyboard events through the top-level frame as well)

Aren't all the actions end up in perform implementation-specific action dispatch steps?

that's right, overlooked that.

[OrKoN]

Keyboard actions are more challenging because they could allow users to trigger shortcuts that access restricted browser features. This would require additional checks to determine what actions are permitted and which ones should be blocked. Maybe we should have a follow-up issue for it?

so actually I am not sure, even with this change it is not quite defined what it means that the actions are dispatched to the top-level browsing context. For example, in Chrome that is the case but still would not give you access to browser shortcuts.
Perhaps we need a better definition of what the dispatch of an event means, perhaps hook into native event handles in https://www.w3.org/TR/uievents/#handle-native-mouse-move-id?

[whimboo]

so actually I am not sure, even with this change it is not quite defined what it means that the actions are dispatched to the top-level browsing context. For example, in Chrome that is the case but still would not give you access to browser shortcuts.

Yes you are right. I mixed it up with the native event dispatching that I'm working on as well right now. In those cases we would have that particular issue but not when dispatching it in the content process of the top-level browsing context.

[OrKoN]

I think we should keep browsing context indicating the context var type

[sadym-chromium]

Yeah, I missed it.

[OrKoN]

should the offset be applied to all origins or only the viewport? I think the origin pointer might already be in the top level context coordinates and not sure if the element origin does some adjustments already.

[sadym-chromium]

I would expect the origin "pointer" to be relative to "context" param of "PerformActionsParameters" of the command.

[OrKoN]

That does not seem to follow from the spec text since input sources are per top-level browsing context.

[OrKoN]

[=navigable/parent=] is a property of a navigable and not the document. Did you mean to get the current navigable instead of a parent (e.g., https://html.spec.whatwg.org/#node-navigable)?

[jgraham]

I agree that in principle this is the model we need to change to in order to work with modern browser designs.

However I think the change as written is insufficient, and makes the spec contradictory. In each action we still claim we're dispatching events to |context| which, afaict after these changes, is not the parent context but the actual child context. The spec is unfortunately rather badly written here, so the text you've modified is apparently normative, but exists in what is essentially an informative section describing the overall model.

I also suggest (perhaps as a followup) that we make it possible for specific actions to provide the context that is used for calculating origins (which must in all cases be an ancestor of the top-level command context). That makes it much easier to construct action chains that interact with both the top level document and iframes.