Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gamepad: Implement GamepadHapticActuator #32046

Merged
merged 27 commits into from
Jul 20, 2024
Merged

Gamepad: Implement GamepadHapticActuator #32046

merged 27 commits into from
Jul 20, 2024

Conversation

msub2
Copy link
Contributor

@msub2 msub2 commented Apr 11, 2024

This addresses another item on #31362 by implementing GamepadHapticActuator, allow haptic effects to play on connected gamepads. Note that this currently implements the main spec version of GamepadHapticActuator and not what's present in the Gamepad Extensions.


  • ./mach build -d does not report any errors
  • ./mach test-tidy does not report any errors
  • These changes fix #___ (GitHub issue number if applicable)
  • There are tests for these changes OR
  • These changes do not require tests because ___

@msub2
Copy link
Contributor Author

msub2 commented Apr 11, 2024

Also wondering if maybe @gterzian or someone has some insight, currently the subtests that test playEffect and reset in idlharness fail, and I'm not sure why. The specific message is:

FAIL GamepadHapticActuator interface: operation reset() - assert_unreached: Throws "TypeError: "this" object does not implement interface GamepadHapticActuator." instead of rejecting promise Reached unreachable code

Copy link
Member

@gterzian gterzian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the start!

Bunch of small comments, plus a big one for the doing reset. I've also filed w3c/gamepad#200

components/shared/embedder/lib.rs Outdated Show resolved Hide resolved
components/shared/embedder/lib.rs Show resolved Hide resolved
ports/servoshell/Cargo.toml Outdated Show resolved Hide resolved
ports/servoshell/webview.rs Outdated Show resolved Hide resolved
ports/servoshell/webview.rs Outdated Show resolved Hide resolved
components/script/dom/gamepadhapticactuator.rs Outdated Show resolved Hide resolved
components/script/dom/gamepadhapticactuator.rs Outdated Show resolved Hide resolved
self.reset_result_promise.borrow().is_some()
}

pub fn resolve_reset_result_promise(&self) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So to do this as per the spec we will have to restructure this via a listener on the IPC router thread and using trusted refs to the promises and also the actuator:

  • Look at to setup a route over IPC, see for example BroadcastListener, and do this inside Reset.
  • Put the IPC sender in the EmbedderMsg::StopGamepadHapticEffect
  • Put both the playing_effect_promise and the reset_result_promise, and also the actuator itself, as trusted on the "listener" in the route.
  • Add something in the reply about whether the effect was "successfully stopped"
  • From the route, when you receive the reply, queue a task that runs all steps 4.3 and 4.4 of https://www.w3.org/TR/gamepad/#dom-gamepadhapticactuator-reset (the spec is wrong because it does 4.4. outside of a task).
  • To do 4.3.1, I think you need to add a "playing effect generation counter", so that you can compare the one you stored on the IPC listener with the current one on the actuator by the time the task runs. Note this is necassary because it plays with the step 4 of playEffect.

components/script/dom/gamepadhapticactuator.rs Outdated Show resolved Hide resolved
ports/servoshell/webview.rs Outdated Show resolved Hide resolved
@mrobinson
Copy link
Member

@msub2 Do you plan to continue working on this PR?

@msub2
Copy link
Contributor Author

msub2 commented Jun 14, 2024

Yes, there's a spec PR I've been waiting to merge for this before continuing but I do intend to finish it

@msub2 msub2 force-pushed the gamepad-haptics branch 3 times, most recently from 38cc0cb to c42e092 Compare July 4, 2024 00:56
@msub2 msub2 force-pushed the gamepad-haptics branch from 91b2948 to 0da7f8e Compare July 14, 2024 08:10
msub2 added 20 commits July 13, 2024 22:10
Signed-off-by: Daniel Adams <[email protected]>
Signed-off-by: Daniel Adams <[email protected]>
Signed-off-by: Daniel Adams <[email protected]>
Signed-off-by: Daniel Adams <[email protected]>
Signed-off-by: Daniel Adams <[email protected]>
Signed-off-by: Daniel Adams <[email protected]>
@msub2 msub2 requested a review from gterzian July 16, 2024 22:01
@gterzian gterzian added the T-linux-wpt-2020 Do a try run of the WPT label Jul 17, 2024
@github-actions github-actions bot removed the T-linux-wpt-2020 Do a try run of the WPT label Jul 17, 2024
Copy link

🔨 Triggering try run (#9973701020) for Linux WPT

Copy link
Member

@gterzian gterzian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lookst mostly good to me, couple of small comments and let's also look further into that test...

components/script/dom/document.rs Outdated Show resolved Hide resolved
@@ -240,7 +245,10 @@ where
gamepad_event = Some(GamepadEvent::Disconnected(index));
},
EventType::ForceFeedbackEffectCompleted => {
gamepad_event = Some(GamepadEvent::HapticEffectCompleted(index));
self.haptic_effects[&event.id.into()]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we not check for the presence of the effect? Is it possible for stop_haptic_effect to be called after EventType::ForceFeedbackEffectCompleted has already been enqueued but not handled yet(in which case the effect would have been removed already). Not sure what guarantees the stop calls give....

Copy link

Test results for linux-wpt-layout-2020 from try job (#9973701020):

Flaky unexpected result (15)
  • FAIL [expected PASS] /_mozilla/css/dirty_viewport.html (#13731)
  • CRASH [expected PASS] /_webgl/conformance/glsl/bugs/temp-expressions-should-not-crash.html (#22050)
  • TIMEOUT [expected OK] /_webgl/conformance/glsl/misc/shader-uniform-packing-restrictions.html (#28103)
  • TIMEOUT [expected OK] /_webgl/conformance/uniforms/out-of-bounds-uniform-array-access.html (#26225)
    • NOTRUN [expected PASS] subtest: Overall test
  • TIMEOUT /css/cssom-view/MediaQueryList-extends-EventTarget.html (#25269)
    • TIMEOUT [expected PASS] subtest: onchange removes listener

      Test timed out
      

    • NOTRUN [expected TIMEOUT] subtest: listeners for "change" type are called
  • OK /html/browsers/browsing-the-web/navigating-across-documents/javascript-url-return-value-handling-dynamic.html (#28066)
    • FAIL [expected PASS] subtest: 0041 set in href="" targeting a frame and clicked

      assert_equals: expected "A" but got ""
      

    • FAIL [expected PASS] subtest: 0080 00FF set in href="" targeting a frame and clicked

      assert_equals: expected "�ÿ" but got ""
      

    • FAIL [expected PASS] subtest: 0080 00FF 0100 set in href="" targeting a frame and clicked

      assert_equals: expected "�ÿĀ" but got ""
      

    • FAIL [expected PASS] subtest: D83D DE0D set in href="" targeting a frame and clicked

      assert_equals: expected "😍" but got ""
      

    • FAIL [expected PASS] subtest: DE0D 0041 set in href="" targeting a frame and clicked

      assert_equals: expected "\ufffdA" but got ""
      

  • OK /html/browsers/browsing-the-web/navigating-across-documents/replace-before-load/form-submit-cross-frame.html
    • FAIL [expected PASS] subtest: No replace before load, triggered by cross-iframe formElement.submit()

      assert_equals: frame's after-submit history.length expected 2 but got 1
      

  • OK /html/browsers/history/the-history-interface/traverse_the_history_5.html (#21383)
    • PASS [expected FAIL] subtest: Multiple history traversals, last would be aborted
  • OK [expected TIMEOUT] /html/interaction/focus/the-autofocus-attribute/supported-elements.html (#24145)
    • FAIL [expected TIMEOUT] subtest: Element with tabindex should support autofocus

      assert_equals: expected "SPAN" but got "BODY"
      

    • PASS [expected NOTRUN] subtest: Non-HTMLElement should not support autofocus
    • FAIL [expected NOTRUN] subtest: Host element with delegatesFocus should support autofocus

      promise_test: Unhandled rejection with value: object "TypeError: host.attachShadow is not a function"
      

    • FAIL [expected NOTRUN] subtest: Host element with delegatesFocus including no focusable descendants should be skipped

      promise_test: Unhandled rejection with value: object "TypeError: host.attachShadow is not a function"
      

    • FAIL [expected NOTRUN] subtest: Area element should support autofocus

      promise_test: Unhandled rejection with value: object "TypeError: w.document.querySelector(...) is null"
      

  • OK /html/semantics/embedded-content/the-iframe-element/iframe-loading-lazy-nav-form-submit.html (#32607)
    • FAIL [expected PASS] subtest: Navigating iframe loading='lazy' before it is loaded: form submit

      uncaught exception: Error: assert_equals: expected "http://web-platform.test:8000/html/semantics/embedded-content/the-iframe-element/support/blank.htm?" but got "http://web-platform.test:8000/html/semantics/embedded-content/the-iframe-element/support/blank.htm?src"
      

  • TIMEOUT [expected CRASH] /html/semantics/embedded-content/the-iframe-element/iframe_sandbox_popups_escaping-1.html (#22647)
  • OK /html/semantics/forms/form-submission-0/text-plain.window.html (#28687)
    • PASS [expected FAIL] subtest: text/plain: 0x00 in value (formdata event)
    • PASS [expected FAIL] subtest: text/plain: backslash in name (normal form)
  • CRASH [expected OK] /html/semantics/forms/the-fieldset-element/disabled-003.html (#31730)
  • TIMEOUT [expected OK] /webstorage/localstorage-about-blank-3P-iframe-opens-3P-window.partitioned.tentative.html (#29053)
    • TIMEOUT [expected PASS] subtest: StorageKey: test 3P about:blank window opened from a 3P iframe

      Test timed out
      

  • OK /workers/WorkerGlobalScope-close.html (#23064)
    • PASS [expected FAIL] subtest: Test sending a message after closing.
Stable unexpected results that are known to be intermittent (22)
  • FAIL [expected PASS] /_mozilla/css/iframe/hide_and_show.html (#15265)
  • PASS [expected TIMEOUT] /css/css-color/animation/opacity-animation-ending-correctly-002.html (#29216)
  • PASS [expected FAIL] /css/css-transforms/transform-input-018.html (#22725)
  • PASS [expected FAIL] /css/css-values/vh_not_refreshing_on_chrome.html (#23385, #15570)
  • TIMEOUT /css/cssom-view/MediaQueryList-addListener-removeListener.html (#24569)
    • TIMEOUT [expected PASS] subtest: listeners are called correct number of times

      Test timed out
      

    • NOTRUN [expected PASS] subtest: listeners are called in order they were added
  • OK [expected TIMEOUT] /css/cssom-view/MediaQueryList-extends-EventTarget-interop.html (#25285)
    • PASS [expected TIMEOUT] subtest: listener added with addListener and addEventListener (capture) is called twice
    • PASS [expected NOTRUN] subtest: removeEventListener removes listener added with addListener
    • FAIL [expected NOTRUN] subtest: removeEventListener (capture) doesn't remove listener added with addListener

      assert_equals: triggerMQLEvent expected 1 but got 0
      

    • PASS [expected NOTRUN] subtest: removeListener removes listener added with addEventListener
    • FAIL [expected NOTRUN] subtest: removeListener doesn't remove listener added with addEventListener (capture)

      assert_equals: triggerMQLEvent expected 1 but got 0
      

    • FAIL [expected NOTRUN] subtest: capturing event listener fires before non-capturing listener at target

      assert_array_equals: triggerMQLEvent lengths differ, expected array ["addEventListener", "addListener"] length 2, got [] length 0
      

  • OK [expected TIMEOUT] /css/cssom-view/MediaQueryListEvent.html (#25275)
    • PASS [expected NOTRUN] subtest: constructor of "change" event
  • OK /custom-elements/form-associated/ElementInternals-setFormValue.html (#29174)
    • PASS [expected FAIL] subtest: Newline normalization - \r\n in FormData name (urlencoded)
  • OK /fetch/metadata/generated/css-font-face.https.sub.tentative.html (#32732)
    • PASS [expected FAIL] subtest: sec-fetch-user
  • TIMEOUT /fetch/metadata/generated/css-images.sub.tentative.html (#29047)
    • TIMEOUT [expected PASS] subtest: background-image sec-fetch-site - HTTPS downgrade (header not sent)

      Test timed out
      

  • TIMEOUT [expected OK] /html/browsers/browsing-the-web/navigating-across-documents/initial-empty-document/iframe-src-aboutblank-navigate-immediately.html (#29048)
    • TIMEOUT [expected FAIL] subtest: Navigating to a different document with link click

      Test timed out
      

    • NOTRUN [expected FAIL] subtest: Navigating to a different document with form submission
  • OK /html/browsers/browsing-the-web/navigating-across-documents/replace-before-load/a-click.html (#28697)
    • PASS [expected FAIL] subtest: aElement.click() before the load event must NOT replace
  • TIMEOUT [expected ERROR] /html/canvas/element/manual/imagebitmap/createImageBitmap-drawImage.html (#32745)
  • TIMEOUT [expected OK] /html/interaction/focus/the-autofocus-attribute/autofocus-dialog.html (#29087)
    • TIMEOUT [expected FAIL] subtest: &lt;dialog&gt;-contained autofocus element gets focused when the dialog is shown

      Test timed out
      

  • TIMEOUT [expected OK] /html/semantics/embedded-content/media-elements/track/track-element/no-cuechange-before-play.html (#31014)
    • TIMEOUT [expected FAIL] subtest: Ensure that the 'cuechange' event is not fired before video playback has begun.

      Test timed out
      

  • TIMEOUT /html/semantics/embedded-content/the-iframe-element/iframe_sandbox_popups_nonescaping-1.html (#24066)
    • NOTRUN [expected FAIL] subtest: Check that popups from a sandboxed iframe do not escape the sandbox
  • CRASH [expected TIMEOUT] /html/semantics/embedded-content/the-iframe-element/iframe_sandbox_popups_nonescaping-2.html (#22154)
  • TIMEOUT [expected OK] /html/semantics/embedded-content/the-iframe-element/iframe_sandbox_popups_nonescaping-3.html (#24066)
  • OK /html/semantics/forms/form-submission-0/multipart-formdata.window.html (#28725)
    • PASS [expected FAIL] subtest: multipart/form-data: \r\n in value (formdata event)
  • OK /html/semantics/links/links-created-by-a-and-area-elements/htmlanchorelement_noopener.html (#23205)
    • PASS [expected FAIL] subtest: Check that rel=noopener with target=_self does a normal load
  • TIMEOUT /resource-timing/test_resource_timing.https.html (#25216)
    • PASS [expected FAIL] subtest: PerformanceEntry has correct name, initiatorType, startTime, and duration (img)
  • OK [expected ERROR] /workers/constructors/Worker/Worker-constructor.html (#22991)

Copy link

✨ Try run (#9973701020) succeeded.

@msub2 msub2 force-pushed the gamepad-haptics branch from a97bf9e to d70e784 Compare July 18, 2024 05:24
@msub2 msub2 requested a review from gterzian July 18, 2024 07:58
Copy link
Member

@gterzian gterzian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last couple of things.

components/script/dom/gamepadhapticactuator.rs Outdated Show resolved Hide resolved
components/script/dom/gamepadhapticactuator.rs Outdated Show resolved Hide resolved
components/script/dom/document.rs Outdated Show resolved Hide resolved

[GamepadHapticActuator interface: attribute effects]
expected: FAIL

[GamepadHapticActuator interface: operation playEffect(GamepadHapticEffectType, optional GamepadEffectParameters)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can fix these as well, as per my comments in zulip re setting up the promise.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, we can't.

Copy link
Member

@gterzian gterzian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of more stuff...

@msub2
Copy link
Contributor Author

msub2 commented Jul 19, 2024

I managed to address everything except checking for document visibility on playEffect and reset. It seems like the document visibility is hidden despite the page having loaded with visible content (I have a simple test page that gives me a readout of the gamepad state), so both promises are unexpectedly rejected. Should we try to track that down now or save it for a followup?

@msub2 msub2 requested a review from gterzian July 19, 2024 02:49
@msub2 msub2 force-pushed the gamepad-haptics branch from 91ce2e7 to e9c8cf3 Compare July 19, 2024 02:50
@gterzian
Copy link
Member

gterzian commented Jul 19, 2024

Should we try to track that down now or save it for a followup?

Let's note a follow-up in an issue.

We can also open a separate issue about the weird IDL failure.

Copy link
Member

@gterzian gterzian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok thanks for the great work! LGTM, let's open issues for the two remaining problems...

@gterzian gterzian added this pull request to the merge queue Jul 19, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jul 19, 2024
@jdm jdm added this pull request to the merge queue Jul 20, 2024
Merged via the queue into servo:main with commit 2c17de7 Jul 20, 2024
11 checks passed
Gae24 pushed a commit to Gae24/servo that referenced this pull request Jul 26, 2024
* Implement Servo side of GamepadHapticActuator

Signed-off-by: Daniel Adams <[email protected]>

* Get build working

Signed-off-by: Daniel Adams <[email protected]>

* Create effect handling on embedder side

Signed-off-by: Daniel Adams <[email protected]>

* Update tracing for GamepadHapticEffect

Signed-off-by: Daniel Adams <[email protected]>

* Update gilrs to point to commit with effect complete event

Signed-off-by: Daniel Adams <[email protected]>

* Implement playing and preempting haptic effects

Signed-off-by: Daniel Adams <[email protected]>

* Update IDL to add trigger rumble

Signed-off-by: Daniel Adams <[email protected]>

* Update WPT expectations

Signed-off-by: Daniel Adams <[email protected]>

* Handle stopping haptic effects from reset()

Signed-off-by: Daniel Adams <[email protected]>

* ./mach fmt, fix test-tidy issues

Signed-off-by: Daniel Adams <[email protected]>

* Add extra validity checks for trigger rumble

Signed-off-by: Daniel Adams <[email protected]>

* Retrieve supported haptic effects from embedder

Signed-off-by: Daniel Adams <[email protected]>

* Fix test expectations

Signed-off-by: Daniel Adams <[email protected]>

* Add missing spec link, pin gilrs commit

Signed-off-by: Daniel Adams <[email protected]>

* servoshell cargo formatting

Signed-off-by: Daniel Adams <[email protected]>

* Fix Cargo.toml

Signed-off-by: Daniel Adams <[email protected]>

* Additional comments, realm proof, naming

Signed-off-by: Daniel Adams <[email protected]>

* ./mach fmt

Signed-off-by: Daniel Adams <[email protected]>

* Update gilrs rev to gilrs-core 0.5.12 release

Signed-off-by: Daniel Adams <[email protected]>

* Implement sequence ids for gamepad haptic promises

Signed-off-by: Daniel Adams <[email protected]>

* Take playing effect promise instead of cloning

Signed-off-by: Daniel Adams <[email protected]>

* Implement listener for reset function

Signed-off-by: Daniel Adams <[email protected]>

* Fix Cargo.lock

Signed-off-by: Daniel Adams <[email protected]>

* Restructure IPC listeners, add comments, handle visibility change

Signed-off-by: Daniel Adams <[email protected]>

* Check that haptic effect still exists before handling ff completion event

Signed-off-by: Daniel Adams <[email protected]>

* Visibility steps, add InRealm bindings for promises

Signed-off-by: Daniel Adams <[email protected]>

* Add Gamepad EmbedderMsg arms to egl servo_glue

Signed-off-by: Daniel Adams <[email protected]>

---------

Signed-off-by: Daniel Adams <[email protected]>
@msub2 msub2 deleted the gamepad-haptics branch August 6, 2024 01:47
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f226b8f4d9bc7da93de8efd8747c6b1086409ca3f4b6d51e9a7f5461a9183fe"
version = "0.10.6"
source = "git+https://gitlab.com/gilrs-project/gilrs?rev=eafb7f2ef488874188c5d75adce9aef486be9d4e#eafb7f2ef488874188c5d75adce9aef486be9d4e"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@msub2 Is this specific Git version necessary? Would it be fine to update gilrs to a newer published release?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This specific revision is necessary because the currently published version on crates.io does not contain the changes I made to support events for force feedback effect completion. I've been meaning to ask the maintainer what's going on with that, as the last time I talked with them it sounded like it would've landed 2 or 3 releases ago

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just left a message on the original gilrs PR, will see what they say

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants