Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[YouTube] Make some improvements and fixes #1127

Merged
merged 22 commits into from
Dec 9, 2023

Commits on Dec 8, 2023

  1. Configuration menu
    Copy the full SHA
    8a9ebcc View commit details
    Browse the repository at this point in the history
  2. [YouTube] Improve payloads and URLs of InnerTube requests

    For every InnerTube request:
    - Always add a `request` object with the following properties:
      - "internalExperimentFlags" set to an empty array;
      - "useSsl" set to "true";
      - "lockedSafetyMode" set to "false".
    - Use proper TODO comment to provide a way to enable restricted mode on every
    request and add it on requests on which it wasn't present.
    
    For YouTube Music:
    - Remove alt query parameter, as it is not used anymore by the website;
    - Add prettyPrint query parameter with false value on YouTube Music search
    continuations.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    d97c9e0 View commit details
    Browse the repository at this point in the history
  3. [YouTube] Add utcOffsetMinutes to all InnerTube payloads

    This should make returned dates consistent between timezones and countries on
    which the extractor is ran.
    
    It was previously only set on YouTube Music search continuations.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    2c94179 View commit details
    Browse the repository at this point in the history
  4. [YouTube] Add missing prettyPrint query parameter to some test InnerT…

    …ube requests
    
    This query parameter for which its value is set to false was not added to two
    requests made in test classes of YoutubeMixPlaylistExtractorTest.
    
    Also remove an unneeded ParsingException exception throwing declaration in a
    test method.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    ec838d7 View commit details
    Browse the repository at this point in the history
  5. [YouTube] Switch to new consent cookie

    Also move the documentation of the consent in its setter method in order to be
    accessible publicly and improve it.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    ff8ed72 View commit details
    Browse the repository at this point in the history
  6. [YouTube] Extract only search results corresponding to a search type

    YouTube returns sometimes videos inside channel search results. As we only want
    results corresponding to the type we requested, this commits makes
    YoutubeSearchExtractor ignoring non-requested search results we get, using the
    extractor LinkHandler's first content filter value.
    
    Also remove an unneeded exception throwing declaration in
    YoutubeSearchExtractor.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    b71ce11 View commit details
    Browse the repository at this point in the history
  7. [YouTube] Support new A/B tested like data and avoid like count conve…

    …rsion from integer to long
    
    Also make minor improvements to current like data extraction and remove
    previous like count data support, as it is not returned anymore.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    3782d9a View commit details
    Browse the repository at this point in the history
  8. [YouTube] Remove Channels channel tab support

    This tab has been removed by YouTube.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    7dea2d0 View commit details
    Browse the repository at this point in the history
  9. [YouTube] Bypass crisis resources blocking search results

    These crisis resources are preventing search results to be returned. See
    https://support.google.com/youtube/answer/10726080?hl=en for more info on them.
    
    This commit changes search parameters to include the property allowing to show
    search results.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    6ba8251 View commit details
    Browse the repository at this point in the history
  10. [YouTube] Fix expected data in YoutubeStreamExtractorRelatedMixTest

    Video's title and tags have been changed by its uploader, so they have to be
    updated.
    
    Also make some package-private constants private, as they are not used outside
    of the class, and remove unneeded test overrides.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    0bcb241 View commit details
    Browse the repository at this point in the history
  11. [YouTube] Fix YoutubeChannelExtractorTest tests

    - Change CarouselHeader test channel to Sports system one, as the Coachella one
    doesn't return this channel header anymore;
    - Fix InteractiveTabbedHeader test by checking whether the test's channel
    description is not empty instead of containing some words, as it is changing
    frequently.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    fc45941 View commit details
    Browse the repository at this point in the history
  12. [YouTube] Fix YoutubeStreamExtractorDefaultTest.RatingsDisabledTest

    Replace the video used in this test class with another one publicly available
    and update the corresponding expected test values.
    
    The test class's mocks will be updated in a different commit.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    e111814 View commit details
    Browse the repository at this point in the history
  13. [YouTube] Fix testRelatedItems method of YoutubeStreamExtractorRelate…

    …dMixTest
    
    This commit fixes the testRelatedItems test method by:
    - accepting consent in the test class, in order to extract mixes in
    recommendations;
    - removing assertion of a music mix inside the recommendations, as YouTube
    doesn't seem to return such mixes anymore, at least for the video used in the
    test class.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    ad3d187 View commit details
    Browse the repository at this point in the history
  14. [YouTube] Fix expected upload dates of StreamExtractor tests

    YouTube is rolling out or A/B testing a new date format returned inside player
    responses, which are precise to the second instead of the day.
    
    This commit makes the StreamExtractor tests use these more precise dates.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    9dc1eab View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    56ab354 View commit details
    Browse the repository at this point in the history
  16. [YouTube] Fix YoutubeSearchExtractorTest.Suggestion

    YouTube doesn't return anymore a suggestion for the query "algorythm", but does
    for the query "on board ing" ("on boarding"). This search query is now used and
    had to be URL-encoded.
    
    URL encoding in the complete YoutubeSearchExtractorTest test class uses now
    extractor's Util class instead of Java's URLDecoder class directly.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    29dc762 View commit details
    Browse the repository at this point in the history
  17. [YouTube] Enable and change testRelatedItems method of YoutubePlaylis…

    …tExtractorTest.LearningPlaylist
    
    This change makes the method test whether there is no more items instead of
    being disabled.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    5fa22ae View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    bedc9e5 View commit details
    Browse the repository at this point in the history
  19. [YouTube] Fix testSubscriberCount method of YoutubeChannelExtractorTe…

    …st.CarouselHeader
    
    The subscriber count is now lower than the expected count as some people
    unsubscribed to the Sports system channel. The expected count has been so
    lowered.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    642bb01 View commit details
    Browse the repository at this point in the history
  20. [YouTube] Fix YoutubeSearchQHTest tests

    As search parameters to bypass crisis resources blocking search results have
    been implemented, they need to be added to search tests, in order to pass
    them.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    06838d7 View commit details
    Browse the repository at this point in the history
  21. [YouTube] Add a blocking crisis resources bypass in search results test

    This test only tests that search results are returned, when no content filters
    are provided and crisis resources blocking search results should be returned.
    
    Searches with blocking crisis resources and content filters should work too, as
    the bypass has been implemented for them.
    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    00a0f1a View commit details
    Browse the repository at this point in the history
  22. [YouTube] Update mocks

    AudricV committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    ec0194c View commit details
    Browse the repository at this point in the history