[YouTube] Make some improvements and fixes #1127
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AudricV
added
bug
This is still EU-only, right? |
2 tasks
AudricV
force-pushed
the
yt_improvements-and-fixes
branch
from
fbcb0be
to
517a649
Compare
|
In addition to the question above, will the users see any noticeable change with this, or is this all under-the-hood stuff? |
This was referenced Nov 27, 2023
AudricV
force-pushed
the
yt_improvements-and-fixes
branch
from
517a649
to
e821b86
Compare
I don't think so, but I am not sure.
Some changes will be seen by users, as #1133 is fixed by this PR. |
Okay. Could you share a test build without the consent cookie? I can perform an action that needs it and see if it still works. |
6 tasks
Stypox
requested changes
Dec 7, 2023
There was a problem hiding this comment.
Thank you! I reviewed the commits one by one and noticed only one suspect change.
I tested on an emulator and everything worked fine: app-debug.zip
As per the crisis resources change, shouldn't the extractor still return the warning as a meta info? E.g. see screenshot below. I will open a PR independent from this one if it's ok for you.
extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
Show resolved
Hide resolved
2 tasks
For every InnerTube request: - Always add a `request` object with the following properties: - "internalExperimentFlags" set to an empty array; - "useSsl" set to "true"; - "lockedSafetyMode" set to "false". - Use proper TODO comment to provide a way to enable restricted mode on every request and add it on requests on which it wasn't present. For YouTube Music: - Remove alt query parameter, as it is not used anymore by the website; - Add prettyPrint query parameter with false value on YouTube Music search continuations.
This should make returned dates consistent between timezones and countries on which the extractor is ran. It was previously only set on YouTube Music search continuations.
…ube requests This query parameter for which its value is set to false was not added to two requests made in test classes of YoutubeMixPlaylistExtractorTest. Also remove an unneeded ParsingException exception throwing declaration in a test method.
Also move the documentation of the consent in its setter method in order to be accessible publicly and improve it.
YouTube returns sometimes videos inside channel search results. As we only want results corresponding to the type we requested, this commits makes YoutubeSearchExtractor ignoring non-requested search results we get, using the extractor LinkHandler's first content filter value. Also remove an unneeded exception throwing declaration in YoutubeSearchExtractor.
…rsion from integer to long Also make minor improvements to current like data extraction and remove previous like count data support, as it is not returned anymore.
This tab has been removed by YouTube.
These crisis resources are preventing search results to be returned. See https://support.google.com/youtube/answer/10726080?hl=en for more info on them. This commit changes search parameters to include the property allowing to show search results.
Video's title and tags have been changed by its uploader, so they have to be updated. Also make some package-private constants private, as they are not used outside of the class, and remove unneeded test overrides.
- Change CarouselHeader test channel to Sports system one, as the Coachella one doesn't return this channel header anymore; - Fix InteractiveTabbedHeader test by checking whether the test's channel description is not empty instead of containing some words, as it is changing frequently.
Replace the video used in this test class with another one publicly available and update the corresponding expected test values. The test class's mocks will be updated in a different commit.
…dMixTest This commit fixes the testRelatedItems test method by: - accepting consent in the test class, in order to extract mixes in recommendations; - removing assertion of a music mix inside the recommendations, as YouTube doesn't seem to return such mixes anymore, at least for the video used in the test class.
YouTube is rolling out or A/B testing a new date format returned inside player responses, which are precise to the second instead of the day. This commit makes the StreamExtractor tests use these more precise dates.
…r.getSearchSuggestion
YouTube doesn't return anymore a suggestion for the query "algorythm", but does
for the query "on board ing" ("on boarding"). This search query is now used and
had to be URL-encoded.
URL encoding in the complete YoutubeSearchExtractorTest test class uses now
extractor's Util class instead of Java's URLDecoder class directly.
…tExtractorTest.LearningPlaylist This change makes the method test whether there is no more items instead of being disabled.
…st.CarouselHeader The subscriber count is now lower than the expected count as some people unsubscribed to the Sports system channel. The expected count has been so lowered.
As search parameters to bypass crisis resources blocking search results have been implemented, they need to be added to search tests, in order to pass them.
This test only tests that search results are returned, when no content filters are provided and crisis resources blocking search results should be returned. Searches with blocking crisis resources and content filters should work too, as the bypass has been implemented for them.
AudricV
force-pushed
the
yt_improvements-and-fixes
branch
from
e821b86
to
ec0194c
Compare
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR does multiple improvements and fixes to the YouTube service in the extractor.
requestobject is set with the following properties:internalExperimentFlagsset to an empty array;useSslset totrue;lockedSafetyModeset tofalse.altquery parameter, as it is not used anymore by the website;prettyPrintquery parameter withfalsevalue on YouTube Music search continuations.utcOffsetMinutesto all InnerTube payloads, previously only set on YouTube Music search continuations, which should make returned dates consistent between timezones and countries on which the extractor is ran;prettyPrintquery parameter to some testInnerTuberequests;Channelschannel tab support, as this tab has been removed by YouTube;For more details, please look at commit messages and code changes.
Fixes #1133, fixes TeamNewPipe/NewPipe#10624.