Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhancement: Feedback when CAPTCHA #50

Open
davidshumway opened this issue Jan 31, 2022 · 1 comment
Open

Enhancement: Feedback when CAPTCHA #50

davidshumway opened this issue Jan 31, 2022 · 1 comment

Comments

@davidshumway
Copy link

davidshumway commented Jan 31, 2022 •
edited
Loading

The search is sometimes rate-limited but PyPaperBot's response is simply "Paper not found...". In this case, PyPaperBot should display the message returned in the HTML response (see below). Optionally, for those working on their own local network, an option could appear to open the URL in a browser to solve the CAPTCHA there.

Response:
HTML status code: 429
HTML response:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head><meta http-equiv="content-type" content="text/html; charset=utf-8"><meta name="viewport" content="initial-scale=1"><title>https://scholar.google.com/scholar?hl=en&amp;q=abc%22&amp;as_vis=1&amp;as_sdt=1,5&amp;start=380</title></head>
<body style="font-family: arial, sans-serif; background-color: #fff; color: #000; padding:20px; font-size:18px;" onload="e=document.getElementById('captcha');if(e){e.focus();} if(solveSimpleChallenge) {solveSimpleChallenge(,);}">
<div style="max-width:400px;">
<hr noshade size="1" style="color:#ccc; background-color:#ccc;"><br>
<form id="captcha-form" action="index" method="post">
<noscript>
<div style="font-size:13px;">
  In order to continue, please enable javascript on your web browser.
</div>
</noscript>
<script src="https://www.google.com/recaptcha/api.js" async defer></script>
<script>var submitCallback = function(response) {document.getElementById('captcha-form').submit();};</script>
<div id="recaptcha" class="g-recaptcha" data-sitekey="6LfwuyUTAAAAAOAmoS0fdqijC2PbbdH4kjq62Y1b" data-callback="submitCallback" data-s="KskQ5aUxKskQnKskQaho-qeu-uodlwNquodlPzUtHOt0SgxuodlK-LDBK8m5HPeJXBMS9x8m5HPeJI0J2v8m5HPeJltxo_1M0kQRb8m5HPeJbfd8pHy0kNPRa2Z_RFJpvQHAs6zrLM1aI5Lca58_waI5Lca51aI5Lca5x3IDmu1ffftae0mAEAvsm4Un_7xFpkcSr7xFpkcSFkD7xFpkcSVwXjYIOOdb_jc"></div>

<input type='hidden' name='q' value='NWU4I8GNWU4I8GIhAxOzGIhAxOZT-uGIhAxOMgFy'><input type="hidden" name="continue" value="https://scholar.google.com/scholar?hl=en&amp;q=abc%22&amp;as_vis=1&amp;as_sdt=1,5&amp;start=380">
</form>
<hr noshade size="1" style="color:#ccc; background-color:#ccc;">

<div style="font-size:13px;">
<b>About this page</b><br><br>

Our systems have detected unusual traffic from your computer network.  This page checks to see if it&#39;s really you sending the requests, and not a robot.  <a href="#" onclick="document.getElementById('infoDiv').style.display='block';">Why did this happen?</a><br><br>

<div id="infoDiv" style="display:none; background-color:#eee; padding:10px; margin:0 0 15px 0; line-height:1.4em;">
This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the <a href="//www.google.com/policies/terms/">Terms of Service</a>. The block will expire shortly after those requests stop.  In the meantime, solving the above CAPTCHA will let you continue to use our services.<br><br>This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests.  If you share your network connection, ask your administrator for help &mdash; a different computer using the same IP address may be responsible.  <a href="//support.google.com/websearch/answer/86640">Learn more</a><br><br>Sometimes you may be asked to solve the CAPTCHA if you are using advanced terms that robots are known to use, or sending requests very quickly.
</div>

IP address: xxx.xxx.xxx.xxx<br>Time: 2022-01-30T12:01:36Z<br>URL: https://scholar.google.com/scholar?hl=en&amp;q=abc&amp;as_vis=1&amp;as_sdt=1,5&amp;start=380<br>
</div>
</div>
</body>
</html>
@ferru97
Copy link
Owner

ferru97 commented Oct 7, 2024

Hi,
sorry for the late reply, but PyPaperBot had been on standby until now.
I've decided to resume development to improve it.

I've created a Telegram channel where you can suggest improvements, report bugs, or request custom data mining scripts. Feel free to join if you're interested with the following LINK

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@davidshumway @ferru97