Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

请教一个使用的问题 #9

Open
gaoyong06 opened this issue Mar 22, 2023 · 1 comment
Open

请教一个使用的问题 #9

gaoyong06 opened this issue Mar 22, 2023 · 1 comment

Comments

@gaoyong06
Copy link

我用这个工具,能正常得到一些代理ip,类似:

http://127.0.0.1:8080/get?type=HTTP&count=2&anonymity=all

[
    {
        "Ip": "117.160.250.133",
        "Port": "9999",
        "Country": "中国",
        "Province": "河南",
        "City": "安阳",
        "Isp": "移动",
        "Type": "HTTP",
        "Anonymity": "透明",
        "Time": "2023-03-21 20:25:57",
        "Speed": "85.7072ms",
        "SuccessNum": 5,
        "RequestNum": 5,
        "Source": "scrape代理"
    },
    {
        "Ip": "117.159.15.99",
        "Port": "9091",
        "Country": "中国",
        "Province": "河南",
        "City": "郑州",
        "Isp": "移动",
        "Type": "HTTP",
        "Anonymity": "普匿",
        "Time": "2023-03-21 20:25:57",
        "Speed": "130.7455ms",
        "SuccessNum": 5,
        "RequestNum": 5,
        "Source": "scrape代理"
    }
]

我将得到的这些ip 和 port 放到我的程序中,我使用了colly 这个工具, 代码类似:

        ...
	// 获取可被抓取的域名
	domains := strings.Split(spiders.Domains, ",")

	// 使用go_proxy_pool 得到的代理ip和port
	proxyUrls := []string{
		"http://117.160.250.133:9999",
		"http://117.159.15.99:9091",
		"http://116.113.68.130:9091",
		"http://221.6.215.202:9091",
		"http://117.160.250.138:8080",
		"http://117.160.250.134:8081",
		"http://117.160.250.131:8899",
		"http://27.15.232.197:9091",
		"http://222.139.221.185:9091",
		"http://42.228.61.245:9091",
	}

	rp, err := proxy.RoundRobinProxySwitcher(proxyUrls...)
	if err != nil {
		log.Fatal(err)
	}

	// Collector
	collector := colly.NewCollector(
		colly.AllowedDomains(domains...),
		colly.AllowURLRevisit(),
	)
	collector.SetRequestTimeout(120 * time.Second)
	collector.SetProxyFunc(rp)
        ...

但是启动后,总是报下面Access Denied的错误,类似:


2023/03/22 15:10:36 (D:/work/aaa/main.go:133 main.Run.func1) [info] OnRequest: Req.ID: 2, urlType:listPage, Req.URL: https://www.baidu.com/list/, q.IsEmpty: true, q.Size: 0, q.Threads: 20
2023/03/22 15:10:46 (D:/work/aaa/main.go:300 main.Run.func4) [info] OnError: [2]listPage, https://www.baidu.com/list/, Get "https://www.baidu.com/list/": Access Denied

请问是哪里,搞错了?

@pingc0y
Copy link
Owner

pingc0y commented Mar 23, 2023

我不清楚你的代码是否有问题,但可能是免费代理的质量问题,如果报错可以更换ip尝试。(可以通过不断使用verify去验证代理,来筛选出高可用的代理ip)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants