README is available in the following languages: Russian | English

Proxy Checker

proxy-checker - The most versatile proxy parser and proxy checker.

Why ?

I, and therefore many people, need to get a proxy to bypass the ban on IP with a large number of requests. This module allows you to perform almost all tasks related to proxy preparation. I, and therefore many people, need to build proxies individually for some sites, I also took care of this. I hope my contribution will make it easier for someone life.

Get started

Install the npm module @prohetamine/proxy-checker

$ npm install @prohetamine/proxy-checker

or

$ yarn add @prohetamine/proxy-checker

Examples and description

Connecting the module

const proxy = require('@prohetamine/proxy-checker')

load

The load function collects proxies and easily returns them via all and random.

object

key	value	default value	required	information
port	number	false	none	used for the port filter
debug	boolean	false	none	used for debugging

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  const count = await proxy.load({
    port: 8888,
    debug: true
  })

  console.log(count) // Number of collected proxies with port 8888
})()

loadInterval

The loadInterval function collects proxies with an interval of N milliseconds and easily returns them via all and random.

options	default	info
function	() => { /* ... */}	callback function returns in a single argument the number of IP loaded
int	5000	interval in milliseconds
object	{ started, port, debug }	additional options

object

key	value	default value	mandatory	information
started	boolean	false	no	use for instant download
port	boolean	false	no	used to filter ports
debug	boolean	false	none	used for debugging

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  const kill = await proxy.loadInterval(count => {
    console.log(`count parse: ${count}`) // Number of collected proxies with port 8080
  }, 60000, { started: true, debug: false, port: 8080 })

  console.log(proxy.all({ port: 8888 })) // All downloaded IP addresses with port 8888
  console.log(proxy.random({ port: 80 })) // One random IP address with port 80

  setInterval(() => {
    if (proxy.all().length > 1488) {
      console.log(proxy.all())
      kill() // kills the current loadInterval
    }
  }, 5000)
})()

all

The all function returns an array of IP addresses when it is loaded load or loadInterval and false when the load has not yet occurred.

object

key	value	default value	required	information
port	number	false	none	used for the port filter

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  const count = await proxy.load({
    debug: true
  })

  console.log(proxy.all({ port: 1488 })) // All downloaded IP addresses with port 1488
})()

random

The random function returns a single random IP address when it is loaded load or loadInterval and false when the load has not yet occurred.

object

key	value	default value	required	information
port	number	false	none	used for the port filter

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  const count = await proxy.load({
    debug: true
  })

  console.log(proxy.random({ port: 8888 })) // One random IP address with port 8888
})()

checkerInterval

The checkerInterval function checks proxies with an interval of N milliseconds for a specific site and easily returns them via get.all and get.random.

parameters	default value	information
string	null	unique identifier
object	{ url, isBrowser, browserConfig, timeout, stream, debug, indicators, session, onData }	additional options

object

key	value	default value	mandatory	information
url	string	null	no	website link
isBrowser	boolean	false	none	used for checking through the browser
trashIgnore	boolean	false	none	used for optimization, removes loading of media, xhr, fetch, websocket, manifest, image, stylesheet, font, script
browserConfig	object	{ ... }	none	used to configure the launcher
onData	function	() => { ... }	none	used to get the body of a successful request (works only for browser)
timeout	int	10000	none	interval in milliseconds
stream	int	2	none	number of concurrent threads
debug	boolean	false	none	used for debugging
indicators	array	[]	none	indicators of correctly loaded data
session	string	false	none	path to save valid IP addresses after completion or saving
quarantineMode	boolean	false	none	used to quarantine non-working proxies

array[object]

key	value	default value	required	information
keyword	string	no	yes	used to search the page by keyword
selector	string	no	yes	used for searching on a page by dom node

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  await proxy.loadInterval(count => {
    console.log(`count parse: ${count}`) // Number of collected proxies
  }, 60000, { started: true, debug: false })

  const { key, kill, save, clean } = await proxy.checkerInterval('chaturbate', {
    url: 'https://chaturbate.com/diana_smiley/',
    isBrowser: true,
    trashIgnore: true,
    browserConfig: proxy => ({
      headless: true,
      ignoreHTTPSErrors: true,
      args: [`--proxy-server=${proxy}`],
    }),
    timeout: 60000,
    stream: 10,
    session: __dirname + '/chaturbate.json',
    debug: true,
    indicators: [{
      //keyword: 'close_entrance_terms',
      selector: '#close_entrance_terms'
    }] // List other metrics in the objects if they are required
  })

  setInterval(() => {
    const proxys = proxy.get(key).all()
    console.clear()
    console.log(proxys, proxys.length)

    if (proxys.length > 20) {
      kill() // Kills the current checkerInterval and saves the session
      save() // Only saves the session
    }
  }, 3000)
})()

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  await proxy.loadInterval(count => {
    console.log(`count parse: ${count}`) // The number of proxies collected with the port
  }, 60000, { started: true, debug: false })

  const keyName = null // A unique identifier for IP accumulation, if set to null, will be generated automatically.

  const { key, kill, save, clean } = await proxy.checkerInterval(keyName, {
    url: 'https://github.com/prohetamine',
    timeout: 30000,
    stream: 20,
    session: __dirname + '/github.json',
    debug: true,
    indicators: [{
      keyword: 'Prohetamine',
      //selector: '#close_entrance_terms'
    }, {
      keyword: 'Stas'
    }] // List other metrics in the objects if they are required
  })

  setInterval(() => {
    const proxys = proxy.get(key).all()
    console.clear()
    console.log(proxys, proxys.length)

    if (proxys.length > 20) {
      kill() // Kills the current checkerInterval and saves the session
      save() // Only saves the session
    }
  }, 3000)
})()

get.all

The get.all function returns an array of checked checkerInterval IP addresses by a unique identifier and false when the download has not yet occurred.

get(key)

parameters	default value	required	information
string	null	yes	unique identifier

object

key	value	default value	required	information
port	number	false	none	used for the port filter

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  await proxy.loadInterval(count => {
    console.log(`count parse: ${count}`) // Number of collected proxies
  }, 60000, { started: true, debug: false })

  const { key, kill, save, clean } = await proxy.checkerInterval('chaturbate', {
    url: 'https://chaturbate.com/diana_smiley/',
    isBrowser: true,
    debugBrowser: false,
    timeout: 60000,
    stream: 10,
    session: __dirname + '/chaturbate.json',
    debug: true,
    indicators: [{
      //keyword: 'close_entrance_terms',
      selector: '#close_entrance_terms'
    }] // List other metrics in the objects if they are required
  })

  setInterval(() => {
    const proxys = proxy.get(key).all()
    console.clear()
    console.log(proxys, proxys.length)

    if (proxys.length > 20) {
      kill() // Kills the current checkerInterval and saves the session
      save() // Only saves the session
    }
  }, 3000)
})()

get.random

The function get.random returns one random of the checked checkerInterval IP addresses by a unique identifier and false when the download has not yet occurred.

get(key)

parameters	default value	required	information
string	null	yes	unique identifier

object

key	value	default value	required	information
port	number	false	none	used for the port filter

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  await proxy.loadInterval(count => {
    console.log(`count parse: ${count}`) // Number of collected proxies
  }, 60000, { started: true, debug: false })

  const { key, kill, save, clean } = await proxy.checkerInterval('chaturbate', {
    url: 'https://chaturbate.com/diana_smiley/',
    isBrowser: true,
    debugBrowser: false,
    timeout: 60000,
    stream: 10,
    session: __dirname + '/chaturbate.json',
    debug: true,
    indicators: [{
      //keyword: 'close_entrance_terms',
      selector: '#close_entrance_terms'
    }] // List other metrics in the objects if they are required
  })

  setInterval(() => {
    console.log(proxy.get(key).random())
  }, 3000)
})()

Other examples from this project can be found here: examples

Contacts

My Telegram: @prohetamine, channel

Email: [email protected]

Donat money: patreon

If you have any questions and/or suggestions, please email me in telegram, if you find any bugs also let me know, I will be very grateful.