Skip to content

Latest commit

 

History

History
48 lines (30 loc) · 1.02 KB

README.md

File metadata and controls

48 lines (30 loc) · 1.02 KB
Raw

🕷️ WebCrawlerX 🚀

A flexible and efficient web crawler written in Rust.

Features

  • Multiple spider implementations (CVE Details, GitHub, Quotes)
  • Configurable crawling parameters (delay, concurrent requests, page limit)
  • Easy to extend with new spiders

Installation

cargo install webcrawlerx

Usage

List available spiders:

webcrawlerx spiders

Run a specific spider:

webcrawlerx run --spider <spider_name>
--spider <spider_name> [--delay <ms>] [--concurrent <num>] [--limit <num>]

Example:

webcrawlerx run --spider cvedetails --delay 200 --concurrent 2 --limit 10

Adding a New Spider

To add a new spider, create a new module in the spiders directory and implement the Spider trait. Then, update the run_spider function in main.rs to include your new spider.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.