spider-rs

All

17 repositories

spider
Public
A web crawler and scraper for Rust
rust crawler spider web-crawler web-scraper web-scraping indexer headless-chrome llm-crawler ai-scraping
Rust
•
MIT License
•102•1.2k•2•0•Updated Nov 27, 2024Nov 27, 2024
case_insensitive_string
Public
A case-insensitive string for Rust
rust string case-insensitive case-insensitive-strings
Rust
•
MIT License
•0•4•0•0•Updated Nov 26, 2024Nov 26, 2024
html2md
Public
Fast HTML to markdown lib for Rust
html markdown rust html2md
HTML
•
MIT License
•1•2•2•0•Updated Nov 25, 2024Nov 25, 2024
spider-clients
Public
Python, Javascript, and Rust libraries for the Spider Cloud API.
crawler scraper ai spider html-to-markdown web-scraping ai-agents supabase ai-scraping llm-webcrawler
Python
•
MIT License
•4•7•1•0•Updated Nov 7, 2024Nov 7, 2024
proxier
Public
Auto proxy whitelisting for 3rd parties
Rust
•
MIT License
•0•2•0•0•Updated Nov 2, 2024Nov 2, 2024
llm-readability
Public
The readability library for LLM's
readability data-cleaning clean-data safari-reader llm-training
Rust
•
MIT License
•0•5•0•0•Updated Oct 25, 2024Oct 25, 2024
ua_generator
Public
Pre-compiled random real User-Agents. Updated weekly for windows, mac, linux, and android
useragent user-agent-string ua-generator
Rust
•
MIT License
•3•25•0•0•Updated Oct 25, 2024Oct 25, 2024
auto-encoder
Public
Auto encoding library
Rust
•
MIT License
•0•0•0•0•Updated Oct 23, 2024Oct 23, 2024
chrome-server
Public
API to manage chromium headless processes easily with xvfb support
chrome headless xvfb chrome-api a11ywatch chrome-server
Rust
•
MIT License
•5•11•0•0•Updated Oct 12, 2024Oct 12, 2024
spider-nodejs
Public
Spider ported to Node.js
nodejs distributed-systems crawler scraper typescript spider indexer headless-chrome
Rust
•
MIT License
•4•30•1•0•Updated Sep 24, 2024Sep 24, 2024
spider-py
Public
Spider ported to Python
python crawler scraper spider web-crawler headless-chrome
Rust
•
MIT License
•5•51•2•0•Updated Sep 24, 2024Sep 24, 2024
spider-cloud-live-code-viewer
Public
VSCode live web crawling
spider web-crawler vscode llm ai-crawler
TypeScript
•
MIT License
•0•1•1•0•Updated Sep 11, 2024Sep 11, 2024
stylo
Public
stylo fork for web scraping
Rust
•18•0•0•0•Updated Sep 8, 2024Sep 8, 2024
homebrew-spider-cloud-cli
Public
Ruby
•0•0•0•0•Updated Aug 5, 2024Aug 5, 2024
web-crawling-guides
Public
How to guides on web-crawling or scraping
crawler scraper html-to-markdown web-scraping agents ai-agents ai-scraping llm-webcrawler clean-markdown fast-webcrawler
2•11•0•0•Updated Jul 21, 2024Jul 21, 2024
async_job
Public
Simple async cron jobs for Rust
rust cronjob async-jobs async-task
Rust
•0•2•0•0•Updated Dec 27, 2023Dec 27, 2023
jsdom
Public
JSDOM for Rust [incomplete]
jsdom dom-parser
Rust
•
MIT License
•0•4•0•0•Updated Aug 2, 2023Aug 2, 2023