-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #4 from iarsham/develop
update:readme.md
- Loading branch information
Showing
1 changed file
with
51 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,51 @@ | ||
# scrapify | ||
# Scrapify🚀: A Go Library for Web Scraping | ||
|
||
This library provides tools for building web scrapers in Go. It allows you to create custom HTTP requests with advanced features for bypassing basic anti-scraping measures. | ||
|
||
## Features | ||
|
||
- **Customizable Headers**: Set various headers like User-Agent and sec-ch-ua to mimic a real browser. | ||
- **TLS Configuration**: Customize the Transport Layer Security (TLS) configuration for secure connections. | ||
- **Browser Emulation**: Specify different browsers (Chrome, Firefox, Edge) to influence the cipher suites offered. | ||
- **Default Headers**: Includes a set of common headers to improve compatibility. | ||
|
||
## Installation | ||
```bash | ||
go get -u github.com/iarsham/scrapify@latest | ||
``` | ||
|
||
## Usage | ||
```go | ||
package main | ||
|
||
import ( | ||
"fmt" | ||
"github.com/iarsham/scrapify" | ||
"net/http" | ||
) | ||
|
||
func main() { | ||
client := &http.Client{ | ||
Transport: scrapify.NewTransport(scrapify.Chrome), | ||
} | ||
req, err := http.NewRequest(http.MethodGet, "https://cloudflare.com", nil) | ||
if err != nil { | ||
panic(err) | ||
} | ||
scrapify.SetHeaders(req, nil) | ||
resp, err := client.Do(req) | ||
if err != nil { | ||
panic(err) | ||
} | ||
defer resp.Body.Close() | ||
fmt.Println(resp.StatusCode) | ||
} | ||
``` | ||
|
||
## Contributing | ||
|
||
We welcome contributions to this library. Please feel free to submit pull requests with improvements and bug fixes. | ||
|
||
## License | ||
|
||
This library is licensed under the MIT License. See the LICENSE file for details. |