Skip to content

Commit

Permalink
Merge pull request #4 from iarsham/develop
Browse files Browse the repository at this point in the history
update:readme.md
  • Loading branch information
iarsham authored Aug 30, 2024
2 parents 593774c + 0fcb12f commit b55f964
Showing 1 changed file with 51 additions and 1 deletion.
52 changes: 51 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1,51 @@
# scrapify
# Scrapify🚀: A Go Library for Web Scraping

This library provides tools for building web scrapers in Go. It allows you to create custom HTTP requests with advanced features for bypassing basic anti-scraping measures.

## Features

- **Customizable Headers**: Set various headers like User-Agent and sec-ch-ua to mimic a real browser.
- **TLS Configuration**: Customize the Transport Layer Security (TLS) configuration for secure connections.
- **Browser Emulation**: Specify different browsers (Chrome, Firefox, Edge) to influence the cipher suites offered.
- **Default Headers**: Includes a set of common headers to improve compatibility.

## Installation
```bash
go get -u github.com/iarsham/scrapify@latest
```

## Usage
```go
package main

import (
"fmt"
"github.com/iarsham/scrapify"
"net/http"
)

func main() {
client := &http.Client{
Transport: scrapify.NewTransport(scrapify.Chrome),
}
req, err := http.NewRequest(http.MethodGet, "https://cloudflare.com", nil)
if err != nil {
panic(err)
}
scrapify.SetHeaders(req, nil)
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
fmt.Println(resp.StatusCode)
}
```

## Contributing

We welcome contributions to this library. Please feel free to submit pull requests with improvements and bug fixes.

## License

This library is licensed under the MIT License. See the LICENSE file for details.

0 comments on commit b55f964

Please sign in to comment.