Skip to main content

​
Introduction

Scraping large amounts of data from websites can sometimes lead to challenges like IP blocking, rate limiting, and detection by anti-scraping mechanisms. Proxies play a crucial role in mitigating these challenges and ensuring the success of web scraping operations.

​
Why Proxies are Needed for Web Scraping

1

IP Blocking and Rate Limiting

Websites often employ measures to prevent automated scraping activities, including IP blocking and rate limiting.
2

Anti-Scraping Mechanisms

Many websites deploy sophisticated anti-scraping mechanisms to detect and block scraping bots.
3

Geographic Restrictions

Some websites enforce geographic restrictions on their content, limiting access to users from specific regions or countries.

​
Types of Proxies

ScrapexLabs offers the following proxy options for scraping websites:
  • Standard - datacenter proxies
  • Premium - residential proxies
  • Stealth - high-quality pool of residential proxies
  • Custom - your own proxy url

​
Standard

Datacenter proxies are IP addresses provided by data center providers. These proxies are suitable for scraping websites, that don’t have anti-bot protections, but have geographic restrictions or rate-limiting procedures. However, datacenter proxies may be more likely to be detected and blocked by websites with stringent anti-scraping measures. To select this type of proxy, set request body parameter proxy_type to standard.

​
Premium

Residential proxies are IP addresses assigned to devices connected to residential ISPs (Internet Service Providers). These proxies mimic real user traffic, making them highly effective for bypassing anti-scraping mechanisms. To select this type of proxy, set request body parameter proxy_type to premium.

​
Stealth

Stealth proxies offer even higher level of anonymity and are less likely to be detected by websites. To select this type of proxy, set request body parameter proxy_type to stealth.

​
Custom

If you prefer to use your own proxies, you can do so by setting request body parameters: proxy_type = custom
proxy_url = your proxy url Valid proxy urls are:
  • username:pass@serverOrIP:port. Example: user1:[email protected]:8080
  • serverOrIP:port. Example: superserver.com:8005

​
Proxy Location

You need to choose the country, where the proxies are located, by setting parameter proxy_country to the country code, e.g. us, et, it. You can see a full list of country codes from in API endpoint page.

​
Rotating proxies

standard, premium, stealth proxies are auto-rotating on every request. If you want to keep the same IP for multiple requests, you can do so by setting parameter proxy_session_id to some string, that you should save on your side. For example, mysession123. Besides, you can control for how long the same IP should be reserved for your session with proxy_session_lifetime. Default is 180 seconds. So, for example, you can use the same IP for your requests during 5 minutes by setting: 
"proxy_session_id": "randomsessid551",
"proxy_session_lifetime": 300