Monday, August 18, 2014

How to Find Good Free Site-Specific Proxies

I'm always looking for a source for good free proxies for my scraping projects. Have you ever tried scraping a site that blocks by geography or limits requests by IP addresses? If so then you will appreciate this post.

Having recently stumbled upon Jrim's Proxy Multiply, it's now my go-to tool for grabbing some quick free proxies. I just hit the "Get proxies!" button and within an hour or two I've got 3,000 tested working proxies.



And if you need some site-specific proxies, it can do that too. Just follow these steps:
  • open the Configure menu
  • go to HTTP Proxy Testing
  • check Check proxies against a specific url and click configure
  • in the dialog that opens, put the homepage (http://www.manta.com, http://www.yelp.com, etc) in the Url to Navigate to textfield
  • in the Content to look for in the page you want to put any text that will always show up on a page (I like to use the google-analytics id)