Pages Navigation Menu

Advanced SEO Tutorials

Categories Navigation Menu

Proxies

Let Others KnowShare on FacebookTweet about this on TwitterShare on Google+Pin on Pinterest

Until May 2013 out advice has always been ‘most people do not need proxies in SEnuke’, that changed when we developed a method to use multiple proxies at once and use proxies in a special way with Proxifier. Interestingly one year later in May 2014 we have to change the recommendations again, the massive number of regbot complaints seen in the USA these days means many more people should be using proxies today.

There are two common ways to hide your IP, one using proxies which are generally application specific and the second using a VPN, which generally works on a machine level and is a cheaper solution.

  • Fully Private Proxies from Squid Proxies
    You have exclusive use of these, and you can authorise them for up to 10 IP, there is no option to password authorise. Squid Proxies are great when you need proxies which you can hammer hard with traffic all day. However they generally are not so good for scraping as they tend to be frequently banned by Google, for SEnuke use this is not an issue though.
  • Semi Private Proxies from Buy Proxies
    Much cheaper per proxy than Squid, but you are sharing these with up to 3 people. You can enable 2 IPs on these, or use password authorisation. Owing to this lower limit if you need to run proxies on 3 or more machines is will be cheaper to go for dedicated proxies from Squid. However one advantage of Buy Proxies is they generally do work for scraping, and in addition they allow them to be used with product like SEnuke.
  • Hide My Ass VPN
    The cheapest option by far, and with the most IPs, but one that should generally be avoided with SEnuke. You can only use one IP at a time, so advanced methods using proxifier (see below) will not work. There are also often problems with email authentication, which again we can get around with other proxies by using proxifier. We do not recommend using HMA with SEnuke.

A proxy is simply another server that acts as a intermediary point between you and the destination. In simple terms you send you data to the proxy server and then it re-sends it to the final destination, any reply is first sent to the proxy server and then forwarded on to you. Essentially the destination computer has no idea of the existence of your computer, and as far as it is concerned it only deals with the proxy server.

Typically proxies are setup on a per application basis, although they can be setup on hardware routers/modems etc. Thus it is possible to have a proxy server setting in Internet Explorer that gives it a US IP, and at the same time run Firefox with a different proxy server, or even no proxy server at all. By default many programs within windows do not have proper proxy controls, and they take their proxy settings from Internet Explorer.

You can add proxies directly in to SEnuke, it is in ‘Options’ > ‘Proxy’. However it is much better to use an external program called ‘proxifier’ which will give you greater control of how proxies are used.

Generally in the past we have advised against using proxies in SEnuke, they just were not needed. However these days we use proxies all the time, and suggest you do to.

Firstly lets look at some of the issues with proxies.

Speed: Proxies Can Be Slow

This is easier to show by way of an example, so lets take a look at how a proxy impacts SEnuke by testing them on a working server:

No Proxies:

One Semi Private Proxy (Same data center)

One Dedicate Squid Private Proxy

Now that is only talking to a nearby server, lets add some distances and go cross Atlantic as many Nukes will do:

No Proxy

One Semi Private Proxy

One Dedicated Squid Proxy (I don’t know why the distance shows as 7550 miles)

Now we are using premium equipment here yet you can see that pushing our connection through a single proxy massively reduced our bandwidth. If you were trying to run 50 threads of SEnuke through that Semi private proxy you would be seeing around 50kbit/s for each thread which is slower than dialup internet! These proxies can drop down to as low as 0.5mbit/s average speeds, and that gives only 10kbit/s for each thread in SEnuke.

So you can see that running with just one proxy has the potential to slow things down so much that you will get timeouts. The problem is with SEnuke only picking one random proxy per campaign you never quite know what you are going to get, if that proxy is on a go slow or has an issue for that period of time then wave goodbye to the entire campaign.

What we need here is the ability to use more than one proxy at once, even if your proxies are only running at 0.5mbit/s if you have 30 to use all at the same time then you still have around 15mbit/s bandwidth to use.

Captcha Issues: Too Many Request Turns Up The Heat

The second big issue that we can resolve with proxies are captchas, or more to the point avoiding getting the ‘angry captcha’. You see there are many types of captcha and when you are just browsing the Internet normally you will generally only ever see the basic ones:

easy captcha

However if you start to send too many requests then you will start being sent harder types:

hard captcha

These are impossible to solve through OCR methods and even the Human solvers will have great trouble, this results in more failures, thus more re-attempts and consequently higher costs to you.

In 2014 there is another issue, the blob captcha, which comes after you make a large number of requests from a particular IP. This captcha is quite hard to solve, OCR for instance pretty much always fails, the only solution to get around it is to use more proxies… however normally this is not economically viable. We do have a limited availability service that provides 15,000+ proxies for this use, please contact us to be added to the waiting list if you are interested.

If you use no proxies or let SEnuke control your proxies then you are effectively making all captcha requests from the same IP over and over gain. It is hard to define exactly when you will start to get these complex captcha types, and each captcha service has its own rules. However when it comes to re-captcha if you were to run a Full Monty with 3 profiles using 50 threads you can be 99% sure by the time you are creating the 3rd set of accounts you will already be getting the harder captcha types.

This can however be resolved by using multiple proxies, because unlike using your proxies directly in SEnuke if you use multiple ones then your captcha requests will come from multiple IPs and thus you can go much longer without getting flagged. If you have 10 proxies you will be able to go 10 times further before you get the harder captcha, and those running with large proxy lists 100+ are unlikely to ever see these harder captchas ever again.

Solutions

So we can see that using proxies can have positive benefits over and above stopping regbot complaints, however you will not get these benefits by simply copying proxies in to SEnuke because it can only use a single proxy at a time. There are however alternatives:

Using Hide My Ass (Not Recommended)

To use Hide My Ass (HMA) with SEnuke you should start SEnuke first, this is not essential but can help you to pass license validation with SEnuke. Then simply open Hide My Ass, enter your password and username if not already done so, select a country from the dropdown list (closer ones to you will be faster) and press ‘Connect to VPN’. If you are working at home you can use any protocol, if you are using one of our Accu VPS then you must only use OpenVPN. If you use PPTP you will lock yourself out of the server, and this will require a reboot to get you back in, it is best to NOT tick ‘Auto-connect after launch’ if you are using the VPS.

HMA is however a poor solution to both the problems listed above. Firstly using SEnuke through HMA is a violation of their T&C, people have had their accounts banned in the past and will again in the future. I recommend if you use HMA then you pay monthly so you don’t get a whole years subscription confiscated. Secondly you will need to set it to change IP every few minutes in order to try and limit the impact of harder captchas, because you can only connect the computer to one HMA IP at any one time, there is no multi-threading. However because their is no multi-threading you are stuck with the slow speed that HMA give you, each IP you get will have a different speed from a few hundred kbit/s to a couple of megabit/s. You however will never know what you are going to get so will have to setup SEnuke for worst case scenario.

We are aware HMA is popular, it is so popular we have a link on the front page of this site, but this is mainly because it is cheap (and perhaps more importantly they pay a much higher affiliate payout than proxy companies do). So we honestly recomend you don’t use HMA unless you really have no alternative.

 

Using Proxifier To Control Proxies

We found this solution by accident while trying to getting MegaOCR to work with a human captcha solver redundancy, and you can see the fruits of that work in our SEnuke Spetsnaz configuration.

Proxifier may not work with cheaper VPS solutions that use Virtuozzo, that is because of the way they share resources. Check with your supplier first. It will work OK with superior VPS like those from Accu which use Hyper-V.

Proxifier sits between your applications and the Internet and allows you to manipulate things pretty easily, it essentially gives you full control over what uses your proxies and how it uses them. So if we want to take all SEnuke traffic and send it through 100 different proxies we can easily set a rule to do this. If we want to make sure email downloading is not slowed by proxies we can add in an exception for that. If we want to redirect calls to Captcha Sniper, which is integrated in to SEnuke, and send them to a different solver we can do that.

It allows us to:

  • Use all our proxies at once for maximum bandwidth
  • Limit our per IP requests to captcha services
  • Stop regbot complaints reaching you
  • Reduce the impact of a failed proxies

There are other programs out there that will do a similar thing, we tested a few and had various performance issues with them all which is why we have stuck with proxifier for our own use.

The only issue with Proxifier is that you can’t copy in a list of proxies in one go, however we have had a small free program made up (download below) which will do this for you so please watch this video on how to use it:

You can get the program for adding proxies on our SEnuke Spetsnaz Configuration page.

Proxifier has a 31 day trial period, so I suggest you make use of this trial before deciding if the application is for you.

Pobman

 

 

 

Let Others KnowShare on FacebookTweet about this on TwitterShare on Google+Pin on Pinterest

Pin It on Pinterest

Share This