Tips, Ticks, Info, and Help about refollow.com

New Twitter api restrictions will not allow us to support all users anymore :(

We are very disappointed to announce that Twitter’s new api restrictions no longer allow Refollow to support any interesting amount of users. Twitter has apparently now limited the number of follow/unfollow operations a single server can send to them (“rate limits”) rather that limiting it for each user via their api call limits. What this means is that we either need to stop supporting the 500K free users we have using Refollow, or we need to spin up many many servers to distribute the api requests to twitter from multiple places (very very expensive!)

We have asked Twitter support to whitelist Refollow to avoid these restrictions, but it doesn’t look like they are willing. It will be the same amount of requests hitting their servers whether they come from a few servers or a hundred servers, so I’m not really sure what their intent is with this move. 

We are open to any suggestions, but in the meantime, our ip address remains blocked by Twitter and it doesn’t sound like they are interested in Refollow continuing to operate as is. We are throwing around ideas including a read-only version of refollow (no follow/unfollow operations), a subscription version (so we can afford to purchase more servers), or just porting refollow to Facebook! Let us know what you think! http://twitter.com/refollow or support@refollow.com

Here is the latest communication with Twitter support:

————————————————————————-

From: Brian Sutorius [mailto:bsutorius@twitter.com]
Sent: Thursday, October 07, 2010 9:45 AM
To: Rob Meadows
Cc: Brian Truebe; Tyler Buck
Subject: Re: Refollow-up

Yes, I’m sorry this new limit was not announced prior to going into effect. We understand the effect it has on applications like yours and will be more transparent about such changes in the future. 

Brian

On Thu, Oct 7, 2010 at 9:40 AM, Rob Meadows wrote:

Also, it appears that this new limit was also affecting our users before our ip was blocked:

http://groups.google.com/group/twitter-development-talk/browse_thread/thread/eed29c3d9efcfad5?pli=1

It would be nice if we could be notified of changes like this that are going to break our user experience and previous assumptions. We are investing a lot of money in hosting this free service on fast servers so we can provide a great user experience. These types of changes cause us huge cost in terms of debugging and customer support which could easily have been avoided with some discussion or notification ahead of time. 

Thanks,

Rob

 From: Brian Sutorius [mailto:bsutorius@twitter.com]
Sent: Thursday, October 07, 2010 10:07 AM
To: Rob Meadows
Cc: Brian Truebe; Tyler Buck
Subject: Re: Refollow-up
 

Whitelisting IPs only affects their rate limit for HTTP GET calls on the REST API, and friendship/create and friendship/destroy calls are both POSTS that are not counted in this rate limit. Additionally, social graph calls are expensive on our end, which is part of the reason we’ve begun to rate limit them separately. 

If load-balancing these requests across multiple servers is not viable, you must at least batch these queries to send them at a steady rate to Twitter and hopefully offload some traffic to off-peak hours. At scale, this of course can cause delays between when the user requests an action and sees the output of this action, but this negative effect on user experience is hopefully better than no service or long outages.

Brian

On Thu, Oct 7, 2010 at 9:57 AM, Rob Meadows wrote:

Is there no way to white-list our ip? Load balancing these requests across more servers would be a significant change to our architecture, and not something that would make sense for us to invest in. In the end, it would result in the same amount of calls to your api’s. We have invested in a few high capacity servers rather than multiple small capacity servers, so this is not a small change you are requesting.

As far as the new ip-based limit, this has the same implication. We would need to spread our requests across many ip’s in order to scale.

We have had a significant increase in the number of users using Refollow over the last few months and have invested in an architecture that will scale. Basically what you are telling me is that Twitter is limiting us on the number of users we can allow to use our service unless we change our architecture completely. Refollow is currently offered for free to your users – no fees, no advertising, etc. The only way we will be able to limit our number of users will be to stop accepting new users or start charging for the service.

From: Brian Sutorius [mailto:bsutorius@twitter.com] 
Sent: Thursday, October 07, 2010 9:25 AM
To: Rob Meadows
Cc: Brian Truebe; Tyler Buck
Subject: Re: Refollow-up
 

Hi Rob,

I can confirm that this IP is being blocked on our side for sending an abusive amount of follow calls. These blocks are placed automatically by our anti-abuse systems, and to prevent this going forward, I suggest adding another box or two and load-balancing your traffic so that the social graph calls you send to us are in line with our API rate limits, even if they’re technically not-rate-limited POST calls.

Please let me know when you’ve made these changes or stepped down the rate of these calls and I will remove this IP from our blacklist. I’m sorry for this inconvenience, and can answer any additional questions you may have.

Brian Sutorius

————————————————————————-