Tips, Ticks, Info, and Help about refollow.com

New Twitter api restrictions will not allow us to support all users anymore :(

We are very disappointed to announce that Twitter’s new api restrictions no longer allow Refollow to support any interesting amount of users. Twitter has apparently now limited the number of follow/unfollow operations a single server can send to them (“rate limits”) rather that limiting it for each user via their api call limits. What this means is that we either need to stop supporting the 500K free users we have using Refollow, or we need to spin up many many servers to distribute the api requests to twitter from multiple places (very very expensive!)

We have asked Twitter support to whitelist Refollow to avoid these restrictions, but it doesn’t look like they are willing. It will be the same amount of requests hitting their servers whether they come from a few servers or a hundred servers, so I’m not really sure what their intent is with this move. 

We are open to any suggestions, but in the meantime, our ip address remains blocked by Twitter and it doesn’t sound like they are interested in Refollow continuing to operate as is. We are throwing around ideas including a read-only version of refollow (no follow/unfollow operations), a subscription version (so we can afford to purchase more servers), or just porting refollow to Facebook! Let us know what you think! http://twitter.com/refollow or support@refollow.com

Here is the latest communication with Twitter support:

————————————————————————-

From: Brian Sutorius [mailto:bsutorius@twitter.com]
Sent: Thursday, October 07, 2010 9:45 AM
To: Rob Meadows
Cc: Brian Truebe; Tyler Buck
Subject: Re: Refollow-up

Yes, I’m sorry this new limit was not announced prior to going into effect. We understand the effect it has on applications like yours and will be more transparent about such changes in the future. 

Brian

On Thu, Oct 7, 2010 at 9:40 AM, Rob Meadows wrote:

Also, it appears that this new limit was also affecting our users before our ip was blocked:

http://groups.google.com/group/twitter-development-talk/browse_thread/thread/eed29c3d9efcfad5?pli=1

It would be nice if we could be notified of changes like this that are going to break our user experience and previous assumptions. We are investing a lot of money in hosting this free service on fast servers so we can provide a great user experience. These types of changes cause us huge cost in terms of debugging and customer support which could easily have been avoided with some discussion or notification ahead of time. 

Thanks,

Rob

 From: Brian Sutorius [mailto:bsutorius@twitter.com]
Sent: Thursday, October 07, 2010 10:07 AM
To: Rob Meadows
Cc: Brian Truebe; Tyler Buck
Subject: Re: Refollow-up
 

Whitelisting IPs only affects their rate limit for HTTP GET calls on the REST API, and friendship/create and friendship/destroy calls are both POSTS that are not counted in this rate limit. Additionally, social graph calls are expensive on our end, which is part of the reason we’ve begun to rate limit them separately. 

If load-balancing these requests across multiple servers is not viable, you must at least batch these queries to send them at a steady rate to Twitter and hopefully offload some traffic to off-peak hours. At scale, this of course can cause delays between when the user requests an action and sees the output of this action, but this negative effect on user experience is hopefully better than no service or long outages.

Brian

On Thu, Oct 7, 2010 at 9:57 AM, Rob Meadows wrote:

Is there no way to white-list our ip? Load balancing these requests across more servers would be a significant change to our architecture, and not something that would make sense for us to invest in. In the end, it would result in the same amount of calls to your api’s. We have invested in a few high capacity servers rather than multiple small capacity servers, so this is not a small change you are requesting.

As far as the new ip-based limit, this has the same implication. We would need to spread our requests across many ip’s in order to scale.

We have had a significant increase in the number of users using Refollow over the last few months and have invested in an architecture that will scale. Basically what you are telling me is that Twitter is limiting us on the number of users we can allow to use our service unless we change our architecture completely. Refollow is currently offered for free to your users – no fees, no advertising, etc. The only way we will be able to limit our number of users will be to stop accepting new users or start charging for the service.

From: Brian Sutorius [mailto:bsutorius@twitter.com] 
Sent: Thursday, October 07, 2010 9:25 AM
To: Rob Meadows
Cc: Brian Truebe; Tyler Buck
Subject: Re: Refollow-up
 

Hi Rob,

I can confirm that this IP is being blocked on our side for sending an abusive amount of follow calls. These blocks are placed automatically by our anti-abuse systems, and to prevent this going forward, I suggest adding another box or two and load-balancing your traffic so that the social graph calls you send to us are in line with our API rate limits, even if they’re technically not-rate-limited POST calls.

Please let me know when you’ve made these changes or stepped down the rate of these calls and I will remove this IP from our blacklist. I’m sorry for this inconvenience, and can answer any additional questions you may have.

Brian Sutorius

————————————————————————-

Twitter has blocked our ip (again)

For the third time, Twitter has managed to block refollow’s ip (preventing anyone from logging in through OAuth). This has come without notification to us, nor do we have any answers from them yet as to why.

Here’s our request in case you are curious!

—————————————————————————

Dear Twitter:

It appears that Twitter/OAuth is blocking Refollow’s ip address again (184.73.201.196) which has resulted in a day of nobody being able to login. We have received 1000’s of complaints already and we are not sure what to tell people.

Is there a process we should go through to prevent this? We could go and change our IP again like we did the last two times we got blocked, but that doesn’t seem like a good long term solution, and will take a couple days for the dns change to propagate.

Also, it appears that this new limit was also affecting our users before our ip was blocked:

http://groups.google.com/group/twitter-development-talk/browse_thread/thread/eed29c3d9efcfad5?pli=1

 It would be nice if we could be notified of changes like this which are going to break our service. We are investing a lot of money in hosting and maintaining this free product for your users on expensive servers so we can provide a great user experience. These types of changes cause us huge costs in terms of debugging and customer support, which could easily have been avoided with some discussion or notification ahead of time. 

We would really appreciate your support in continuing to provide your users with a reliable, positive experience.

—————————————————————————

Stop getting your Twitter accounts suspended! :)

So Twitter has informed us that the number of accounts getting suspended for using Refollow to aggressively follow is growing. Of course the number of Refollow users is also growing, but who really paid attention in stats class anyway?

But seriously, we don’t want you getting suspended (unless you are one of those annoying spam bots, then at least you don’t have feelings when you get suspended). Twitter asked us to remove the “Select All” and “Select Page” options from Refollow. We’ve explained that we are not comfortable sacrificing the user experience for the majority because of a small percent of users who will just find another way to abuse Twitter with or without Refollow.

So as a compromise, we’ve just removed the “Select All” button (which let you blindly select all users from all pages). You can still do a “Select Page”, which will select the 100 users you can see on the current page. Don’t you think you should at least look at the picture of someone you are about to follow, unfollow, or lock? There are some scary profile pictures out there, and we’re still working on the Hot or Not filter :)

Thanks to all our loyal users and keep refollowing (carefully)!

Good news - OAuth is back!

After some constructive discussions with the Twitter API Support group, Twitter has lifted the suspension and re-enabled OAuth for Refollow. We discussed some valid concerns that Refollow could be used as a tool for spammers to aggressively follow and target users. The challenge is that it’s the same user experience that enables the non-spammers to discover interesting people and help narrow down those they would like to follow or unfollow.

We agreed to add some features to the next release to help limit aggressive following (which will ultimately cause Twitter to suspend an account anyway), and to think about ways to help catch/discourage spammers. Definitely continue to crowd-tag the spammers within Refollow since that’s a great filter to avoid following them by accident. If anyone has other ideas, please share with us via Twitter (@refollow) or email (support@refollow.com)!

Thanks again to Brian and the rest of the Twitter API support group for being proactive and reaching out to work together on a resolution.

We’ve been suspended by Twitter :(

Despite our best efforts to reach out to Twitter to clarify what they are asking us to change about Refollow (which went ignored by them), Twitter has decided to suspend Refollow. This means you can no longer use OAuth to login (but for now, basic Twitter authentication still works).

We assume Twitter knows what is best for their overall ecosystem, but we certainly question what exactly that vision is. Without the ability to discover the nature of your relationships with others (or potential new relationships) and take actions, what’s the point?

We will continue to reach out to Twitter support and see if we can get a dialog going, but in case that fails, thank you to all 100,000+ of you that regularly use Refollow! Feel free to keep on Refollowing until Twitter figures out how to block basic authentication as well :) Guess it’s time to start the port to Facebook!

In case anyone is interested, here are the emails we exchanged with Twitter support:

From Twitter Support:


From: sutorius [mailto:notifications-support@twitter.zendesk.com]
Sent: Tuesday, January 19, 2010 10:01 AM
To: Support
Subject: #827777 Twitter Support: update on “Your site Refollow.com”

Support, Jan 19 10:01 am (PST):

Hi,

We’re writing to let you know that your site, Refollow, breaks our Automation Rules and Best Practices (http://help.twitter.com/forums/10711/entries/76915). Specifically, it facilitates automated bulk user following and unfollowing, which is not allowed. It’s best for both our users and your users if your application follows the rules, so please make the necessary changes to bring your application into compliance.

We’ll check back in on your application on Monday, February 1st. If these changes aren’t made by then, we may take action against it. Thank you in advance for your cooperation.

Let us know if you have any questions!
Brian
Twitter API Support

Our reply:


From: Rob Meadows
Sent: Tuesday, January 19, 2010 11:33 AM
To: ‘Twitter Support’
Cc: Tyler Buck; Jeremy Miller
Subject: RE: #827777 Twitter Support: update on “Your site Refollow.com”

Hello,    Can we please setup a time to discuss the specifics of this request? Refollow does not automate following or unfollowing any more than the Twitter web interface itself. Users must still go through and select other users to take action on, then manually choose to follow/unfollow the selected users. The only enhancement Refollow provides beyond following/unfollowing each user individually is that it optimizes the use of your api calls to follow/unfollow in groups of 20. But users must still select specific users after filtering and examining them.    Refollow is designed to help users discover things about their twitter social relationships (such as users who have become inactive, users who are interested in certain things, users who are most influential, etc.) and perform actions on these users such as following/unfollowing/blocking/tweeting. The application is not intended for the purpose of spamming, aggressive following, building followers, etc., nor does the application encourage such actions.    Please let us know the next steps.   Thanks,
Rob  

Rob Meadows
CEO Originate Labs
www.originatelabs.com

Guess not…:


From: sutorius [mailto:notifications-support@twitter.zendesk.com]
Sent: Monday, February 01, 2010 11:18 AM
To: Support
Subject: #845863 Twitter Support: update on “Your site Refollow.com”
Support, Feb 01 11:17 am (PST):

Hi,

This is a second notice that your application, ReFollow, breaks our Automation Rules and Best Practices (http://help.twitter.com/forums/10711/entries/76915). Specifically, it facilitates bulk user following and unfollowing, neither of which are allowed. Please remove these features from your application to bring it into compliance with our policies by next Monday, February 8th. If you are unable to meet this deadline, your application may be prohibited from functioning on Twitter.

Thank you in advance,
Brian
Twitter API Support

New update released

It seems like everyone is liking the new features from the 1.0 release, and we have received some great suggestions. This release (1.0.3) includes the top suggestions as well as a work-around for the recent unreliability of OAuth.

New features:

  • Option to login through basic authentication or through OAuth: OAuth (which we were already using) is more secure since you are directed through Twitter to provide your username and password, then you redirected back to Refollow with an authentication token (so we never actually see your password). OAuth is our recommended login method. But if you get an error page because OAuth is unavailable (which has been happening a lot lately!), then you can now go back to the main page and login directly there with your username and password. Or if you don’t really care that much about security (or just really trust us), this basic login method will save you an extra screen :)
  • Link from @mentions in last tweet: You can now click on a mentioned user (@username) within a tweet to load the details page for that user.
  • Link from urls in last tweet: You can now click on a url (http://site) to open that link in a new browser window.
  • Search in website and location: These are now options in the dropdown for Search and Exclude filters.
  • Reset Filters button: Clear all the settings at the top (back to the default of showing all users).
  • Jump to first and last page: When there is more than one page of users, you’ll see links at the bottom to go to “First” and “Last” page.

*Important* One more quick note: Please do be careful when following many users at a time (whether through Refollow, the web, or any other application). We have had a few users report that their Twitter accounts were suspended for aggressive following. There are no objective rules (that we know of) for how Twitter determines that following is “aggressive”, but we do know that the number of users that block your account or report your account as spam plays a role in that determination, as well as the number of users you follow each day. We recommend following no more than a hundred or two users per day (you can’t possibly take in more content than that, can you?), and making sure that you are tweeting useful content (i.e., not spam).

Enjoy!

Version 1.0 Launched!

(Actually now version 1.0.2 after a couple quick bug fix releases last night…) We’ve received so many great feature suggestions from users since our beta launch a few months ago, and this release incorporates some of our favorites:

With/Without Picture filters

If you can’t come up with a picture for Twitter, the odds are you wont come up with much content either.. These filters let you track down (or weed out) those faceless users!

Tweeted in the last X days filters

Unless you are Guy Kawasaki, you only have a precious amount of people you can follow. Why waste these follows on inactive users? These filters let you group users by their last tweet activity.

Search for/Exclude terms filters

These are multi-purpose, powerful filters for narrowing down the users in your view. These search through the loaded users’ names, bios, tags, and last tweets. The “Search for” only displays matches, while the “Exclude terms” filters out the matches. You can type in specific text to search for, or if you leave the “(Any value)” in the text field, the search will make sure there is at least *something* in that field. Some useful examples:

  • Search for [Rob] in [Name] - Show only users with a “Rob” somewhere in their name.
  • Exclude terms [Expert] in [Bio] - Filter out all self-proclaimed experts.
  • Search for [(Any value)] in [Bio] - Show only users with a Bio
  • Exclude terms [(Any value)] in [Last Tweet] - Show only users who have never Tweeted
  • Search for [Twitter] in [All] - Find “Twitter” in any field we can and only show those users.

Hopefully you get the point..

More info on mouse-over popups

We’ve packed the relationship icon, username, display name, last tweet, bio, location, website, crowd tags (more below), and number of followers, friends, tweets, and comments (more below on that, too) all into the popup you see when you move your mouse over a picture.

You’ll also notice a “Details” button - more on that in a minute. The username will link to the user’s Twitter page. Clicking “Friends” or “Followers” will automatically load those friends or followers (as if you typed that username into the “Show me” friends or followers of fields.)

User Details

By clicking the “Details” button on a user’s mouse-over popup, a “file” for that user will appear on the right. You can close this window with the X in the top right (or by clicking on “Details” button again). Some of the info here is redundant with the info on the popup, but if you like to move your mouse around the screen while you read, then this is the place to do it!

You’ll also notice some additional new stuff in that window: My Tags, Crowd Tags, and Comments.

My Tags

"Tags" are simple categories you can attach to each Twitter user. For now, we’ve limited the tags you can choose from (with full anticipation that you will immediately start complaining about our choices!). Feel free to suggest others and we’ll add them, plus I’m sure we will finish thinking through how to support your own custom tags soon enough. Right now, your choices are:

  • Products (Sells products)
  • Services (Sells services)
  • Funny (Makes me laugh)
  • Smart (Makes me think)
  • Useful (Useful content)
  • Expert (Expert in area)
  • Quotes (Great quotes)
  • Links (Great links)
  • Pics (Great pics)
  • Spam (Spammer)

Tags are useful for your own purposes (try tagging some users then applying the filter “Search for [Funny] in [My Tags]”), but when aggregated from everyone on Refollow, they become very powerful (we call these “Crowd Tags”)

Crowd Tags

Each Twitter user can have at most 3 “Crowd Tags” associated with them. These are simply the 3 top tags that all other users have applied to them. Obviously it will take some time before we gather a critical mass of tags and this becomes useful. But once we get there, this feature will be fun!

Comments

Beyond tagging a user (which is anonymous by the way), sometimes it would be nice to know what one particular user thinks about another (in 140 characters or less of course). We are allowing each user to leave one (and only one) comment about each other user. You can always go change or delete that comment, but this is meant to be more of a “review” of a user than the conversation between users you would find on Twitter.  Cool concept - let’s see how this one plays out in practice!

By the way, if you don’t like a comment a user has left for you, click your picture next to the logout link, and you can delete it from there.

Show me Users who have @mentioned me

Finally, here’s our personal favorite feature of this release. This loads all users who have previously “mentioned” you in a tweet (that is, they referred to you with an @username). Each time you load more, we attempt to load 100 users that mentioned you. One warning though - this one is a little expensive in terms of API calls. We can only load “tweets” you are mentioned in (then extract the user from there), and that costs 1 API call per 20 tweets. So each load will cost 5 API calls, and will probably not really load 100 unique users (since one user may have mentioned you in multiple tweets).

But the value of this feature is worth burning through some API calls! For example, try loading 500 or so mentioning users (once the status bar says “loaded 0”, then you’ve loaded all of them). Then filter by “I’m not Following”. If there are any, then Select All and Follow (that’s good Twitter etiquette!). Then turn off your filters, Select All, then Lock. This will ensure that you never accidentally unfollow those most valuable users that tweet about you!

That’s it.

Hope you enjoy the new features, and definitely let us know what you think! You can always send us ideas, suggestions, complaints, etc. via email support@refollow.com, or via Twitter @refollow.

Getting started (the basics)

The first time you login to Refollow, you’ll notice that there’s a lot going on. We chose to pack as much functionality and info as we could into a single screen rather than making you click through multiple screens in the app. It may be a little confusing at first, but it should quickly become very intuitive and powerful!

Here’s the screen layout:

  • At the top of the screen are your filters with check boxes to turn them on or off.
  • Below that are your sort options. These work like you would expect, reversing direction if you click them twice.
  • On the left is your grid of Twitter users - 100 max shown per page. Move your mouse over a user to see a popup of info. Click the picture to select or unselect that user (you’ll see a red check appear over the picture).
  • Below that you’ll see little numbers to navigate to other pages of the grid.
  • To the top right of the grid is your picture (click to see your info), and a link to logout.
  • Below that are actions you can take on selected users (follow, unfollow, lock, block, etc.)
  • The next row has the selection count and options for selecting all the users you see, or all the users that are loaded across all pages.
  • The “Show me” box lets you load users from various places. By default, your own followers and following relationships are loaded. But you can also load other’s followers, other’s following, etc.
  • To the bottom right of the grid is a status bar to tell you what just happened (like loading users or following people)
  • Below that is info about how many API calls you have left (you get 150 an hour from Twitter!)
  • And finally at the bottom are a couple links to follow us.

When you first login, the first thing to happen is an automatic load of 100 of your friends (people you are following) and 100 of your followers. This will take a few seconds. If you want to load more, you can click the “more” or “all” options in the “Show me” box. Each load of 100 costs 1 API call.

If you move your mouse over one of the pictures in the grid, a popup will show you some info about that user. At the top left of that popup is a little icon that let’s you know your relationship (a pink bar means you’re following them, a yellow bar means they are following you, and both bars mean you are following each other. A red “X” through the pink bar means you previously followed them, but not anymore).

Now try turning on some filters to hide some of the users in the grid. For example, check the box for “I’m Following” and also check the box for “Not Following me”. Shame on these people! If you want to stop following them since they obviously don’t care about you, just click on the “Select Page” button (notice the selection count is updated), then click on the “Unfollow” button. Watch the status bar at the bottom for an update on the progress (Twitter will take a little time to process). It’s that easy!

That should be enough to get you started. We’ll cover some of the more advanced features next!

Here’s your blog!

We’re up to almost 15,000 loyal Refollow users and everyone keeps asking for a blog with more detail than our Twitter feed at @refollow. We’ve been spending most of our time writing code instead of blogs, but now that we are out of beta, I guess it’s time to give you your blog!

For those that don’t know us, Refollow.com is a Twitter relationship manager that was developed by Originate Labs. You can do cool things like quickly view info about friends and followers, explore other’s friends and followers, filter, sort, bulk follow/unfollow/block, tag users, add user comments, and lock relationships. Most of that won’t mean much until you dive in and play with it, so give it a spin (did we mention it’s free?): http://refollow.com

And the other good news is that people seem to like us so far! Here are some of the nice things people have said:

Anyway, we’ll try to keep this blog updated with the latest info, insights, tips, and tricks about Refollow, and we welcome any and all feedback, suggestions, criticism, and praise you feel like sharing.

-@originatelabs