In case you’ve never heard of it, I wrote a little Twitter friend/follower cross-reference tool a few years ago. Basically, I was wondering which of the people I followed on Twitter also followed me back, who didn’t follow back, who followed me that I didn’t follow back, and all the permutations around those ideas. After a couple of days of hacking, Twitual was born.

One of the main problems with Twitual, however, is that the data gathering and analysis is linear. When you submit a Twitter username, the server has to fetch all of the friends, fetch all of the followers, and do all of the set calculations before it can send a single thing back to your browser. Due to the way the Twitter API works, it may have to make many HTTP calls to the Twitter servers. It can be slow, and you’re left with a browser that appears to be frozen, with nothing happening, until it finally updates the page all at once.

For a long time, I’ve wanted to rewrite Twitual to be more dynamic, and give more friendly feedback while it’s working. So some time ago, I started toying around with rewriting things using nodejs instead of PHP. My early experiements were promising. I used socket.io for real-time client/server updates, found a good Twitter API library for node, and threw together a quick prototype. But after playing with it for a bit, I kept seeing occasional failures where the server side just seemed to stop working, along with a mysterious error in my browser’s JavaScript console.

Continue reading Dealing with UTF in node.js →