Growing Pains
Over the last few weeks lastfmlovetweet has gone through some growing pains. Here is a summary of what happened, and what changed:
I. Uh Oh
I was on vacation with my family, and got home to find out that my web hosting provider decided I was abusing my account by running my little lastfmlovetweet ruby script every 10 minutes.
He had a point - let me explain: Each run of the lastfm_love ruby script generated about 1800 http requests to lastfm (to check each user for new loved tracks), and then 2 more requests for each tweet generated (One for the url shortening, and one to twitter). So, I was doing about 10,080 requests per hour, and parsing as many xml documents, so yeah, my host had a point - that isn’t what a shared hosting account is meant to be used for. So, he cut me down to only running once an hour, and told me to optimize my code. That meant that loved tracks would only show up once an hour. Many users of lastfmlovetweet love multiple tracks per hour, so some users were hours and hours behind. Not good.
II. I Had a Plan. The Plan was Flawed
Fear not though, as I had expected this day to come, and I had a plan. I already had a few yahoo pipes designed that would dramatically reduce the number of requests. Basically the pipe would allow me to group together 20 users at a shot, bringing 1800 down to 90. A huge difference. The code to use the yahoo pipe was old, but it didn’t require much work to get it working again. I just needed to test it out, and so I spent an afternoon doing just that. I got it working, and then pounded on it for a while. Good thing I did. Yahoo pipes is an amazing tool, but 100% reliable it is not. I found that it was failing a lot, and seemed to be sporadically giving me cached results which are useless to me.
III. VPS to the rescue
Virtual Private Server to the rescue! I really wanted to use amazon’s EC2, but for my purposes it was a bit pricey, so after looking around a bit I decided that my current host’s VDS service was fine for my needs (the devil you know…) I spent a few more hours configuring and testing the VDS and officially moved to it Monday night (Jan. 26th). It’s been running beautifully since then.
IV. Threads
In addition to moving to a VDS, I also modified my code to use ruby threads so I can grab multiple lastfm feeds at once. I currently grab 20 at a time and that cut the processing time from about 15 minutes down to about 5 minutes. Thats what allowed me to speed up the service so that it now polls every 6 minutes.
V. Follow limit
Ugg. Just when I thought things were going smoothly, I realized that the lastfmlovetweet bot was no longer able to follow new users. This is a problem, because updating the tweet format (and new unannounced features) require a direct message, and twitter accounts can only receive directs from people they follow. I’ve opened a ticket with twitter, but so far they haven’t lifted the cap. I think they capped my account in error - it is only supposed to kick in when your friendships are reciprocated - and one way or another I’m gonna get that cap lifted! In the meantime, I unfollowed a bunch of people that are no longer using the service, so I am once again below the 2000 threshold… for now…
VI. Whats next?
The response to lastfmlovetweet has been terrific, and its been a lot of fun for me. I have a number of new features coming out that I hope you will enjoy. In addition, I am thinking about ways to make any money from this service to recoup the monthly expense of my VDS and maybe have enough left over to buy a cup of coffee. More on that later.
Any ideas, suggestions, issues, comments, thoughts, musings, PLEASE leave comments!
1 year ago