Twitter changes break TwitBlock [again]

Last week a another change to Twitter caused me problems with my personal project TwitBlock. For the impatient, see my Google groups post about it. (It didn't go down very well).

If you're a Twitter user, you're probably familiar with this image:

It is/was the default profile image for users that have not uploaded a custom avatar. You may also have noticed last week that Twitter has introduced a new version. Actually they they made seven of them in different colours:

At least I think they made seven; I can't find any more, but I can't find any official document stating how many are out there either.

So what?

So … TwitBlock crawls Twitter for duplicate profile pics to help identify spam accounts. The app needs to know what images are the default ones, because otherwise it will penalize people heavily for having what appears to be the same image as thousands of other people.

This relies rather delicately on factors that are liable to change and that aren't strictly a part of the developer API, so I have to keep a close eye on things. I concede this is not a very robust solution, and I certainly wouldn't base a commercial product around such weak "technology". In fact I'm not sure I'd base a commercial product around Twitter at all.

I get a lot of emails and DMs from people telling me that they've received errors using TwitBlock. Almost always this is due to the Twitter API failing to respond – either timing out or sending back some HTTP error. It's quite embarrassing, and I can only imagine how much worse this would be if people were paying for a Twitter-based service.

A Twitter app doesn't just rely on the API, it relies on everything that makes up the Twitter service. This includes its full feature set and its hardware infrastructure. I am of the opinion that the above-described profile image change was significant enough that Twitter should have documented the change in advance. Facebook do a good job of addressing the community far in advance of changes, and I think this is yet another indicator that Twitter is out of its depth.

Tags: rant, twitblock, twitter

This entry was posted on Saturday, September 19th, 2009 at 7:43 pm and is filed under General. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

8 Responses to “Twitter changes break TwitBlock [again]”

Quix0r Says:
September 19th, 2009 at 8:46 pm
My approach for X default avatars would be:
- Get MD5/SHA1/SHA256 hash from all images twitter is using as default
- Save these in an array:
+ Index is the file name (no path or domain)
+ Values are the hashes
Then you should be able to easily lookup default images:
- If the basename()-name matches with an index (use isset() ) and the MD5 hash of the downloaded (or don’t you download avatars?) matches you have found a default image.
- If you don’t download any avatar to your server for checking then forget about the hash and you need to verify the whole URL (then you need an index array and use in_array() to lookup).

Tim Says:
September 19th, 2009 at 8:54 pm

I’m currently ignoring the file paths, but if anyone wants them the hashes are as follows:

$ curl "http://s.twimg.com/images/default_profile_normal.png" 2>/dev/null | md5
df22f2c8e1a4bd024d02b10a3f3bd1cc
$ curl "http://s.twimg.com/a/1252980779/images/default_profile_0_normal.png" 2>/dev/null | md5
596e7d6cfb43be42dcce2cba80d55838
$ curl "http://s.twimg.com/a/1252980779/images/default_profile_1_normal.png" 2>/dev/null | md5
e616bb5ea460fdef652185e67a112189
$ curl "http://s.twimg.com/a/1252980779/images/default_profile_2_normal.png" 2>/dev/null | md5
045491d2d7051d144f31d35d64328475
$ curl "http://s.twimg.com/a/1252980779/images/default_profile_3_normal.png" 2>/dev/null | md5
1a67b35708dba15234866f8742f1d26b
$ curl "http://s.twimg.com/a/1252980779/images/default_profile_4_normal.png" 2>/dev/null | md5
9880fcc60b9b07c852c015d43c0336de
$ curl "http://s.twimg.com/a/1252980779/images/default_profile_5_normal.png" 2>/dev/null | md5
dcff70117d93e84acac5cfdc1bef70d2
$ curl "http://s.twimg.com/a/1252980779/images/default_profile_6_normal.png" 2>/dev/null | md5
c82efa3074189567b80c85752f5ed441

Pepijn de Vos Says:
September 20th, 2009 at 6:47 am
How about this option: Have GD check the color of some pixels in the bird if they’re white. All the default pics have the same bird. With this approach Twitter could even add new colors without breaking your code.

Anyway, if I see a default pic not listed here I’ll tell you.
Patrick Says:
October 17th, 2009 at 11:15 am
Are you going to integrate automatic spam reporting for the new twitter spam report feature?
Patrick Says:
October 17th, 2009 at 11:27 am
Disregard the question, i see it’s already implemented.
Raul Carrillo Garrido aka metsuke Says:
December 21st, 2009 at 12:16 pm
Maybe turning the image into a B/W could do the work?
Guido Waldenmeier Says:
February 8th, 2010 at 4:09 pm
OFF TOPIC

Tweets via API
This account’s last tweet was from an unregistered app.
This can happen if you enter your password directly into a bad app. Only use apps that authenticate you via Twitter.com

BULLSHIT MISTER !
i use the blog CMS
b2evolution.net for my blog
and there is a plugin i can send blog post to twitter
it s a serious software in long term use a lot of peoples
serious as wordpress
!!!
Tim Says:
February 8th, 2010 at 7:33 pm
yes, off topic, and unecessarily agressive too.

Firstly, it is trivial to post to Twitter with your app name rather than “API” – through my observation, bots and apps that post as “API” are low quality spammy apps written by people who don’t know the Twitter API well enough to do it properly.

Secondly, this indicator has a very low weighting in TwitBlock – it is a very minor factor when scanning your followers, mainly due to the fact that people will continue to give their password to apps like these that tweet on their behalf.

Re sending your blog posts to Twitter: The popular twitterfeed app identifies itself as such, which is the correct way to build this kind of software.

Web 2.1