Web 2.1 » twitter

Omnibus post – 05 Apr 2010

Tim — Sat, 10 Apr 2010 14:23:15 +0000

I joked yesterday about writing one omnibus tweet per week. But actually, that’s not a bad idea at my current blogging rate. So here goes, my week in the Twittersphere -

The Digital Economy Bill

Not only am I totally unqualified to write any real critique on this topic, but there’s so much debate online already that any post I could write would be but a drop in an already soaking wet ocean.

On a personal note though, as much as the rushing-through of the bill exemplifies the illusion that I’ve always imagined democracy to be, it at least makes politics relevant to me; something I can’t say I’ve ever really experienced. I don’t think I’m alone here either as the comments on this TechRadar post illustrate. My own comment was as follows:

For me this whole affair has highlighted that some of us Digital Natives (I speak for myself) live in a bubble, and assume that the big issues facing our country are taken care of by others more knowledgeable in politics and economics. I had never watched the Parliament Channel before either, but if I was a teacher or a nurse you can bet I would have done. My point, if there is one, is that I agree with Gary that it is “business as usual” we just don’t usually notice.

My best hope from all of this is that the DEBill debacle has made people like me that have little contact with the grown-up world and who struggle to keep up with current affairs, realise that the Internet is just as much of a political issue as healthcare, education and taxation, and hopefully it will give me better reason to choose how I vote from now on.

Apple getting evil

Apple alters the legal terms of their developer SDK as they roll out iPhone OS4. Another topic on which there are volumes of content superior to what you’ll find here. If you need somewhere to start, my particular favourites are as follows:

Daring Fireball appears to have become the de-facto reaction and is a great read.
Lee Brimlow on the Flash Blog (I’m curious how much legal clout the “opinions are my own” line really carries in the end)
Level headed response from Unity 3D (it’s not all about Adobe, you know!)
Phone Gap appears unaffected as Webkit is a valid way to deploy an app. nothing on their blog yet.
I enjoyed this existential angle

On a personal note again (deeply cynical as usual) I have never been under any illusion that there is any such thing as a benevolent corporation. (There are such organisations – they are called charities). Obviously Apple want to control every aspect of deploying to their platform, they are going to continue to do so, as sure as Google are going to enter every market place they physically can until our lives are 100% reliant upon their existence. So, am I just gong to lie down and die, you ask? (Yes, probably). Seriously though, roll on W3C Widgets – I want to see the web go mobile, not see the mobile industry lock down our web.

Nearly forgot to mention UTweet

On Tuesday the Twittersphere (or at least the digital/media corner of it) erupted in massively over-the-top debate about Uniqlo’s UT campaign site, which I imagine was aimed firmly at aforementioned (ahem) influencers. For 24 hours the dialogue went like this:

People tweeted the link and said they liked it. Some even said it was incredible

In purposeful contrast to this hype, further commentators said it was nothing special and didn’t see what all the fuss was about.
The protagonists then either backed off or defended their position, creating more fuss.
The fuss was played down as being disproportionate to the event, creating more fuss.
People started blogging about it, and the fact that it was never intended to be anything more than a nice looking bit of fun in the first place. (oh, the irony)
Those that waited a whole 24 hours before commenting had the hindsight to comment on the herd itself

Not a bad exercise in getting people to talk about your brand, really.

I think I’m done. I’ve got some coding to do.

I’ve been busy, ok?

Tim — Sat, 10 Apr 2010 14:02:15 +0000

I haven’t written a post for over a month, which is pretty shameful, especially as I have masses to talk about; I just have no time. Here’s a brief catch up on my recent musings.

Birdhouse IM – A real-time replacement for Twitter DM

My last pet project BirdhouseIM is live, although still in an invite-only beta phase. I won’t replicate the FAQs here, but in short it’s a browser-based, instant messaging client that uses your Twitter identity. It’s just a bit of fun and not intended to compete with your favourite IM client, but if you’re chatting to one or more people on Twitter and want to take the conversation seamlessly into private chat, then it works pretty well.

The project was 100% an excuse to play with NodeJS, which is a very neat bit of kit. I’m a JavaScript/PHP developer, and although there’s a lot of power in that combination, mucking about with socket servers and realtime clients is limited. So you can imagine NodeJS has opened up a new world for me. I just need more excuses to play with it.

My next hair-brained scheme?

Not one to actually finish a project, I’m onto my next. I won’t go into details, except to say that it involves Twitter [obviously] and Foursquare … and it’s on the theme of hyperlocal. It doesn’t have a name yet, which always bugs me when I start developing, because I don’t know what to call stuff. I hope to have a prototype ready for the next Devnest, which gives me 10 days. In fact I realise that as I waste my time writing this I could have got some work done.

*buckles down*

Beating noisy Twitter apps

Tim — Sun, 29 Nov 2009 12:25:51 +0000

I woke up this morning to the apparent viral spread of the TweetCloud app that unoriginally, but very nicely displays your most tweeted words of the year, or month, or .. you get the idea. Here’s mine ->

If you’re impatient, you may wish to skip to the good bit.

Preamble

Now, how did this app manage such spread when there are so many like it? Possibly because it tweets from your account when your results are ready. This is not uncommon and it can be a nice feature that I might recommend. With the difference that it should be a 100% opt-in feature. TweetCloud’s start button says “make and tweet cloud“, so it does warn you. But people don’t read – they click.

TweetCloud insists that you log in before you can use it. It uses OAuth for this which is good (+1 point). Doing this means it can make calls to the Twitter API within your hourly request limit, rather than exhaust its own. (useful if you’re not whitelisted). But the real reason you must authenticate with TweetCloud is so that it can update your status. When building an app you have to seriously justify asking the user to authenticate/register etc.. As a general rule, the user should see that this action is for their benefit, not yours.

Good examples of this done right would be:

TwitPic, which has a genuine use for tweeting on your behalf.
Canabalt, a game where you want to share your score for social kudos.

Both of these apps make the tweet opt-in each time.

The good bit

While TweetCloud was busy generating the cloud (which took a minute or so) I dived off to my Twitter settings and revoked the permission I had granted the app. If you don’t know how to do this, it’s under “settings > connections”, or here: http://twitter.com/account/connections

As soon as you revoke this permission the app can no longer use the access key that it has obtained. It needs this for any API call that must be authenticated. e.g. getting your public timeline of tweets does not require authentication, whereas updating your status does.

Interestingly the cloud generation continued to churn away. This suggests that the app was actually paging through my timeline without even using authentication. i.e. making public API calls under its own rate limit.

Lo and behold, upon completion there was no tweet from my account.

I also decided to post my cloud as a TwitPic, just to say … well, you know. TwitPic doesn’t use OAuth, which it should, but that’s another post.

A few other things to note about “connections”:

When you grant access to an app, it can store its access key forever. i.e. Twitter don’t provide a key expiry feature like Facebook do. So you should revoke permissions from any app that you’ve stopped using.
My statistics from TwitBlock suggest that about 1% of people actually do this. (about 400 of 30,000 users have revoked my key)
Signing out of Twitter does not prevent the app using this access. An app with an access key can tweet from your account whenever it wants until you revoke
The read/write permission you can see is set by the app, not by you. Twitter doesn’t offer granular permissions like Facebook do

Is Facebook Connect a phishing scam waiting to happen?

Tim — Wed, 14 Oct 2009 23:07:06 +0000

Two things happened today that inspired me to write this post tonight.

A brief back-and-forth on Twitter with @kaigani where I outlandishly claimed that Facebook Connect is a phishing scam waiting to happen
The warning of another Twitter scam that typically exploits the layman’s inability to spot a fake URL.

Facebook and Twitter both offer authentication services arguably known as “single sign-on”. Facebook Connect is a proprietary system, and Twitter offers a system based on the OAuth standard. These services do something quite marvellous – They allow you to authenticate with a another website without the third party ever seeing your password. What’s makes it even more handy is that you’re probably already signed in to these popular services, so you may not need to enter your password at all. The problem is when you do.

If the mother service decides you aren’t logged in, it will have to present you with a username/password prompt just as if you were entering the main site. Here’s an example Facebook Connect popup:

A complacent user is likely to fill in these credentials without checking whether this page belongs to Facebook. This is the classic Phishing model, and I would argue that it is made worse by the additional trust the user may place in this familiar system. Furthermore, some implementations present this dialogue in a overlay form where no address bar appears at all.

There are various lines of defence available to the user at this point, and they are all in the browser.

1. The URL
Most phishing scams use cleverly manipulated URLs that can easily trick an untrained eye. The fact is that the address bar and the URL are (from an end-user perspective) quite technical aspects of using the Internet. These “connect” dialogues are prone to this problem, and to make it easier for the phishing gangs they don’t have to recreate the whole home page, just one small window. Even for Internet professionals, an accurately copied design may provide little reason to glance at the address bar.

2. The SSL Certificate
In the unlikely event that hackers have infiltrated your ISP, you still have the server certificate to ensure the site is legit. Observant readers will notice that the above image does not show a secure page. This is a failing of the vendor and of Facebook. A secure page does exist for Facebook Connect [see below] but Facebook should not offer standard HTTP at all and in this example the vendor should have used the SSL version.

Twitter also fails to restrict their authentication screen exclusively to SSL. To make matters worse their SSL screen does not contain full identity information (see below). Many Twitter apps don’t use the SSL page, and in fact the application settings page for developers lists the OAuth service URLs as HTTP variants only.

Is this a technology problem, or a human problem?

These scams exploit ignorance and complacency – Two things that user-friendly web services like these can only perpetuate. All the cryptography magic and clever security models behind these services can’t actually prevent phishing scams, and as they become more common and more trusted, perhaps they just make phishing scams easier to pull off.

I’m not convinced these problems can be solved by technology; at least not by technology in the websites themselves. I think this can only be solved by something that sits between the user and the trap – For example: the web browser, your ISP, or the HTTP protocol itself.

Chrome and IE8 both offer a neat address bar feature where the host name is bolder than the rest of the URL making fake URLs much easier to spot;
Firefox has more obvious server certificate and identity information, makes more of song and dance about invalid certificates and shows the host name in the status bar;
Various browsers offer warnings of known scam URLs and no doubt many ISPs aid this effort

However, these features still require education and awareness. Above all, any solution requires the attention of the complacent masses who just want to get on with their life and click “OK” until they get what they want.

Twitter changes break TwitBlock [again]

Tim — Sat, 19 Sep 2009 19:43:58 +0000

Last week a another change to Twitter caused me problems with my personal project TwitBlock. For the impatient, see my Google groups post about it. (It didn’t go down very well).

If you’re a Twitter user, you’re probably familiar with this image:

It is/was the default profile image for users that have not uploaded a custom avatar. You may also have noticed last week that Twitter has introduced a new version. Actually they they made seven of them in different colours:

At least I think they made seven; I can’t find any more, but I can’t find any official document stating how many are out there either.

So what?

So … TwitBlock crawls Twitter for duplicate profile pics to help identify spam accounts. The app needs to know what images are the default ones, because otherwise it will penalize people heavily for having what appears to be the same image as thousands of other people.

This relies rather delicately on factors that are liable to change and that aren’t strictly a part of the developer API, so I have to keep a close eye on things. I concede this is not a very robust solution, and I certainly wouldn’t base a commercial product around such weak “technology”. In fact I’m not sure I’d base a commercial product around Twitter at all.

I get a lot of emails and DMs from people telling me that they’ve received errors using TwitBlock. Almost always this is due to the Twitter API failing to respond – either timing out or sending back some HTTP error. It’s quite embarrassing, and I can only imagine how much worse this would be if people were paying for a Twitter-based service.

A Twitter app doesn’t just rely on the API, it relies on everything that makes up the Twitter service. This includes its full feature set and its hardware infrastructure. I am of the opinion that the above-described profile image change was significant enough that Twitter should have documented the change in advance. Facebook do a good job of addressing the community far in advance of changes, and I think this is yet another indicator that Twitter is out of its depth.

Diluting Block Counts

Tim — Mon, 24 Aug 2009 21:42:02 +0000

I made a major change to TwitBlock the other night. The change was made to protect people who are heavily blocked, but are not “spam”. Of course that depends on your definition. (A topic for another day)

Originally each block on account would yield 10 points. Then I became aware of just how murky this issue is. Barack Obama is blocked by many accounts (Republicans no doubt) plus some people with extreme right wing views were being blocked heavily. Then the complaints started. People whose businesses survive on a huge Twitter following accused me of destroying their reputations, and generating further blocks on their account by showing the number of existing blocks.

So now two things have changed for the time being:
1. Clicks on “not spam” are deducted from blocks;
2. Blocks are diluted by the size of a user’s following. 10 points are added for every 1%. So, if you’re blocked by 40 people, but are followed by 8,000 this will only yield 5 points.

Although this has stemmed the complaints, the scanner is less aggressive and lots of real spam accounts are not showing up with high enough scores. I am struggling to find the balance in the face of all of this and may have to tweak it again.

TwitBlock trialling whitelist feature

Tim — Fri, 21 Aug 2009 23:50:27 +0000

- or – “I told you it was in Alpha”

I’ve rolled out an experimental TwitBlock feature designed to reduce “false positives” for legitimate accounts that are being blocked. Whitelist entries are now subtracted from blocks. i.e. accounts marked as “not spam” will have their blocks counteracted on a 1:1 basis. If this feature is abused, it will be removed. It survives on the premise that the spam bots are not capable of whitelisting each other.

Here’s the full story:

As well as trying to work on TwitBlock in my “spare” time, I’ve also been manning the Customer Service department (i.e. Twitter) and the Press Office (with the help of my personal press officer @adamvincenzini). Monitoring a Twitter search for TwitBlock shows that most people are pleased with the service. Amongst the tweets there is some valuable feedback and feature requests, but also quite a few vocal complaints, mostly directed at me personally.

The number one complaint is that legitimate accounts are getting spam scores due to being blocked. In relative terms an account with a lot of blocks is more likely to be spam than an account with a few or none. But in reality people get blocked for various reasons – sometimes out of animosity, whether for their political or religious views, or just because they don’t like the person. Worst of all, and somewhat ironic, is that TwitBlock encourages blocking – that’s its MO – and I have been worrying that this may aggravate the situation, especially if people are too trigger happy and accept the spam scores blindly.

One of TwitBlock’s competitors has been arguing that blocks are a poor indicator of spam, and I think they have a point. I’d supply a link to said competitor except for the fact that they are a commercial enterprise. (TwitBlock is not a business, a spam-free life should be free).

So every time you click “not spam” on an account this will be used to counter every person that clicked “block”. This is an experiment, because it could be abused. That’s just the nature of what we’re doing here. Try it out, I look forward to more quality feedback.

Top 20 Faces of Twitter Spam

Tim — Tue, 18 Aug 2009 23:06:43 +0000

As we approach 3,000 TwitBlock users, we know of over 100,000 blocks and have stored 20,000 profile pic checksums. I figured it was time to start crunching some numbers.

The first of many reports shows the top 20 most duplicated avatars that we know about.

Many spam accounts use identical avatars across hundreds of accounts. TwitBlock uses this fact as an indicator of a likely spam account. This report just shows the top 20 that we’ve identified, but there are many more.

This indicator is one of the best ways Twitter could prevent spam accounts from signing up in the first place. Clearly bots have been developed that continually generate new accounts and Twitter does not seem able to prevent this despite the most prolific accounts displaying such identical properties. With a tiny 0.01% of Twitter accounts authenticated with TwitBlock one can only imagine how many of these accounts are out there.

Identical profile pics on Twitter

Tim — Sun, 09 Aug 2009 23:15:45 +0000

The list of Twitter accounts below all have something in common – They all have an identical profile image, which looks like this:

At the time of writing none of these accounts have been suspended. Whether they are breaking any laws or not I don’t know, but it is clearly a syndicate whichever way you look at it. The profiles all point to a Korean-registered “Cash generator” website, which [I would hazard a guess] is a con.

TwitBlock unearthed this statistic from a list of ~~only 18,000~~ 100,000 blocked accounts provided by under ~~400~~ 3,000 TwitBlock users . When you consider the size and growth of Twitter, you can well imagine that there are far more than ~~120~~ 288 profiles in this syndicate. You also have to wonder how much of Twitter’s growth figures can be attributed to this junk.

[ UPDATE: 18 Aug ]
Many of these accounts have been suspended, but TwitBlock is discovering new ones each day – currently 248 accounts known with this image.

[ UPDATE 19 Aug ]
I’ve produced a report of the top 20 most duplicated profile pics identified by TwitBlock

Warning: Do not sign up, or give any of your details to the organizations operating these Twitter accounts. I am publishing them only to exemplify the problem of Twitter spam. I am not responsible for any interaction you have with them, which unless you are insane, should be none.

TwitBlock spam ratings explained

Tim — Mon, 03 Aug 2009 22:12:13 +0000

A detailed explanation of the scoring mechanism used by TwitBlock.

Some people have complained that they get a high spam score and point out that they are not spammers. There are a number of important things to note about this.

This software is in alpha – these indicators and the scoring mechanisms attached to them will change.
As the system gathers data it will rely less on heuristics and more on cross-referencing (e.g. how many people have blocked an account)
Some of these tests are only indicators of automation, not specifically of malicious behaviour.
The spam rating has no limit – Scoring 40 may be high for a “legimate” account, but in a list with real spammers scoring 300+ you’ll be way down the bottom.
If you display characteristics of a spammer then perhaps this amounts to the same thing as being a spammer. Most normal users score zero.

Roughly in order of accuracy, here are the 8 tests currently performed in the standard TwitBlock scan.

1. Ignore factor.

This could also be called “inverse popularity”. If you follow 200 people and only 50 follow you back your ignore factor is 75%. Whether or not these 50 are the same people you follow is not analysed. The cut-off for scoring is 50%. Every 1% above 50 currently yields one point.

This simple and easily calculable factor is quite accurate because it reflects real human behaviour that can be observed. An account that is clearly spam, such as an “adult” account will have many times less followers than friends.

Naturally some spammers have found ways to beat this indicator. In some cases spam accounts follow each other to build up numbers, but a more cunning technique is the “sleeper” approach. Sleeper accounts pose as real people using stolen tweets pulled from the public timeline. TwitBlock may eventually crawl Twitter looking for these accounts, so expect more about this in future posts.

2. Follow Rate

The average number of people you follow per day forms your follow rate. This is calculated as the number of people you follow divided by the number of days you’ve been on Twitter. Although it’s a crude average, it is very telling and probably the second most reliable heuristic indicator. Even if you occasionally add a hundred people in a day it’s unlikely you can keep this up, so your average will drop. Averages are generally low even for power users, so a higher value is a strong indication of automation. The current cut-off (considered normal) is 10 per day. A point is added for every follower per day above 10.

[UPDATE - Aug 12]
Many popular accounts have high follow rates due to a “following back” policy, whether automated or not. The rate at which an account is followed is now subtracted from this value. This may result in lowering spam scores of real spammers, but it also reduces the number of false positives. So now, the rate at which an account follows without reciprocation is known as the “Stalking rate”.

3. Blocked by others

When you log into TwitBlock the system has access to your blocks and currently refreshes this list once per day until you revoke your authorization of the app. This is a key indicator that will become much more interesting as TwitBlock gathers data. Currently 10 5 points are applied for each block on an account.

[ UPDATE - Aug 22 ]
Whitelisting now used to counteract blocks

[ UPDATE - Aug 24]
Blocks are now diluted by follower count

4. Identical profile pics

Spammers commonly reuse the same image on multiple accounts. This is particularly common with the “adult” accounts. TwitBlock crawls all the blocked accounts it knows about and stores an MD5 checksum of the profile image file. This way any account’s profile image can be cross-reference with this database. 10 points are applied for each account known to use the same image.

This test could be easily foiled by spammers. Even using the same photo, it would be trivial alter the checksum. So far however, they appear not to be doing so.

5. Tweets via API

Status updates that are submitted without using a registered application (e.g. TweetDeck) will appear as having come “from API” (See Twitter FAQ). This is very useful, because spammers don’t want their activity to be tied to a registered application. If they start to do so then a list of known spammer applications will have to be compiled. 10 points are applied for API updates, although only the most recent tweet is analysed for performance reasons.

The points applied are deliberately low because people often give their password to applications that tweet on their behalf. e.g. “I just signed up to this awesome app and got 1,000 new followers”. This practice seriously needs to die out, but that’s another blog post for another day. Additionally many spam tweets appear as “from Web”, which suggest they are using the public web interface.

6. Missing profile info

This is not a very reliable indicator and may be dropped. There are 4 profile fields that can be left empty: (Bio, Location, URL and profile image). Most legitimate users fill in at least two of these. Currently 2 points are added if you leave all 4 empty. A drop in the ocean compared with other indicators. As I write this I realise this is due a review.

7. Username looks dodgy

For a human this is a strong indicator, but incredibly hard to implement programmatically. Currently this test performs some very crude tests on the username, such as being all numbers, having no vowels, and checking for a common format used by spammers where two words are followed by a number. Further research is required in this area, but it’s unlikely to form a reliable indicator going forward because it’s so easy to fool. 10 5 points are applied for a username that looks randomly generated.

8. Spammy words in bio and status

This more traditional test merely checks the bio against a list of bad words. The word list needs development and is currently not big enough to be useful. I intend to use the known blocked accounts to build a list of most common words found in spam accounts. An arbitrary score per word found is currently applied. For example “Naughty videos” yields 10 points.

Stay tuned for updates, as all these indicators are likely to change.