Web 2.1 » spam

Beating noisy Twitter apps

Tim — Sun, 29 Nov 2009 12:25:51 +0000

I woke up this morning to the apparent viral spread of the TweetCloud app that unoriginally, but very nicely displays your most tweeted words of the year, or month, or .. you get the idea. Here’s mine ->

If you’re impatient, you may wish to skip to the good bit.

Preamble

Now, how did this app manage such spread when there are so many like it? Possibly because it tweets from your account when your results are ready. This is not uncommon and it can be a nice feature that I might recommend. With the difference that it should be a 100% opt-in feature. TweetCloud’s start button says “make and tweet cloud“, so it does warn you. But people don’t read – they click.

TweetCloud insists that you log in before you can use it. It uses OAuth for this which is good (+1 point). Doing this means it can make calls to the Twitter API within your hourly request limit, rather than exhaust its own. (useful if you’re not whitelisted). But the real reason you must authenticate with TweetCloud is so that it can update your status. When building an app you have to seriously justify asking the user to authenticate/register etc.. As a general rule, the user should see that this action is for their benefit, not yours.

Good examples of this done right would be:

TwitPic, which has a genuine use for tweeting on your behalf.
Canabalt, a game where you want to share your score for social kudos.

Both of these apps make the tweet opt-in each time.

The good bit

While TweetCloud was busy generating the cloud (which took a minute or so) I dived off to my Twitter settings and revoked the permission I had granted the app. If you don’t know how to do this, it’s under “settings > connections”, or here: http://twitter.com/account/connections

As soon as you revoke this permission the app can no longer use the access key that it has obtained. It needs this for any API call that must be authenticated. e.g. getting your public timeline of tweets does not require authentication, whereas updating your status does.

Interestingly the cloud generation continued to churn away. This suggests that the app was actually paging through my timeline without even using authentication. i.e. making public API calls under its own rate limit.

Lo and behold, upon completion there was no tweet from my account.

I also decided to post my cloud as a TwitPic, just to say … well, you know. TwitPic doesn’t use OAuth, which it should, but that’s another post.

A few other things to note about “connections”:

When you grant access to an app, it can store its access key forever. i.e. Twitter don’t provide a key expiry feature like Facebook do. So you should revoke permissions from any app that you’ve stopped using.
My statistics from TwitBlock suggest that about 1% of people actually do this. (about 400 of 30,000 users have revoked my key)
Signing out of Twitter does not prevent the app using this access. An app with an access key can tweet from your account whenever it wants until you revoke
The read/write permission you can see is set by the app, not by you. Twitter doesn’t offer granular permissions like Facebook do

TwitBlock trialling whitelist feature

Tim — Fri, 21 Aug 2009 23:50:27 +0000

- or – “I told you it was in Alpha”

I’ve rolled out an experimental TwitBlock feature designed to reduce “false positives” for legitimate accounts that are being blocked. Whitelist entries are now subtracted from blocks. i.e. accounts marked as “not spam” will have their blocks counteracted on a 1:1 basis. If this feature is abused, it will be removed. It survives on the premise that the spam bots are not capable of whitelisting each other.

Here’s the full story:

As well as trying to work on TwitBlock in my “spare” time, I’ve also been manning the Customer Service department (i.e. Twitter) and the Press Office (with the help of my personal press officer @adamvincenzini). Monitoring a Twitter search for TwitBlock shows that most people are pleased with the service. Amongst the tweets there is some valuable feedback and feature requests, but also quite a few vocal complaints, mostly directed at me personally.

The number one complaint is that legitimate accounts are getting spam scores due to being blocked. In relative terms an account with a lot of blocks is more likely to be spam than an account with a few or none. But in reality people get blocked for various reasons – sometimes out of animosity, whether for their political or religious views, or just because they don’t like the person. Worst of all, and somewhat ironic, is that TwitBlock encourages blocking – that’s its MO – and I have been worrying that this may aggravate the situation, especially if people are too trigger happy and accept the spam scores blindly.

One of TwitBlock’s competitors has been arguing that blocks are a poor indicator of spam, and I think they have a point. I’d supply a link to said competitor except for the fact that they are a commercial enterprise. (TwitBlock is not a business, a spam-free life should be free).

So every time you click “not spam” on an account this will be used to counter every person that clicked “block”. This is an experiment, because it could be abused. That’s just the nature of what we’re doing here. Try it out, I look forward to more quality feedback.

Top 20 Faces of Twitter Spam

Tim — Tue, 18 Aug 2009 23:06:43 +0000

As we approach 3,000 TwitBlock users, we know of over 100,000 blocks and have stored 20,000 profile pic checksums. I figured it was time to start crunching some numbers.

The first of many reports shows the top 20 most duplicated avatars that we know about.

Many spam accounts use identical avatars across hundreds of accounts. TwitBlock uses this fact as an indicator of a likely spam account. This report just shows the top 20 that we’ve identified, but there are many more.

This indicator is one of the best ways Twitter could prevent spam accounts from signing up in the first place. Clearly bots have been developed that continually generate new accounts and Twitter does not seem able to prevent this despite the most prolific accounts displaying such identical properties. With a tiny 0.01% of Twitter accounts authenticated with TwitBlock one can only imagine how many of these accounts are out there.

Identical profile pics on Twitter

Tim — Sun, 09 Aug 2009 23:15:45 +0000

The list of Twitter accounts below all have something in common – They all have an identical profile image, which looks like this:

At the time of writing none of these accounts have been suspended. Whether they are breaking any laws or not I don’t know, but it is clearly a syndicate whichever way you look at it. The profiles all point to a Korean-registered “Cash generator” website, which [I would hazard a guess] is a con.

TwitBlock unearthed this statistic from a list of ~~only 18,000~~ 100,000 blocked accounts provided by under ~~400~~ 3,000 TwitBlock users . When you consider the size and growth of Twitter, you can well imagine that there are far more than ~~120~~ 288 profiles in this syndicate. You also have to wonder how much of Twitter’s growth figures can be attributed to this junk.

[ UPDATE: 18 Aug ]
Many of these accounts have been suspended, but TwitBlock is discovering new ones each day – currently 248 accounts known with this image.

[ UPDATE 19 Aug ]
I’ve produced a report of the top 20 most duplicated profile pics identified by TwitBlock

Warning: Do not sign up, or give any of your details to the organizations operating these Twitter accounts. I am publishing them only to exemplify the problem of Twitter spam. I am not responsible for any interaction you have with them, which unless you are insane, should be none.

TwitBlock spam ratings explained

Tim — Mon, 03 Aug 2009 22:12:13 +0000

A detailed explanation of the scoring mechanism used by TwitBlock.

Some people have complained that they get a high spam score and point out that they are not spammers. There are a number of important things to note about this.

This software is in alpha – these indicators and the scoring mechanisms attached to them will change.
As the system gathers data it will rely less on heuristics and more on cross-referencing (e.g. how many people have blocked an account)
Some of these tests are only indicators of automation, not specifically of malicious behaviour.
The spam rating has no limit – Scoring 40 may be high for a “legimate” account, but in a list with real spammers scoring 300+ you’ll be way down the bottom.
If you display characteristics of a spammer then perhaps this amounts to the same thing as being a spammer. Most normal users score zero.

Roughly in order of accuracy, here are the 8 tests currently performed in the standard TwitBlock scan.

1. Ignore factor.

This could also be called “inverse popularity”. If you follow 200 people and only 50 follow you back your ignore factor is 75%. Whether or not these 50 are the same people you follow is not analysed. The cut-off for scoring is 50%. Every 1% above 50 currently yields one point.

This simple and easily calculable factor is quite accurate because it reflects real human behaviour that can be observed. An account that is clearly spam, such as an “adult” account will have many times less followers than friends.

Naturally some spammers have found ways to beat this indicator. In some cases spam accounts follow each other to build up numbers, but a more cunning technique is the “sleeper” approach. Sleeper accounts pose as real people using stolen tweets pulled from the public timeline. TwitBlock may eventually crawl Twitter looking for these accounts, so expect more about this in future posts.

2. Follow Rate

The average number of people you follow per day forms your follow rate. This is calculated as the number of people you follow divided by the number of days you’ve been on Twitter. Although it’s a crude average, it is very telling and probably the second most reliable heuristic indicator. Even if you occasionally add a hundred people in a day it’s unlikely you can keep this up, so your average will drop. Averages are generally low even for power users, so a higher value is a strong indication of automation. The current cut-off (considered normal) is 10 per day. A point is added for every follower per day above 10.

[UPDATE - Aug 12]
Many popular accounts have high follow rates due to a “following back” policy, whether automated or not. The rate at which an account is followed is now subtracted from this value. This may result in lowering spam scores of real spammers, but it also reduces the number of false positives. So now, the rate at which an account follows without reciprocation is known as the “Stalking rate”.

3. Blocked by others

When you log into TwitBlock the system has access to your blocks and currently refreshes this list once per day until you revoke your authorization of the app. This is a key indicator that will become much more interesting as TwitBlock gathers data. Currently 10 5 points are applied for each block on an account.

[ UPDATE - Aug 22 ]
Whitelisting now used to counteract blocks

[ UPDATE - Aug 24]
Blocks are now diluted by follower count

4. Identical profile pics

Spammers commonly reuse the same image on multiple accounts. This is particularly common with the “adult” accounts. TwitBlock crawls all the blocked accounts it knows about and stores an MD5 checksum of the profile image file. This way any account’s profile image can be cross-reference with this database. 10 points are applied for each account known to use the same image.

This test could be easily foiled by spammers. Even using the same photo, it would be trivial alter the checksum. So far however, they appear not to be doing so.

5. Tweets via API

Status updates that are submitted without using a registered application (e.g. TweetDeck) will appear as having come “from API” (See Twitter FAQ). This is very useful, because spammers don’t want their activity to be tied to a registered application. If they start to do so then a list of known spammer applications will have to be compiled. 10 points are applied for API updates, although only the most recent tweet is analysed for performance reasons.

The points applied are deliberately low because people often give their password to applications that tweet on their behalf. e.g. “I just signed up to this awesome app and got 1,000 new followers”. This practice seriously needs to die out, but that’s another blog post for another day. Additionally many spam tweets appear as “from Web”, which suggest they are using the public web interface.

6. Missing profile info

This is not a very reliable indicator and may be dropped. There are 4 profile fields that can be left empty: (Bio, Location, URL and profile image). Most legitimate users fill in at least two of these. Currently 2 points are added if you leave all 4 empty. A drop in the ocean compared with other indicators. As I write this I realise this is due a review.

7. Username looks dodgy

For a human this is a strong indicator, but incredibly hard to implement programmatically. Currently this test performs some very crude tests on the username, such as being all numbers, having no vowels, and checking for a common format used by spammers where two words are followed by a number. Further research is required in this area, but it’s unlikely to form a reliable indicator going forward because it’s so easy to fool. 10 5 points are applied for a username that looks randomly generated.

8. Spammy words in bio and status

This more traditional test merely checks the bio against a list of bad words. The word list needs development and is currently not big enough to be useful. I intend to use the known blocked accounts to build a list of most common words found in spam accounts. An arbitrary score per word found is currently applied. For example “Naughty videos” yields 10 points.

Stay tuned for updates, as all these indicators are likely to change.

TwitBlock is born

Tim — Mon, 27 Jul 2009 22:36:04 +0000

A bulk blocking and spam filter tool for Twitter

www.twitblock.org

I’ve finally got round to building the Twitter app I’ve been thinking about for months. While everyone else is preoccupied with making fun, or cool apps, I’ve been thinking about the increasing problem of spam and junk followers on Twitter. I won’t go into why I think this is such a problem right now, plenty of time for that later.

This is just a quick announcement to say that I’ve released an early alpha version of a tool that I hope to develop into something genuinely useful. Currently it’s a simple scanner that analyses your followers for signs of “spammy” behaviour. I’ll post more details about these indicators soon, and I’ll also share some of the interesting discoveries I’ve been making about Twitter spam as I go on my mission.

UPDATE: I have posted about these indicators

Data mining for good, not evil

One of the principal aims of TwitBlock is to gather data in order to improve the service – i.e. to make it accurate enough that it could [in theory] be used to automatically filter spam out like an email junk filter endeavours.

By logging into TwitBlock (via Twitter OAuth of course) you are sharing the list of people that you block. As long as the app is authorized I can update this list and the app can learn from it.

Additionally I will be writing various bots (crawlers) that analyse Twitter activity in terms of suspicious behaviour and mine more data. More about these bots later too :)

Job board spam

Tim — Thu, 03 Jul 2008 09:20:58 +0000

I do marvel at the ingenuity of spammers sometimes. Despite being crap programmers, they do have a knack for coming up with new ways to deliver their poisonous junk. It’s almost enough to make me consider life of crime.

After placing a job advert on Gumtree, we (my current employer and I) received a touching email from someone intersted in our junior developer role. Clicking their portfolio link took us to a spammer’s “search” portal complete with gambling site pop-ups.

We hate spam like you do?

Tim — Thu, 27 Mar 2008 09:30:38 +0000

I just became aware of an apparently legitimate US-based company who I shall not provide a link to;
[whois guard] [dot] [com] – operated by [name cheap] [dot] [com].

Their opening gambit “We hate spam like you do” is somewhat ironic when you consider that their services are of enormous help to cyber criminals such as phishing gangs. These ‘people’ need to operate domain names, but they must remain untraceable. Protecting their whois data is an obvious step towards concealing their identity. I am not suggesting that companies offering such services are corrupt, rather that it highlights the dichotomy of the internet privacy problem.

I am not just musing – I identified a Facebook phishing attack this morning, (separate post shortly), and this is where they were hiding:
[view][hyphen][facebook profiles][dot][com]

Only NameCheap Inc of Los Angeles will know what country these criminals operate from. Their company contact details are available online, so why not give them a ring and ask them.

We don’t like the idea that the police can turn up at our office and demand all our confidential client details, but if the FBI turn up at NameCheap’s office because one of their clients is blatantly an organized crime gang, what do we think then of our privacy ideals?