If you don't already know that I'm a huge cynic, then you will do shortly. I'm going to lay out my observations as factually as I can, but they will be tainted with my usual dose of suspicion, fear and resentment. Below is a list of feature creep that I've observed, but there is an underlying point. If you don't want to read the list, just skip to the bit at the end.

Account verification via mobile phone

I thought I'd start with this one, because it erks me the most. My personal account has long since been verified. i.e. Facebook is satisfied that I am a real person, and not a robot.  If you aren't verified you must pass a CAPTCHA test for any significant activity such as posting, or friend-adding. This isn't new, but what seems to be new is that the only option for verifying that you are human now seems to be a SMS-based security check.

What erks me about this is that the CAPTCHA itself is the human/robot test – the mobile phone check is not proof of life; it is in fact little better than an email-based method which just proves an email address exists; it doesn't prove that there's a person at the end of it. I question Facebook's motivation here. The upshot of this is that if you don't give Facebook your mobile number you will be badgered with CAPTCHAs until you get so annoyed you verify. It also suggests they put a lower value on your email address. (Project Titan anyone?)

The "username" (vanity URL) feature is also denied to you if you do not verify. I particularly like the prompt to try another time.

iPhone address book feature

Continuing the mobile phone number theme: The superb Facebook iPhone app recently added a new feature which allows you to add your Facebook friends' profile pictures to the corresponding entries in your phone's address book. Before you enable this feature, you must make this fabulous clickwrap agreement (see image):

Now, don't get me wrong – I'm not suggesting that Facebook are doing evil things with your friend's numbers, I'm sure they really do need the phone numbers to automate this feature. I assume it's the only unique identifier that could associate an address book entry with a Facebook profile and a manual process would ask too much of the user. But regardless of the technical reasons, this is still an example of the increasingly prevalent ‘features for data' trade we are becoming more comfortable with. It can only make us more complacent about our own privacy.

"Tim W is no longer in a relationship"

This was actually the first of my recent observations. Facebook used to have user settings that allowed you to prevent certain events from being published as ‘news stories'. The example of relationship status is pertinent in that it is so personal. I actually went though a ‘Facebook breakup' in 2008, but had the publishing of this story disabled. I also had the "X is now friends with Y" story disabled (for reasons I can't be bothered to go into), but the point is that these options have disappeared. Why? Because Facebook want more activity not less, and that's no secret.

The real-time shift accelerated by Twitter at al is to blame for this. Much of Facebook's UX tweaks in 2009 were blatantly geared towards encouraging more chatter, more sharing, more data. Again we're encouraged to publish more activity, but in this example it's been achieved through denying us the right to keep it private. In my view, that's sneaky.

"friends-of-friends"

This relatively new privacy option took me by surprise. Certain rather innocuous settings, such as showing the "Add as a friend" button have had their tightest option reduced to friends-of-friends. This seems reasonable, but with 300 friends each having 300 of their own, we're talking about 90,000 people. The chance of a colleague, employer, or client being amongst that is high – In my case it's guaranteed. Fortunately the most sensitive settings, such as photo albums, still have much tighter options, but the option itself is still a nudge toward publishing more data to more people.

The bit at the end

I must point out that I'm not accusing Facebook of any evil-doing, or breaking any laws. These examples are here to illustrate what I think is a clear direction in our use of the free (as in beer) web, and our relationships with companies like Facebook and Google where our activity has become both product and payment.

The greatest threat to our privacy is our own complacency over it. We want features, we want them for free, and we're increasing willing to hand over whatever data is required to access them. What worries me is not what companies are doing with this data now, and not even what they might do with it in future; what worries me is how this creep is discreetly changing our behaviour such that we [as a society] no longer even care about our privacy.

We need to keep an eye on the direction in which we're being nudged and keep an eye on the organisations that are doing the nudging. Geolocation is clearly the next thing for us to get complacent about, and personal data doesn't get much more personal than your physical location. The recent Please Rob Me site, however tongue-in-cheek, should be enough for even the least educated to sit up and pay attention to the potential dangers of publishing your location.

What's the new feature?

Essentially it's Facebook's version of forwarding content through the network, as retweeting is to Twitter. I can't find an official announcement of this feature. Mashable and All Facebook have referred to it as the "via" feature, or the "Facebook retweet". Both ugly terms. I prefer "repost", although "reshare" would be more inline with the Facebook lexicon. To cut a long story short, it lets you share with your friends something that a friend of yours has shared with you. At first glance this may seem like it circumvents the walled garden of your friend network. i.e. people you don't want seeing your content seeing your content. gasp.

The repost privacy test

From my personal account (where I am simply known as Tim W and have every privacy feature locked down its strictest setting) I posted a link with appropriately alarming status update, as follows:

Then from my professional account (which is 100% public having the loosest privacy settings) I temporarily befriended myself and naturally saw the post from the elusive Tim W in its full glory. Nothing unexpected so far. I clicked the "share" button and reposted this link for all my friends on this profile to see.

Now I needed a third account that wasn't friends with Tim W, but as a Facebook member could access the second profile in full. So I set up a Facebook account in the imaginative name of Mit Kcoltihw (Polish I imagine) and accessed the page of my public, professional profile. Lo and behold there was the reposted link, as below:

( ignore, if you can, the fact that my profile photos are the same. The repost doesn't show the original poster as it does in Twitter)

What we see is that the posted link, and its meta data are visible, but my status update is missing. A non-friend seeing your link even if you set "Posts by me" to "Only friends" looks rather like a breach. It seems Facebook doesn't consider this content as being "by you" as it came from a public source. However, if you change the link description text to your own text when posting the original link, this is also visible, and that's definitely a loophole.

Links are one thing, but photos are a bit more serious. I tried the same test with a wall photo post, and you'll be relieved to hear that the repost was completely invisible to third parties. Good.

So in conclusion – not so black and white. Your privacy settings are maintained to an extent, and you don't need to worry about status updates and photos getting into the wrong hands, but there is definitely creep here. I think the  "posts by me" privacy setting is now misleading and needs addressing, and the reposting of custom link descriptions is highly questionable. I won't lose any sleep over this one, but I won't be getting complacent any time soon either. If you've spotted a loop hole let me know by commenting below, but double check your privacy settings first.

More in my next post about Facebook privacy creep.

Firstly, the debate around this movement is a total mess. People seem confused about what they are supposedly protesting against.

• It clearly isn't rejecting commercialism, or the record industry, because it involves buying a record. In fact, as the purchasers of ‘Killing in the Name' probably wouldn't have purchased Joe McElderry's track in the first place, this protest most likely generated more record sales than the industry would have otherwise.
• It clearly isn't rejecting crowd mentality either, because the whole notion is a collective endeavor, and arguably worse, because I doubt all the participants even like the music of RATM.

The intial protagonists of the campaign never suggested it protested against any of these things. It was simply to prevent Simon Cowell's "kareoke act reaching number 1" as a "protest to the X-factor monotony". As Simon Cowell himself pointed out, it was a direct attack on him and the content he is responsible for producing. And yes, I think that content is tripe too. Cowell may well be spoon-feeding the next genertion a self-perpetuating culture of celebrity-obsessed drivel, but this protest is just worthless.

• The X-factor monotony doesn't bother me [most of the time], because I avoid it. I don't watch Cowell's TV programmes, I don't read trashy tabloid entertainment columns, and I don't listen to commercial radio. Easy.
• If you like the music of RATM then the UK pop charts are probably not your cup of tea. People who follow the pop charts take enjoyment from it. Why ruin it for them, just because you don't like their music?
• I have not been aware of a Christmas No.1 since I was at school. The cultural meme of the Christmas No.1 has no impact or relevance to my life, and I go to no effort to achieve this. This year has obviously ruined that.
• Getting RATM to number 1 surely didn't mean less sales to Cowell, because they are clearly two separate groups buying the records
• Who is supposed to take notice of this protest? It won't stop Cowell pumping out drivel and it won't stop people liking it.
• It seems only to have served to generate a load of press, social media noise, and a bunch of extra record sales for Epic Records who have signed other such subversive musical acts as Jessica Simpson and Shakira.

If you're impatient, you may wish to skip to the good bit.

Preamble

Now, how did this app manage such spread when there are so many like it? Possibly because it tweets from your account when your results are ready. This is not uncommon and it can be a nice feature that I might recommend. With the difference that it should be a 100% opt-in feature. TweetCloud's start button says "make and tweet cloud", so it does warn you. But people don't read – they click.

TweetCloud insists that you log in before you can use it. It uses OAuth for this which is good (+1 point). Doing this means it can make calls to the Twitter API within your hourly request limit, rather than exhaust its own. (useful if you're not whitelisted). But the real reason you must authenticate with TweetCloud is so that it can update your status. When building an app you have to seriously justify asking the user to authenticate/register etc.. As a general rule, the user should see that this action is for their benefit, not yours.

Good examples of this done right would be:

• TwitPic, which has a genuine use for tweeting on your behalf.
• Canabalt, a game where you want to share your score for social kudos.

Both of these apps make the tweet opt-in each time.

The good bit

While TweetCloud was busy generating the cloud (which took a minute or so) I dived off to my Twitter settings and revoked the permission I had granted the app. If you don't know how to do this, it's under "settings > connections", or here: http://twitter.com/account/connections

As soon as you revoke this permission the app can no longer use the access key that it has obtained. It needs this for any API call that must be authenticated. e.g. getting your public timeline of tweets does not require authentication, whereas updating your status does.

Interestingly the cloud generation continued to churn away. This suggests that the app was actually paging through my timeline without even using authentication. i.e. making public API calls under its own rate limit.

Lo and behold, upon completion there was no tweet from my account.

I also decided to post my cloud as a TwitPic, just to say … well, you know.  TwitPic doesn't use OAuth, which it should, but that's another post.

A few other things to note about "connections":

• When you grant access to an app, it can store its access key forever. i.e. Twitter don't provide a key expiry feature like Facebook do. So you should revoke permissions from any app that you've stopped using.
• My statistics from TwitBlock suggest that about 1% of people actually do this. (about 400 of 30,000 users have revoked my key)
• Signing out of Twitter does not prevent the app using this access. An app with an access key can tweet from your account whenever it wants until you revoke
• The read/write permission you can see is set by the app, not by you. Twitter doesn't offer granular permissions like Facebook do

• Download jParser 1.0.0 (recommended)
  
• Download jParser devel package (Full source and build scripts)
• See the library examples running at timwhitlock.info/jparser

The library has been split in two:

1. jTokenizer – A JavaScript tokenizer designed to mimic the PHP tokenizer.
2. jParser - The fully blown JavaScript syntactical parser which generates a parse tree.

The reason for the split is that for most purposes where you think you need a parser, you in fact just need a tokenizer. The tokenizer library is about 15KB, whereas the parser is over 700KB (minified), so you can see why you might not want to include it unnecessarily.

The library files jparser.php and jtokenizer.php are self-contained, minified files for production use. If you wish to inspect or modify the code you will need to download the devel package. This package provides a build script which collates the libraries into their distributable files.

jTokenizer

Possible uses for the tokenizer include code highlighting and simple manipulation of JavaScript source code.

The main function you will want to use is j_token_get_all which behaves the same as the PHP token_get_all function with the addition of a column number as well as a line number. Additionally there is the j_token_name as per the PHP token_name function.

jParser

This is a full, syntactical parser. On its own it simply generates a parse tree which can be traversed and manipulated. There is no proper documentation on this yet, but take a look at the node classes in the devel package if you are serious about doing something useful with this parser.

Some other notes in no particular order

The full parser uses a lot of juice. I recommend giving PHP loads of memory, and be careful what you throw at it if you're going to run it on a production server.

A parser is not an interpreter or a JavaScript engine. If you want to develop such a thing in PHP you might be insane, but it could be done with this parser as a base.

The JParser parse tree is purposefully not a full tree, it collapses redundant nodes to save memory. If you want to see a full tree then take a look at the JParserRaw class. (devel package required)

Splitting the parsing process into two parts (tokenize/parse) is probably not the most efficient and probably uses more memory than it would another way. However, I figured it would be neat to mimic the PHP tokenizer functionality so that parsers could be built that take a stream of PHP tokens.

After scrounging myself a B-list Google Wave preview, I've been playing around with it for a week or so. Rather than read more and think deeply about it, I thought I'd blurt out my half-formed opinions now. In fact, this is one of those posts I'll probably regret in a year's time. It might look as naive as some of my early thoughts on Twitter when I didn't quite get it, but that's blogging for you… so here goes.

Comparison with existing paradigms

When a new service comes along, it's normal to bring along preconceptions from services you're already comfortable with. New Twitterers tend to bring their Facebook habits, because they're pretty similar right? Well it's natural anyway; how else can you understand something so abstract? Look no further than email which is not much different from sending a letter in the post. This probably accounts for its widespread adoption by even the least technical of people.

The situation with Google Wave at the time of writing is that nobody really knows what they're doing. There are no hardened Wavers to learn from, everyone is a n00b. So how do we use it? If I wrote this blog in a wave you could edit it, so it's a wiki? It looks more like a discussion forum, but there's an inbox, so it's more like email? One of the KSPs is that the whole thing is real-time, so it's a chat room?

The way I described Google Wave to Public was "MSN on steroids has a love child with Facebook". I will undoubtedly regret this statement before long, if not already. The fact is that these preconceptions are not that useful. They may get us off to a flying start, but they constrain us to habits which may mask the format's real potential. First and foremost, it's a real-time medium, but it brings with it some associations that we may do well to shake off.

Google Wave as chat client

Of all the comparisons we might make, its most common adoption initially seems to be chat – Most of us are just using it like MSN. It's far too easy to do this because the real-time element is so compelling. However, as soon as you fall into this trap you find things that break the model. How do you know when the person is online? How do you end a conversation when it stays in your "inbox" forever? These may only be superficial features that need to be addressed, but maybe not – maybe we're just missing the point – it's just not a chat room and we just need to get over that.

[ Update: 19 Oct ]
As of this morning I see that there is now an online status indicator.

Google Wave as email

We are well used to the idea of sending messages that won't get an immediate response. The etiquette of email may not be perfect, but at least it's universally understood as an asynchronous exchange. You send an email in the same way you send a letter – it leaves your control and your copy is literally that. Eventually you may receive a new message.

Google Wave has a kind of inbox, although this familiar term is less useful when you realise it's also an outbox. The process of starting a wave can appear similar to email – you add the addressee[s] and fire it off for their attention. As with email, your recipients may not be online but that's not a requirement for sending an email; you don't even need to know.

Perhaps the idea of basing electronic mail so closely on paper-based mail was as temporary solution that is well out of date and well overdue a successor. Google must want to fix this – Certainly Microsoft tried to fix email in the form of Microsoft Exchange, but that's another blog post and I don't want to get upset right now.

Google Wave as living document

A Wave need not be correspondence at all; it can just be a document. You can just publish it, whatever that means. Maybe you'll add collaborators later when you're ready. Maybe you'll make it public and tag it with keywords. So in this sense it is more like a wiki, but let's stop with the comparisons – Ultimately it's what ever you want it to be – in Google's own words:

"Google Wave is an online communication and collaboration tool that makes real-time interactions more seamless — in one place, you can communicate and collaborate using richly formatted text, photos, videos, maps, and more."

The real-time shift

The biggest issue that underpins conversation around Google Wave is the web's apparent evolution from archive to a real-time medium. However, I think even this distinction is something we have to get over. There's a reason we no longer refer to the Internet as the "information super highway" … we know it's full of information, so we don't need to bang on about it all the time. Eventually the novelty of real-time will be as transparent as it is in the real world. Your consciousness and your memory are not mutually exclusive – you kinda need them both.

I think Google Wave will play a big part in continuing this blend of time and data, much like Twitter has started to do. It's no secret that Google Search is not so hot when it comes to up-to-the minute data. They have certainly made improvements in this area, but their search index is fundamentally an archive.

The most exciting changes Google Wave brings about probably won't be in the Wave client that we're currently mucking about with – I can't wait to see what developers start doing with gadgets, and bots and most importantly the underlying Wave protocol, but that's another post.

1. A brief back-and-forth on Twitter with @kaigani where I outlandishly claimed that Facebook Connect is a phishing scam waiting to happen
2. The warning of another Twitter scam that typically exploits the layman's inability to spot a fake URL.

Facebook and Twitter both offer authentication services arguably known as "single sign-on". Facebook Connect is a proprietary system, and Twitter offers a system based on the OAuth standard. These services do something quite marvellous – They allow you to authenticate with a another website without the third party ever seeing your password. What's makes it even more handy is that you're probably already signed in to these popular services, so you may not need to enter your password at all. The problem is when you do.

If the mother service decides you aren't logged in, it will have to present you with a username/password prompt just as if you were entering the main site. Here's an example Facebook Connect popup:

A complacent user is likely to fill in these credentials without checking whether this page belongs to Facebook. This is the classic Phishing model, and I would argue that it is made worse by the additional trust the user may place in this familiar system. Furthermore, some implementations present this dialogue in a overlay form where no address bar appears at all.

There are various lines of defence available to the user at this point, and they are all in the browser.

1. The URL
Most phishing scams use cleverly manipulated URLs that can easily trick an untrained eye. The fact is that the address bar and the URL are (from an end-user perspective) quite technical aspects of using the Internet. These "connect" dialogues are prone to this problem, and to make it easier for the phishing gangs they don't have to recreate the whole home page,  just one small window. Even for Internet professionals, an accurately copied design may provide little reason to glance at the address bar.

2. The SSL Certificate
In the unlikely event that hackers have infiltrated your ISP, you still have the server certificate to ensure the site is legit. Observant readers will notice that the above image does not show a secure page. This is a failing of the vendor and of Facebook. A secure page does exist for Facebook Connect [see below] but Facebook should not offer standard HTTP at all and in this example the vendor should have used the SSL version.

Twitter also fails to restrict their authentication screen exclusively to SSL. To make matters worse their SSL screen does not contain full identity information (see below). Many Twitter apps don't use the SSL page, and in fact the application settings page for developers lists the OAuth service URLs as HTTP variants only.

Is this a technology problem, or a human problem?

These scams exploit ignorance and complacency – Two things that user-friendly web services like these can only perpetuate. All the cryptography magic and clever security models behind these services can't actually prevent phishing scams, and as they become more common and more trusted, perhaps they just make phishing scams easier to pull off.

I'm not convinced these problems can be solved by technology; at least not by technology in the websites themselves.  I think this can only be solved by something that sits between the user and the trap – For example: the web browser, your ISP, or the HTTP protocol itself.

• Chrome and IE8 both offer a neat address bar feature where the host name is bolder than the rest of the URL making fake URLs much easier to spot;
• Firefox has more obvious server certificate and identity information, makes more of song and dance about invalid certificates and shows the host name in the status bar;
• Various browsers offer warnings of known scam URLs and no doubt many ISPs aid this effort

However, these features still require education and awareness. Above all, any solution requires the attention of the complacent masses who just want to get on with their life and click "OK" until they get what they want.

If you're a Twitter user, you're probably familiar with this image:

It is/was the default profile image for users that have not uploaded a custom avatar. You may also have noticed last week that Twitter has introduced a new version. Actually they they made seven of them in different colours:

At least I think they made seven; I can't find any more, but I can't find any official document stating how many are out there either.

So what?

So … TwitBlock crawls Twitter for duplicate profile pics to help identify spam accounts. The app needs to know what images are the default ones, because otherwise it will penalize people heavily for having what appears to be the same image as thousands of other people.

This relies rather delicately on factors that are liable to change and that aren't strictly a part of the developer API, so I have to keep a close eye on things. I concede this is not a very robust solution, and I certainly wouldn't base a commercial product around such weak "technology". In fact I'm not sure I'd base a commercial product around Twitter at all.

I get a lot of emails and DMs from people telling me that they've received errors using TwitBlock. Almost always this is due to the Twitter API failing to respond – either timing out or sending back some HTTP error. It's quite embarrassing, and I can only imagine how much worse this would be if people were paying for a Twitter-based service.

A Twitter app doesn't just rely on the API, it relies on everything that makes up the Twitter service. This includes its full feature set and its hardware infrastructure. I am of the opinion that the above-described profile image change was significant enough that Twitter should have documented the change in advance. Facebook do a good job of addressing the community far in advance of changes, and I think this is yet another indicator that Twitter is out of its depth.

These bots are usually easy to spot because their tweets all show as being "From API", meaning that the update wasn't sent by a registered app using OAuth. If I was a spammer, I'd be wanting to fix that because it's a dead give away. I've also seen other services such as HelloTxt being used by these bots, but just now I spotted something new. – Tweets from Tweetie.

This example (@margy6727) was flagged in a TwitBlock scan due to aggressive following and being largely ignored. This search result shows the original tweet below, and the one above is the bot cloning the tweet and updating somehow via Tweetie.

Now how have they done this?
Have they reverse engineered one of the Tweetie apps and obtained the API key and secret to use in their own bots? Have they developed something that can automate Tweetie externally? (AppleScript was suggested by @pepijndevo). Do they have a sweatshop of chinese teenagers cut-and-pasting for 50 cents/hr? Perhaps there's some loophole on the Twitter end? Twitter aren't exactly buttoned up on the security front and they clearly don't check new app registrations very rigorously.

Suggestions welcome.

[ UPDATE ]
This Twitter FAQ is misleading. I am informed by Tweetie that Basic Auth apps registered "pre-OAuth" can indeed still pass the [now] discontinued source parameter to spoof the user-agent. I just tried it and all you have to do to make your tweet appear to have come from Tweetie is add "source=tweetie" to your API call. That's a real shame.

It' s great, here's a pic of the iPhone interface (courtesy of @pepijndevos)

WPTouch iPhone screen grab