I had a quick go with the W3 Total Cache WordPress plug-in, but it seems rather biased to running Apache (which I’m not) and I experienced some strange errors that I failed to immediately fix. Rather than wrestle with it, or try other WordPress caches, I decided to get to grips with Varnish. This is something I’d been meaning to do for ages, and of course it isn’t limited to WordPress – Varnish is a fabulous caching solution for whatever site you’re building.

The challenge with deploying Varnish to any site is to ensure you cache as much as you can while ensuring dynamic content is fresh when required. With WordPress as an example, the cases are fairly obvious. While logged in to the admin screens you don’t want to cache anything, but the outside world reading your posts should get cached pages whenever possible. In their simplest forms, these cases almost work out of the box with Varnish, but there are a few gotchas. There is also scope to cache more aggressively if you choose to, and most importantly you’ll want to ensure that when content is altered, the changes are visible to everyone immediately.

Cookie gotcha no.1

By default, if you send a cookie to Varnish it assumes quite sensibly that you are in some way expecting dynamic content from the back end. e.g. you may be an admin user, logged in to the site. Of course Varnish can’t actually know what the cookie is for, so when your browser sends a completely redundant Google Analytics UTM cookie, you’ll miss the cache. On this site, that means Varnish would only serve you a maximum of one cached page per visit.

Anonymous session cookies

If a visitor comments on a WordPress site, they are pseudo-logged-in by way of a simple cookie exchange. This means the site can do niceties like pre-populate comment forms for commenting again. These niceties are exactly that, so if you overlook this functionality and block those cookies you can serve more cached pages. In my case, I am happy to make this trade.

The following snippet from my configuration shows how I’ve maximised anonymous user caching while ensuring admin users always get dynamic content

sub vcl_recv {
  # ...
  # admin users always miss the cache
  if( req.url ~ "^/wp-(login|admin)" || req.http.Cookie ~ "wordpress_logged_in_" ){
    return (pass);
  }
  # ignore any other cookies
  unset req.http.Cookie;
  return (lookup);
}

Purging when content changes

HTTP defines a wealth of cache control headers that allow applications and clients to specify what is to be cached and for how long, etc.. but there is no standard way to tell a cache that certain content is to be purged. This is the main challenge of deploying Varnish to a site like WordPress.  A post may be updated, or a user may comment. You’ll want to purge any affected pages from the cache immediately, regardless of their original expiry. Putting that business logic into your VCL configuration would be hugely impractical – it’s really your application’s jurisdiction.

I looked at various purging techniques, and by far the best approach in my opinion is for your application to talk directly to the Varnish administration CLI which listens on the network on a different port to the cache itself. As long as your backend can access your front end by a secure, internal address, you can execute commands on all your Varnish front ends any time you want from your application.

To achieve this, I first had a look at the wp-varnish plug-in which uses the HTTP PURGE method. I don’t like this approach at all; it’s very limited, too closely coupled to the VCL config, and also this plug-in doesn’t purge all pages relating to changes, such as tag and archive indexes.

So, I rolled my own

I started by writing a VarnishAdminSocket class, which talks to Varnish from PHP. This was pretty easy to do, because the varnishadm program uses a very basic text-based protocol. This isn’t designed for WordPress, this is designed for use in any PHP application.

My php-varnish toolkit is on Github. It includes a WordPress plug-in which I am using on this site and you can see the VCL configuration I am using too.

I have a like/hate relationship with WordPress. That is to say that it does a lot, it has a great admin area and there’s a large community producing themes and plug-ins. However, I am a PHP developer of many years, and every time I come into direct contact with the core WordPress code-base I end up being sick in my mouth.

What’s to hate? Well it kind of reminds me of that old BMW advert (or was it Audi?) the one with the swan – On the surface it glides effortlessly, but under the water it’s an ugly son of a bitch, churning away to get the job done.  I had the misfortune of working on a nasty WPMU project last year and found myself looking under the bonnet more than was good for me. Not only is WordPress a horribly written piece of PHP, (and possibly responsible for many a bad word said about PHP itself) it has a legacy of incremental improvements alongside a legacy of community-contributed code that strives to remain as reverse compatible as possible.

If you don’t want to read my rantings, skip to the bit at the end, otherwise … WordPress, let me count the ways a hate thee.

1. Base URL hell

I wanted to get as much of my static assets onto a CDN as I could – a topic I’ll cover in full later. This means serving static files (such as images and css) from a different base URL. It should be simple to upload your theme and plug-in directories to another server and ensure that WordPress points to the correct assets in your HTML. It is not simple. Just look at the hideousness of determining base URLs in WordPress. This simple task should not require a complex plug-in to cover all possible legacy issues. Furthermore, WordPress allows all plug-ins to mess around with base URLs, which means one plug-in might completely undo the work of another.

Oh, by the way – WordPress defines the constant WP_CONTENT_URL before it loads any plug-ins, so the only way to override this is to define it in wp-config.php.

2. Canonical domain ridiculousness

If you access a WordPress site from a domain that is not hard-wired into the site’s database, it will redirect you to the ‘correct’ URL. This is the default behaviour. So, if you want to host the same exact site from a different domain, or different port, (e.g. for debugging a distributed system) you will have to write a WordPress plug-in to override this behaviour. Why does this annoy me?

1. It is not PHP’s job to decide what domain a site lives on – that is the job of the web server. A domain redirect like this should be done at server config level.
2. No system should hard-wire its domain into database-saved preferences. This is nuts, and WordPress does it in multiple places.

3. Magic quotes are the Devil\\\’s work

Magic quotes are so utterly evil that they have been removed from PHP 5.3. They are testament to the amateur reputation of PHP. However, even in earlier versions of PHP, magic quotes could be easily disabled, and anyone who knows what they’re doing with PHP will have been doing so  for years. Furthermore any system written properly would insist on them being disabled. Not so with WordPress. In fact, being able to disable magic quotes poses a problem for WordPress. How can it guarantee to work if you can forcibly disable magic quotes?

Answer: By forcing them even more forcibly. WordPress 3 forces all script input through its own wp_magic_quotes function. Funnily enough this isn’t documented, but it looks like this. Rather than mess around reversing this process, I wrote this simple utility for accessing the unharmed postdata.

4. mysqli support – or not.

The core WordPress code-base uses the old mysql database driver. When I say old, consider that the mysqli (improved) extension was introduced for versions of MySQL > 4.1.3  in PHP 5, which came out of beta in 2004. WordPress does have a drop-in replacement system though, which allows you to place your own db.php file in wp-content. Here’s the third-party-developed mysqli adapter for WordPress that I am using on this site.

So that’s okay then, right? Well, judging by the required location for the db.php file, this adapter system looks rather like an after-thought. Using the alternative database adapter means calling require_wp_db() rather than just including the original wp-includes/wp-db.php file. Great – except is every plug-in out there doing this? Even WordPress itself isn’t fully patched to support this. At the time of writing, the “famous 5-minute installer” uses the mysql adapter only – I had to patch wp-admin/install.php to be able to install WordPress with only mysqli on my system.

5. Depreciated code handling

Watch out for depreciated function names. For example: WordPress changed wp_specialchars to esc_html. This will result in notices being raised in older themes, which you’ll only see if you enable WP_DEBUG. It’s a good idea to find-and-replace these if you’re upgrading an old theme to WP3. I don’t really hate WordPress for this, but it seems a bit of a pointless change, and isn’t it a bit late in the day for being pedantic about naming conventions? A whim like this just contributes to the amount of legacy in the system and the amount of code devoted to handling depreciation. There’s a 2,500 line file devoted to depreciating functions that are to be removed “later”. Making a clean break with WordPress 3 would have prevented many people from upgrading, so I understand why they do this; I just don’t like the effect it has on the code-base.

I could go on, but I’m hungry. If you want more, see what this guy has to say on the topic.

The bit at the end

All in all, I’m disappointed that WordPress 3 wasn’t taken as an opportunity to improve upon its own legacy. It is fairly clear that popularity with amateurs, ‘ease of use’ and backward compatibility is more important to the team than creating good, robust software that developers will appreciate. Drupal sets a much better example. I’m not saying I don’t have any gripes with Drupal, but they are much happier to take reduced backward compatibility on the chin when they release new versions. This means more incompatible modules, but such is the price of progress.

PHP 5.3 with FastCGI Process Manager

Running PHP as a FastCGI (or even a regular CGI) means that it’s decoupled from the web server, and if you’ve historically run mod_php, then this is something you’ll have to get used to. For example, restarting your web server won’t refresh your PHP settings. The CGI listens on a port much like the web server itself does, and that’s how the web server passes requests though to PHP. There’s plenty of information on why this is good news, like here.

The nginx wiki has plenty of information and examples of running PHP as a FastCGI, but I had some bad experiences using these examples. Quite possibly this was due to my tuning it badly, but regardless, discovering PHP-FPM was a major result. Furthermore FPM is now bundled with PHP 5.3.3 – I thoroughly recommend, if you can upgrade to PHP 5.3.x that you do so. If you can’t, and you still want to use PHP-FPM, it’s a lot more effort to install and you may have to roll back your version of PHP 5.2.x – I won’t be covering that here, check the various patches here instead.

There doesn’t seem to be a huge amount of information about configuring PHP-FPM in PHP 5.3.3, so I’ll share my configs here. To start with building php-fpm from the downloaded PHP sources is as simple enabling the option. The rest of the build process is as normal, but you’ll have a php-fpm binary installed at /usr/local/sbin/php-fpm

$ ./configure --enable-fpm

The following is a single nginx server block, which defines a virtual host capable of executing PHP via FastCGI.

server {
  listen       80;
  server_name  *.2point1.com 2point1.com;
  root         /home/vhosts/2point1.com/httpdocs;
  access_log   /home/vhosts/2point1.com/log/access.log main;
  error_log    /home/vhosts/2point1.com/log/error.log  warn;
  # Process PHP files with fastcgi
  location ~ \.php$ {
    if ( !-f $request_filename ) {
      return 404;
    }
    include /etc/nginx/fastcgi_params;
    fastcgi_pass   127.0.0.1:9000;
    fastcgi_index  index.php;
    fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
  }
  location / {
    index  index.php;
  }
}

Notice the inclusion of /etc/nginx/fastcgi_params. This file may be bundled with your nginx installation already. If not, it looks like this:

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
#fastcgi_param  REDIRECT_STATUS    200;

I also added the following lines, on the recommendation I found here (in fact this is probably what went wrong with my earlier experiments with FPM under 5.2.x). I recommend adding these lines.

fastcgi_connect_timeout 60;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
fastcgi_buffer_size 128k;
fastcgi_buffers 4 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_intercept_errors on;

That’s got nginx ready to pass PHP execution onto port 9000, so now we have to get PHP-FPM up and running on that port. PHP-FPM has it’s own configuration, which as of this new release has shifted from an XML format to the familiar ini style. This is a great move, firstly because it looks nicer, and secondly because it also allows us to add additional PHP ini settings.

My main php.ini and php-fpm.conf files are under /usr/local/etc which is the default location. The php-fpm.conf file contains global settings and settings for individual resource pools. Each pool listens on its own port, and can have its own PHP settings.

If you need to support multiple hosts as I do, then you’ll probably want each nginx server to pass PHP requests on to a different pool, specified by its port. To achieve this, I removed all resource pools from the main config and added a wild-card inclusion much like I did with the nginx config. My main php-fpm.conf file looks like this:

[global]

pid = /usr/local/var/run/php-fpm.pid
error_log = /usr/local/var/log/php-fpm.log
log_level = notice
emergency_restart_threshold = 0
emergency_restart_interval = 0
process_control_timeout = 0
daemonize = yes

;  pools defined in virtual hosts
include=/home/vhosts/*/conf/php-fpm.include

Then each host has its own config in php-fpm.include as follows.

[mypool]

listen = 127.0.0.1:9000
listen.backlog = -1
listen.allowed_clients = 127.0.0.1

; Unix user/group of processes
user = nginx
group = nginx

; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 50
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 10
pm.max_requests = 100

; Pass environment variables
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

; host-specific php ini settings here
php_admin_value[open_basedir] = /home/vhosts/2point1.com/httpdocs:/tmp

Any directives missing here were just left as default. Note that I set the user to nginx, although I don’t know whether this is a good idea, or not. This is possibly just a throw-back to my mod_php days where PHP would always run as the apache user.

Process manager tuning

There’s not much documentation on the pm.* directives, namely max_children, start_servers, min_spare_servers, max_spare_servers, and max_requests. However, they are analogous to other settings you may be familiar with. The well documented Apache directives,  MaxClients, StartServers, MinSpareServers, MaxSpareServers and MaxRequestsPerChild are worth a look to make sense of these options.

Once a process has been used it still occupies memory, even when idle, so you don’t want more of these processes running than your server can handle. I kept my spares low, and my max children under a number I was confident the server would have enough memory to cope with. This is not a particular area of expertise for me, so I suggest doing your own research.

To start up PHP, run the program telling it where your configs are:

# php-fpm -c /usr/local/etc  -y /usr/local/etc/php-fpm.conf

If you need to reload any configuations, you can do a graceful reload will the kill command, as follows taking the process id from disk:

# cat /usr/local/var/run/php-fpm.pid | xargs kill -s USR2

And that’s it. One speedy new nginx server powered by PHP 5.3.3. My next task was getting WordPress 3 running in this environment – post to follow shortly.

Performance issues aren’t just for high traffic sites. I’m lucky if I get 50 visitors a day to this site, but by using scaling techniques popular with the big boys, I figured I could increase page load speeds, (good for visitors and good for SEO). If I could achieve this and use less resources, perhaps I could even save some money on my hosting bills. I currently run a 512MB VPS on Slicehost, and I’d rather not increase this right now.

With a few days off work, I decided to take the plunge and swap out some of the server tech powering this blog. From the bottom up, so to speak, this was as follows -

1. Replace Apache with Nginx (below)
2. Upgrade to PHP 5.3.3 and run as a FastCGI (next post)
3. Upgrade to WordPress 3
4. Deploy a CDN
5. Add a Varnish cache for extra speed

I’ll go through my experience across a number of posts, starting with Nginx. I shan’t replicate any existing documentation; I’m just going to go through what I did and point you at the resources you’ll need.

Nginx (Engine X)

Apache has long been my comfort zone, as I imagine it is for most PHP developers, but I’ve been listening to people over the past couple of years saying the old work horse is bloated, and needs shooting. It has certainly been falling out of favour with high-traffic sites, while Nginx seems to be gaining popularity. Before this upgrade I’d already tried Nginx out on a few other sites, and it’s great. Read about Nginx here. Here’s an Nginx-Apache comparison if you want some stats – I have none to offer you.

Installing nginx

I found I could install a fairly recent version on Fedora using yum, so # yum install nginx was all I needed to do. I found it was totally unavailable in other repos, including CentOS, so you may have to build from source. I won’t replicate any such tutorials here, except to say that you will most likely have to update libevent. You’ll need a recent version of that for the rest of this stack anyway, so may as well build that upfront.

Virtual hosts

As I have a couple of other small sites on this sever, I needed to set up virtual hosting. I did this with an include directive in the main /etc/nginx.conf, as follows:

http {
  # ....
  include /home/vhosts/*/conf/nginx.include;
}

So each of my host directories has a server {…} block defined in nginx.include. I actually removed the default server block from the main config, although that is entirely up to you. All nginx core directives listed here.

Starting nginx is as simple as running # nginx, and a graceful reload of altered configs can be done with # nginx -s reload. If you install via yum, you may also have a service script at /etc/init.d/nginx. To ensure nginx is up on reboot, I just chucked a start command into /etc/rc.local although, I’m sure there’s a better way to do that.

PHP

So, nginx is running, what to do about PHP? Waving goodbye to Apache also means waving goodbye to mod_php. If you’ve previously run PHP as a FastCGI, then you’re ahead of the game on this one. If not, then you’re about to leave another comfort zone. I’ll cover running PHP-FPM next »»

Anyhow, this isn’t the first time I’ve been annoyed by auto-tweeting, I’ve written about it before. It’s happened more times than I care to remember, so I won’t go into the details of today’s particular example, except to say that its author has [sort of]  apologised. It’s already been blogged anyway if you’re curious.

What I will harp on about though, is the fact that this is nothing short of spam – Twitter needs to agree, and needs to make it easier to report badly behaved apps.

Best practice guidelines for auto-tweeting

Best practice in my opinion is to at the very least provide a clear statement of intent, that by authorizing this application it will tweet from your account immediately. At best this should be an opt-in mechanic, or an after-the-fact sharing mechanic. Maintaining moral integrity when your app needs traction may be frustrating, but a good viral encourages users to spread the message because they want to; i.e. passing the message on provides value.

I checked Twitter’s growing API terms today to see if there are any guidelines around this specific issue. I couldn’t find anything. There are plenty of  guidelines around automation, but nothing about moral use of the statuses/update method when used to post a status update via another user’s account. I believe the use of this method should insist that if it is not invoked directly and knowingly by the owner of the account, that the owner must at least be been warned, and ideally given the opportunity to opt-out.

I posted a query about this on Quora to see if anyone else could find anything in black and white about this.

Defining Twitter spam

Twitter’s definition of spam doesn’t include this practice either. An application auto-tweeting without consent, or warning is using other people’s accounts to distribute bulk messages without their permission, and consequently without the recipients’ permission. That’s about as close to a definition of spam as I think you need to get.

However, despite Twitter’s maturing terms of service, they doesn’t seem to think the same. They do say that their definition of spam “will continue to evolve as [they] respond to new tactics”, but this tactic is not in the least bit new.

If I were a cynic (cough), I may even suggest that identifying more activity as spam would not be in the company’s interest …  statistically speaking.

Reporting applications

Twitter provide a way to report spam, (or you can do it through TwitBlock ) – but this is really for individual user accounts; not all applications have a corresponding Twitter account, and the registered author may just be doing as they’re told by a client, or employer. There is no specific way to report an application for misbehaviour. You can complain by way of a support ticket, but that’s not quite good enough. Facebook make reporting an application much easier. All applications have profile pages one click away from the Connect dialogue, and a report button, (albeit a discreet one) is available there. I want to see mandatory application profiles for Twitter, with clearer flagging and revoking facilities.

Write access and revocation

You can’t discuss this topic for long without the old issue of opt-in write access being raised. i.e. Should users be able to choose what privileges they grant an application? Personally, I think it should be made visually clearer (see Facebook), but if an app requires write access to function, I think it is reasonable that the user cannot concoct their own privilege cocktail. i.e. “This app requires write access to perform its primary function; if you don’t trust us, don’t use it”.

OAuth revocation is too well hidden on Twitter too. It may only be two clicks away from any part of the main UI, but most people don’t know it’s there. How do I know this? Firstly, by speaking to people who aren’t all developers, and secondly – of the 43,500 users who have authenticated on TwitBlock, only about a thousand have revoked their ‘connection’, that’s less than 3%.

While both these issues are important and have room for improvement, I don’t believe they alone can solve the problem of applications being spammy. That requires a better definition of what it means to be spammy, and a better way to report those applications.

JASPA is/was a cross-compiler written in PHP which converts an ActionScript3-like language to native JavaScript prototypes. Essentially this is a language abstraction; an approach to JavaScript which is has its critics. I felt however that with ActionScript being a derivative of ECMAScript that some such criticism was not quite applicable. It worked pretty well, and I used it some of my own work, but it has received little public interest, and I was unable to maintain it well enough for those who were interested. Thanks to all the people who did email me about the project – I may pick up the project again sometime, but I can’t make any promises.

My ultimate reason for physically bringing the site down was that was that it was powered by Mediawiki, which doesn’t appear to be compatible with PHP 5.3.x, and I wanted to upgrade so I could use the new version of PHP-FPM (post about that to follow). I could have debugged the wiki to see if 5.3.x was really the problem, but I decided instead to cut my losses. R.I.P.

–> Dictionary.com <– drag to toolbar

Dictionary.com provide an ‘official’ bookmarklet, here:
http://dictionary.reference.com/tools/bookmarklets.html

I’ve improved it a bit

• Selected text uses window.getSelection(), not document.getSelection()
• Trims junk out of current selection
• Opens a new window so you don’t lose the current page
• Uses encodeURIComponent(), not escape()

Here’s the uncompressed source on Github

I’ve been having fun playing with node.js over the past year, but have had little, or no excuse to use it in any production work, so I thought I’d set myself a challenge and build a module. That challenge was firstly to create a simple AMF gateway for Flash remoting, and secondarily to see if an RTMP socket server was achievable in node.

If you don’t know about “node” – It’s a JavaScript runtime that allows you to write socket servers. I like it a lot – it brings asynchronous, event-driven programming to the server side and provides a truly global variable scope across all connections. I’ll blog about it in more detail later, perhaps.

At Public we do a lot of Flash work, and regularly implement Flash remoting using a PHP AMF gateway. I wasn’t necessarily looking to replace this stock approach with node, but node offers proper socket connections that PHP can’t, so I was imagining the possibilities of using node as a free, and more flexible alternative to Flash Media Server. Not for streaming media, but for real-time messaging, for example in multi-player games. If I’m honest though, I did this mostly for fun, an academic exercise and as an excuse to work with node.

node-amf

node-amf is on GitHub as a public repository: http://github.com/timwhitlock/node-amf

The first step was to write a pure JavaScript AMF implementation. i.e. a library for serializing and deserializing AMF packets. If you’re not familiar with AMF (Action Message Format) – it’s a binary serialization and messaging format invented by Adobe for passing data back and forth between Flash and a remote server. I am not the only person to implement this in JavaScript: mid-way through my project I noticed amf.js appear – just a little too late for me. This library is probably much better than my own, but mine is designed specifically for node, so I shall stick with it for now.

With a working AMF library all that remained was to create a HTTP gateway. There are many examples of AMF gateways in other languages, including PHP. The general approach is that a single request/response exchange carries one or more messages, each one calling a web service on the server and returning the result. AMF also has the ability to invoke methods in the Flash client on response – an underused feature.

Node makes it incredibly easy to implement an HTTP server, all that I had to do was decide how to expose the web services to the gateway. I decided to do this by passing a user-defined JavaScript object when initializing the server – each property of the object is a function callable by name – example AMF gateway here. This suitably sandboxes the method calls, and ensures the client cannot execute arbitrary JavaScript functions – that would be bad.

This part of the project is largely complete and working. It has not been used in production yet, so if you’re brave enough to use it, please let me know how it’s going!

node-rtmp

node-rtmp is currently under the same project as node-amf because it’s dependant upon the AMF library. This incomplete, highly unstable, and experimental work is under the node-rtmp subdirectory with examples here.

If you’re not familiar with RTMP (Real Time Messaging Protocol) it provides bi-directional streaming media and messaging over persistent socket connections. It’s the messaging I’m interested in. Flash talks to Flash Media Server over RTMP sockets and can securely call methods in both directions.

As it turns out the RTMP specification is very badly written. (The AMF specs were quite easy to read, although possibly much simpler). If I was a cynic, I may even suggest that Adobe has purposefully written them badly to avoid third party developers being able to adhere to the usage license which insists in adequate conformance to the spec. However, it doesn’t seem to have stopped numerous projects such as librtmp from gluing the pieces together.

This is the point where I realised I might be out of my depth. I am currently in the process of glueing the pieces together myself. Armed with the dodgy specification, Wireshark, and the fragmented information dotted around the Internet, I am making some slow progress. At the time of writing I have the RTMP handshake working correctly, and have just about deciphered the command messaging packets.

I may give up before this is at all useful.

librtmp and node add-ons

As you may have noticed, I’m talking about implementing all of this in pure JavaScript, and you may be thinking that’s nuts. You are probably right. As libraries, these don’t really need to be written in JavaScript. In addition to pure JavaScript modules, node supports add-ons – compiled libraries which can be built against C and C++ system libraries. A good example of this is node-memcache.

C and C++ is unfortunately outside my skill set, but I imagine it may be possible to build an RTMP node add-on using librtmp. Anyone fancy a crack at that?

There is so much to discuss around this and it’s not even out of the lab yet. In a rare display of focus, I’ll devote my first post on the topic to one of the more obvious questions – Can they (or do they need to) get 400 million people to migrate away from Facebook?

The idea of a decentralized, open source social network where you truly own your data appeals to many a privacy-concerned geek, but I think perhaps the announcement of Diaspora and their rapid public funding is timely more than anything. After the F8 conference Facebook are predictably under the spotlight again – this time there’s even infoporn - See: Mat McKeon and the New York Times.

So we’re all ‘concerned’ about our privacy, and maybe even what Facebook are up to in general, but as Fernando Rizo muses on his blog today, are you going to quit? No, of course not. Well, not without a decent alternative, because you don’t want to miss out. (See FOMO). Well let’s assume for a moment that Diaspora becomes that alternative – what then?

Tipping the other way

In theory I don’t see a reason the Network Effect can’t work in reverse. It takes early adopters to populate a site like Facebook in the first place – perhaps a trend in rejection could result in a tipping point in the opposite direction. If you joined Facebook because your friends did, and they went somewhere else – you’d eventually go too. Somebody has to go first of course.

I grumble about Facebook all the time, but I use it as much as the next guy – in fact more than most of my friends. I don’t want to shut my account down. Going cold turkey would be a serious commitment. I think for this to happen for me there would have to be some kind of transitional phase.

If Diaspora allowed me to view and publish content to and from Facebook, that would surely defeat its primary function. You could argue that it depends what the content was, but it would still mean keeping my Facebook account active. It might however be a way to soften the blow, and at the same time entice my peers into migrating too.

I don’t have the solution, (and I probably don’t understand the problem), but many of us are far too attached to our digital homes for this to be a clean break. As Fernando points out we’ve seen mass migration before (away from MySpace) but I’d say it’s a bigger deal this time. I remember quitting MySpace (~2007) and I really didn’t miss it. I had a handful of photos and about 30 friends. It was also incredibly annoying. Despite my moaning, I really like Facebook, it’s a very usable site and there’s vastly more content than I had access to three years ago.

Would an exodus be necessary?

Diaspora are proposing a hosted, turn-key option for their software (a la WordPress) and perhaps, as is common with open source products, providers will be permitted to package up and sell the product themselves in a healthy, competitive fashion. To move 400 million people over to Diaspora, this would surely be essential – how many Facebook users know what a GPG key is?

I joked earlier (complete with typo) that if Diaspora took off, perhaps Facebook could move to a hosted-Diaspora revenue model. Perhaps this wasn’t such a joke. Facebook need your data to profit, if you’re going to abscond and not give them any more data and not look at any more ads, then a premium service where you can interact with your friends without getting ‘graphed’ seems reasonable to me. The privacy concerned few could pay, while the complacent masses continue to trade their personal lives for a free ticket.

I’m thinking out loud and probably sound like an idiot, but I’m hungry and need to go home…. just gotta check my Facebook.

You didn’t expect me to have only nice things to say, did you? There are a couple of things I have to question.

It’s only a draft

Despite this spec being a draft, Facebook (who are represented in the working group) have gone ahead and implemented it anyway. Although this is a step up from the non-standard methods they’ve employed to date, it does make me wonder. Will the spec be finalised according to their implementation? Will they change their implementation if the spec changes? Or will they end up going in separate directions? (think ECMAScript 4/ActionScript). As with my gripes about the Open Graph, how “open” are standards when we have self-interested corporations in the driving seat.

Looser security for JavaScript clients

The so-called “user_agent” journey serves the needs of front-end applications that don’t have access to a web server. (i.e. JavaScript only apps). This support comes at a cost to security because request signing is not required. (More to the point, signing would be redundant). The risk is a limited one – the “bearer tokens” must only be sent over SSL, so the worst you can do is take control of an app under the authentication of your own account. Still, I imagine it would be possible to post content that the app did not intend. (use your imagination!) My main gripe here is in justifying the trade off. The loosening of security is in favour of making apps easier to implement for more people – i.e. a Facebook business interest. I don’t think that’s a good enough reason to weaken the specification.