• Download jParser 1.0.0 (recommended)
  
• Download jParser devel package (Full source and build scripts)
• See the library examples running at timwhitlock.info/jparser

The library has been split in two:

1. jTokenizer – A JavaScript tokenizer designed to mimic the PHP tokenizer.
2. jParser - The fully blown JavaScript syntactical parser which generates a parse tree.

The reason for the split is that for most purposes where you think you need a parser, you in fact just need a tokenizer. The tokenizer library is about 15KB, whereas the parser is over 700KB (minified), so you can see why you might not want to include it unnecessarily.

The library files jparser.php and jtokenizer.php are self-contained, minified files for production use. If you wish to inspect or modify the code you will need to download the devel package. This package provides a build script which collates the libraries into their distributable files.

jTokenizer

Possible uses for the tokenizer include code highlighting and simple manipulation of JavaScript source code.

The main function you will want to use is j_token_get_all which behaves the same as the PHP token_get_all function with the addition of a column number as well as a line number. Additionally there is the j_token_name as per the PHP token_name function.

jParser

This is a full, syntactical parser. On its own it simply generates a parse tree which can be traversed and manipulated. There is no proper documentation on this yet, but take a look at the node classes in the devel package if you are serious about doing something useful with this parser.

Some other notes in no particular order

The full parser uses a lot of juice. I recommend giving PHP loads of memory, and be careful what you throw at it if you’re going to run it on a production server.

A parser is not an interpreter or a JavaScript engine. If you want to develop such a thing in PHP you might be insane, but it could be done with this parser as a base.

The JParser parse tree is purposefully not a full tree, it collapses redundant nodes to save memory. If you want to see a full tree then take a look at the JParserRaw class. (devel package required)

Splitting the parsing process into two parts (tokenize/parse) is probably not the most efficient and probably uses more memory than it would another way. However, I figured it would be neat to mimic the PHP tokenizer functionality so that parsers could be built that take a stream of PHP tokens.

↓ JavaScript grammar file for jParser

This file is in a (probably non-standard) variant of BNF notation. I’m not generating the tables with a tool like ANTLR primarily because I don’t write C. I should also point out that I don’t come from a formalized computer science background, so don’t expect this to be 100% conventional.

I’ve developed a native PHP parse table generator that in turn uses a parser (based on itself) to parse this BNF grammar into a table. If you understand grammar files like this you’ll notice something a bit odd – The terminal symbols don’t go right down to individual characters, the grammar expects the source code to already have been tokenized into significant chunks, such as J_NUMBER representing an already identified numeric token. This was done deliberately to be compatibile with PHP’s own Tokenizer functions. The underlying parser framework was designed such that PHP token based grammars could also be developed.

> Try it here: Obfuscate and minify your JavaScript code

Minifying

There are other minifiers out there;

[ Link to project removed due to incredibly rude email from its author ]

The parser framework behind my attempt allows a great deal more power at the expense of being a much heftier package. The source code is about 700k, so this is anything but light-weight. The extra power means that, unlike many minifiers, line breaks are not required at all. Any JavaScript program can be stripped down to one, very long line of code. The reason for this is that it performs automatic semicolon insertion. It can do this because rather than just performing a lexical scan of the source, it compiles the full syntax of the program into a parse tree. Of course, a program with a syntax error will not be minified, but then why would you want it to?

Obfuscation

Another advantage of the syntactical parsing is that we stand a better chance of safely altering identifier names. This still has serious limitations though, because the code is still not actually being executed there are many situations that could result in disaster. The current version obfuscates all explicitly named, top-level entities, such as function names, function arguments, labels, and variable declarations. Member expressions are particularly problematic, and so I have not attempted to obfuscate these. I may at some point in future work on improving this, but as this is only a side project, I am not going to hold my breath.

You can always take additional steps in your original code to better preprare for effective obfuscation. Consider this;

var myDocument = document;
var myElement = myDocument.getElementById('myId');

Try it and see

Code highlighting does not require the full syntactical parser, it just uses the tokenizer and does not break when a bad character is found.

What’s in?

When I say full Unicode support, what I mean is that Unicode characters inside string literals and comments were always implicitly supported, but now it can cope with Unicode characters in identifiers too, (as per the ECMAScript standard). Support also includes the use of Unicode escape sequences, although for the sake of speed, the validity of these is not checked. Checking could be done at a later stage when the parse tree is employed to do something useful. The full range of whitespace and line-terminating characters have also been added, although not tested.

Unicode support makes the tokenizing process slower. Because of this it may be switched off if it is not needed. Most programmers with English as their first language are unlikely to use Unicode characters in ‘hand-written’ identifiers, and I have to wonder if anyone has ever done so with an escape sequence.

There are many things that you can write in JavaScript that are perfectly valid syntax, but that you probably never will write. Here are a few that raised an eyebrow or two.

Comma Operator

What would you expect to see if you ran this code in your browser?

var test = [ 'a', 'b', 'c' ];
alert( test[ 0, 1, 2 ] );

You might expect it to be a syntax error, but in fact test[0,1,2] evaluates in this example to "c". The expressions ( 0, 1, and 2 ) are all evaluated, but only the final one can return the single value of the expression.

Similarly pointless constructs:

if( alert("Hello"), alert("World"), false ){
    alert("You will never see this");
}
var a = ( 1, 2, 3 ); // a will be set to 3

The “/” character

The syntax does not discriminate between operand types. For example, you can attempt to divide any expression by any other expression, even if it makes no sense to do so; such as:

['a'] / 2

This will evaluate to NaN, because the array’s value can never be a number, but it is perfectly valid syntax. The real point I’m getting to is that the following is an exception and will raise syntax error:

{ a:1 } / 2;

Why? Because the lexical analyser will expect "/" to be the start of a Regular Expression Literal, which it isn’t. It gets it ‘wrong’ in this case, because the "}" a is tricky so-and-so; it could either be the end of an expression, or the termination of a block statement. The lexical analyser is not a full syntax parser; it knows nothing of the full grammar of the language, and so it makes a choice based on some fairly weak rules.

The following IS valid, but it is not a division operation, it is just a pointless mess:

{ a:1 } / 2 /i;

undefined vs null

undefined is not a literal, or even a reserved word, whereas null is. undefined is a special built-in object, so if you were stupid enough, you could do this:

undefined = true;

This is allowed by the same token that allows you to foolishly do this:

Array = null;
var a = new Array();
// TypeError: Array is not a constructor

Section 7.9 of the ECMAScript language specification details the circumstances under which you are permitted to omit a semicolon in a statement which otherwise requires it. In short, there are situations where this lazy practice is not the end of the world; it is fairly clear what the programmer is trying to do, so it lets you off.

I’ve managed to implement most of these conditions, but not all. I still have some work to do on the more peculiar conditions, but the ones that are relevant to most common practice should be covered. Give it a bash, and let me know if you can break it!

If you don’t know what I’m talking about, consider this:

a = b
c = d

Most JavaScript programmers would be disciplined enough write:

a = b;
c = d;

However, there are some conditions that probably catch us all out.
What is the difference between this:

function xfunc(){
}

and this?:

var xfunc = function(){
}

The former is a Function Declaration, and does not require termination, but the latter is a Function Expression, which in this case forms an Expression Statement and so it requires a semicolon to terminate it. A very common ‘mistake’, but solved quite comfortably by the rules of Automatic Semicolon Insertion, which transform it into:

var xfunc = function(){
};

There are plenty more cases I could discuss, but if you’re really interested, I suggest reading the ECMA-262 specification yourself.

This article is rather old now – the jParser code has been released
–

Despite the glorious sunshine this week, my week off, I managed to put some time into my pet project of developing a full JavaScript parser written in 100% native PHP. Actually, I’ve been developing a generic parser suite for some time, and using it to build a full JavaScript parser was my ultimate goal to be satisfied that it all works and is powerful enough to be useful. I’ve written a bunch of blogs about developing a parser generator in PHP, (click “parsing” to do a tag search).

Before I start wittering on,
Click here to play with the online example of JParser

Here are the main difficulties I encountered while building the JavaScript parser:

1. Performance
Generating the parse table was taking about 30 minutes and using several hundred megabytes of memory. Going back to the drawing board with certain parts of the parse table generator, I’ve managed to get this down to about 7 minutes on my humble Mac Mini.

2. Special rules
The ECMAScript standard states certain special cases in the grammar rules. One of particular note (clause 12.4) says that an ExpressionStatement may not begin with a "{" or a "function". This special rule avoids ambiguity and therefore avoids parse table conflicts, but the rule is effectively outside of the grammar. I’ve finally found the right part of the parser architecture to implement such rules

3. Automatic semicolon insertion
As you probably know just from writing JavaScript, the ECMAScript standard permits the lazy omission of semicolons at the end of some statements, as long as you terminate with a line break instead. This is actually more complex than it sounds, but more to the point, it is another special rule that is not directly a part of the grammar and is handled at parse time.
[UPDATE: Automatic semicolon insertion now implemented, See! ]

If you haven’t read Part 1, or Part 2 they are there for the reading.

I’m going to demo a JSON parser in this post. It’s 100% native PHP code, and is based on the work I’ve done toward my ultimate goal of a full JavaScript parser.

Click here to play with the interactive JSONParser demo

I thought I’d get this example online now as my ultimate goal is taking longer than I had hoped. I shan’t go into the details, suffice to say that the JSON grammar below is a very tiny subset of the full JavaScript grammar and doesn’t really have any complex rules.

Here’s the JSON grammar I put together.

<JSON_OBJECT_LITERAL>
	: "{" "}"
	| "{" <JSON_PROP_LIST> "}"
	;

<JSON_PROP_LIST>
	: JSON_STRING_LITERAL ":" <JSON_LITERAL>
	| <JSON_PROP_LIST> "," JSON_STRING_LITERAL ":" <JSON_LITERAL>
	;

<JSON_LITERAL>
	: JSON_STRING_LITERAL
	| JSON_NUMERIC_LITERAL
	| <JSON_ARRAY_LITERAL>
	| <JSON_OBJECT_LITERAL>
	| "true"
	| "false"
	| "null"
	;

<JSON_ARRAY_LITERAL>
	: "[" "]"
	| "[" <JSON_ELEMENT_LIST> "]"
	;

<JSON_ELEMENT_LIST>
	: <JSON_LITERAL>
	| <JSON_ELEMENT_LIST> "," <JSON_LITERAL>
	;

The grammar notation of the full JavaScript language may only be about 12 times the size of the JSON grammar above, but the parse table it generates is hundreds of times bigger. The JSON parse table was generated in just a few milliseconds and the PHP source code for the table alone is about 3K. In comparison; my current JavaScript parse table generator takes about 7 minutes and the table source code is about 800k.

Anyhow, I digress. The purpose of showing off the JSON parser is to underline the usefulness of the parse tree, as I touched on in part 2, and of course to make it relevant to PHP :)

Parse node classes

Each node in the parse tree is assigned a different PHP class which extends a vanilla flavoured node class. You can manipulate these nodes much as you would with an XML or DOM tree. Most importantly you can write custom routines to evaluate them. When you evaluate the root node you begin a recursive procedure which ultimately gives you a value, or object that represents the whole structure. In this case an associative array which is the deserialized JSON object.

These nodes don’t need much code either. For example, the terminal symbol JSON_NUMERIC_LITERAL has a class assigned to it who’s evaluate method simply returns its string value as a native PHP number. The nodes for JSON_OBJECT_LITERAL and JSON_ARRAY_LITERAL are obviously a bit more complex, but I’m sure you get the idea. It doesn’t take much imagination to see that once the parser has given you a tree it’s very easy to hook in whatever logic you want.

This was only an academic exercise, particularly as PHP5 has a JSON extension enabled by default. The PHP json_decode function is much faster than my parser, and of course my parser is only one-directional, but it shows that if an extension doesn’t exist for what you want to parse, it is possible to write one in native PHP.

As is usual for me, this blog has very little direction and I am not sure what the topic of the next post will be, or if there was even a topic for this one. However, my goals for this body of work are clear and I look forward to demonstrating a fully working JavaScript parser some day soon. I will also release some code eventually – honest.

Syntactical vs Lexical

Looking again at the simple grammar of part 1. You may notice that the rule: <D_DIGIT> ::= "0" | "1" ... "9" is a bit different to all the others. It does not really contribute to the syntax of our language, it merely describes the legal characters that make up a single digit. It is convenient to view this aspect of the language as a subset of the grammar; one that is concerned only with what input ‘looks like’ rather than where it appears. This can be called the lexical grammar. The rest of the language which is concerned with syntax can be called the syntactical grammar.

If we were to run our date string through a lexical parser first, it could output a more organized set of symbols that we could then feed into our syntactical parser as input. The syntactical grammar could then safely take for granted that a symbol D_DIGIT is a perfectly valid terminal symbol and wouldn’t need to worry about what characters it might have contained. So the input for our simple date parser could be transformed from:

"1976-11-03 18:15:00"

- to something like:

[ D_DIGIT D_DIGIT D_DIGIT D_DIGIT "-" D_DIGIT D_DIGIT "-" D_DIGIT " " D_DIGIT D_DIGIT ":" D_DIGIT D_DIGIT ":" D_DIGIT D_DIGIT ]

And because the non-terminal symbol <D_DIGIT> has been scrapped in favour of the terminal symbol D_DIGIT, a grammar rule like :

<D_MONTH> ::= <D_DIGIT> <D_DIGIT>

- could become

<D_MONTH> ::= D_DIGIT D_DIGIT

This might seem a bit pointless at this stage. Having two grammars (and two parsers) for our simple date language is probably not necessary. One of the factors that makes our original test grammar so simple is that all of its terminal symbols are single characters, and only a restricted set of characters at that. This made it fairly easy to embed the lexical information within a single grammar. However, as we consider more complex languages this practice soon becomes a problem and slows things down immensely. Consequently, this separation of the two grammars becomes increasingly useful. Categorizing symbols together like this doesn’t just make our grammar neater, it also makes the parser hugely more efficient.

Tokenization

The lexical parsing process (called ‘lexing’, or ‘tokenizing’) can often be much simpler and easier to achieve than the full, syntactical parse. If we wanted to parse the PHP language itself we don’t have to worry about this step at all, because we have a built in tokenizer at our disposal – the PHP Tokenizer. There are other benefits to having a separate tokenizer too; Perhaps you want to parse a string, but you don’t need a full parse tree or even need to check the syntax. A good example of this is code highlighting.

It looks like our vocabulary is getting bigger, so let’s have a terminology catch-up as I start to talk about tokenizing in the context of PHP.

A ‘symbol‘, whether terminal or not, is always scalar. As with our date grammar a terminal symbol could represent a piece of input literally, like “2″, or as with PHP, it could stand for a type of input such as T_LNUMBER who’s value may be any numeric string. The latter type of symbol is represented internally as an integer constant. Non-terminal symbols never represent actual input, and so will always be integer constants.

A ‘token‘ represents a single piece of [pre-processed] input. It is identified by a terminal symbol, but it may contain more information. For a symbol like T_LNUMBER, an input token needs to contain both the symbol and a literal value. We use an array as the simplest way to express this:
array ( T_LNUMBER, "123" ); The literal value at offset 1 is not actually consulted during syntactical parsing but if you’re going to do anything useful with your parser, other than just check syntax, you’ll almost definitely need this data later on. It is of course also useful for debugging and error messages during parsing.

It made sense for me to base my parser’s notion of a token on the PHP tokenizer’s model, but really there are many ways to represent such concepts and it is largely a programming design issue. Hopefully by now you get the basic premise that tokenizing raw input into more manageable data makes life easier in performing a full, syntactical parse.

My parser package does not currently contain the kind of automated tokenizer that you’ll find in the world of [ahem] proper programming languages, (e.g. Lex for C), but this is in the pipeline. If you have to write your own tokenizer, you can go about it any way you like as long as what you end up with is an array of tokens supported by the parser. In theory you could build a lexical grammar with rules like <T_FUNCTION> ::= "f" "u" "n" "c" "t" "i" "o" "n", and use the same parser that your syntactical grammar will use, but this is a pretty bad idea. I have found it a hell of a lot easier and infinitely more efficient to write a bespoke function. It doesn’t have to be complex – here is an example of a possible tokenizer for our humble date grammar: The source and the output. PHP also provides the strtok function, but I have not found it useful.

Parsing PHP

In the case of PHP of course, we don’t even need to bother with all this, we just call token_get_all, and we’re ready to parse, so before this article gets too long, let’s create a mini grammar that can use the PHP Tokenizer, and then my parser as before. The following language merely adds up numbers, that’s it – that’s all it does.

1. <NT_PROGRAM> ::=
2.    T_OPEN_TAG <NT_SUM> |
3.    T_OPEN_TAG <NT_SUM> T_CLOSE_TAG;
4. <NT_SUM> ::=
5.    <NT_NUMBER> |
6.    <NT_NUMBER> "+" <NT_SUM>;
7. <NT_NUMBER> ::=
8.    T_LNUMBER |
9.    T_DNUMBER

This tiny subset of PHP syntax must start with <?php simply for the tokenizer to work, but the closing tag is optional. This is defined in the first rule on lines 1-3. The second rule on lines 4-6 is recursive. It defines that sums may be added together with further sums until ultimately a number is produced. The third rule on lines 7-9 defines that both integers and doubles are valid numbers in this language.

If we parse this source: "<?php 100 + 2.2 ?>", we get the following parse tree:

[009] <NT_PROGRAM> :
[000] .  "T_OPEN_TAG" = '<?php'
[007] .  <NT_SUM> :
[002] .  .  <NT_NUMBER> :
[001] .  .  .  "T_LNUMBER" = '100'
[002] .  .  </NT_NUMBER>
[003] .  .  "+"
[006] .  .  <NT_SUM> :
[005] .  .  .  <NT_NUMBER> :
[004] .  .  .  .  "T_DNUMBER" = '2.2'
[005] .  .  .  </NT_NUMBER>
[006] .  .  </NT_SUM>
[007] .  </NT_SUM>
[008] .  "T_CLOSE_TAG" = '?>'
[009] </NT_PROGRAM>

Click here to try the interactive version.

The structure of this parse tree shows that the <NT_SUM> nodes are nested recursively. This is a natural occurrence of the grammar rule on lines 4-6. It is simple enough to ‘flatten’ the tree if you needed, but ultimately this structure reflects the inherent nature of the language which might be essential for whatever you’re going to do with it next.

You’ll see that the interactive demo actually gives the result of the sum. This is achieved by evaluating the parse tree, which is when things start to get a bit more useful, and a lot more interesting. I’ll get to that topic in a other post to follow soon.

A rod for my back

But what if you want to parse something really complex, like say – an entire JavaScript program. What if you can’t find a third party library that works for you? Well I tried to find one. I found some very promising projects. But they ranged from abandoned projects, to dodgy alpha releases, to ones that just plain didn’t work and with no documentation to help. The most serious looking projects were so sophisticated that I didn’t even have the knowledge to start using them. I decided, as I often do, that I need empowering with the knowledge to write my own parser should I need one for – well, whatever.

What transpired as the Googling began was that I had bitten off rather more than I have ever attempted to chew. The fact is that we take the concept of parsing a little for granted and we use the term quite casually. I have recently learned quite what a huge and fundamental topic of computer science this really is – It has been around longer than computers, which perhaps says it all.

Before I go any further with my ramblings I’d like to point out that everything I now know about this topic I owe to one book; Parsing Techniques – A Practical Guide. Entirely as result of studying this book I have managed to implement a powerful LR(1) parser and parse table generator in PHP. If you don’t know what that means, I suggest you read the book, because I will not attempt to provide a tutorial on these deep topics. I will however whet your appetite with a little introduction, some demos, (aka showing off), and eventually some code.

Our notion of parsing

Let’s go back for a moment and look at our understanding of what parsing is. Suppose you want to parse a UK date (by hand so to speak) – in PHP you could do something like this:

1. $sDate = ‘03 / 11 / 1976′;
2. $aDate = preg_split( ‘/\D/’, $sDate, -1, PREG_SPLIT_NO_EMPTY );
3. $timestamp = mktime(0, 0, 0, $aDate[1], $aDate[0], $aDate[2], false );

Easy peasy – You have parsed the date, in so far as it has been transformed from a string of bytes to something tangible within your code. You can get meaningful handles on the data and do stuff with it. Great! – but not powerful or versatile; This is an entirely bespoke process that works for this purpose alone. Proper parsing techniques allow a problem like this to be standardized greatly, and will allow you to use the same code base to implement parsers for as many different things as your imagination and time allows you. Furthermore, it is possible to fully harness the expressive nature of a language rather than just chopping a sentence up and grabbing bits of it, like I have shown above.

As a proof of concept I’m going to use a humble date format to demonstrate the parser I have developed for PHP. In reality using such a powerful technique would be total overkill, but it makes for a pretty good first example.

Proof of concept

Let’s take a date format as the language for which we wish to create a parser. A legitimate expression (or sentence) in this language could be as follows: 1976-11-03 18:15:00

Starting with the top-most, and largest component of our language, let’s examine what it consists of and define ourselves a grammar which expresses its rules and syntax.

The complete sentence is quite simply a date – This entity is what we call the “goal symbol“, because it is our ultimate result. Let’s create a sensibly named symbol with an appropriate namespace.

<D_DATE>

We can split the D_DATE symbol into two separate components for date (1976-03-11) and time (18:15:00), Let’s call these symbols D_DATE_COMPONENT & D_TIME_COMPONENT. They are separated by a space, which we write literally.

<D_DATE> ::= <D_DATE_COMPONENT> " " <D_TIME_COMPONENT>

The literal space symbol is called a terminal symbol because it is an actual character in the date we are parsing and consequently can not be divided into any further symbols. The other symbols we have defined so far are called non-terminal because they are made up of further symbols, and are not physically present in the original date string. They are, to all intent and purpose, imaginary symbols created by us to provide an understanding of what the sentence means.

You get the gist – hopefully – So here’s the whole grammar.

1.  <D_DATE> ::=
2.     <D_DATE_COMPONENT> " " <D_TIME_COMPONENT>;
3.
4.  <D_DATE_COMPONENT> ::=
5.     <D_YEAR> "-" <D_MONTH> "-" <D_DAY>;
6.
7.  <D_TIME_COMPONENT> ::=
8.     <D_HOUR> ":" <D_MIN> ":" <D_SEC> ;
9.
10. <D_YEAR> ::=
11.    <D_DIGIT> <D_DIGIT> <D_DIGIT> <D_DIGIT> ;
12.
13. <D_MONTH> ::=
14.   <D_DIGIT> <D_DIGIT> ;
15.
16. <D_DAY> ::=
17.    <D_DIGIT> <D_DIGIT> ;
18.
19. <D_HOUR> ::=
20.    <D_DIGIT> <D_DIGIT> ;
21.
22. <D_HOUR> ::=
23.    <D_DIGIT> <D_DIGIT> ;
24.
25. <D_MIN> ::=
26.    <D_DIGIT> <D_DIGIT> ;
27.
28. <D_SEC> ::=
29.    <D_DIGIT> <D_DIGIT> ;
30.
31. <D_DIGIT> ::=
32.    "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" ;

This is written in a notational format called Backus Naur form (BNF). Non terminal symbols are written inside angled brackets and the terminal symbols are all written inside quotes as they are [in this case] literal. As you can see, the only terminal symbols of the language are the punctuation and numeric characters that a legal date string in this language will consist of. The last non-terminal D_DIGIT illustrates a choice of terminal symbol – the 10 decimal characters, (lines 31-32).

The parse tree

As we drill down from our goal symbol into its child symbols you may realise that we are seeing a tree-like structure. This tree has the goal symbol at it’s root, and along each branch we eventually find a leaf – a terminal symbol. This is important, because it is precisely the parser’s purpose in life to take a stream of terminal symbols and build them into a parse tree. Without further ado, here is the parse tree of our example date string.

[041] <D_DATE> :
[021] .  <D_DATE_COMPONENT> :
[008] .  .  <D_YEAR> :
[001] .  .  .  <D_DIGIT> :
[000] .  .  .  .  "1"
[001] .  .  .  </D_DIGIT>
[003] .  .  .  <D_DIGIT> :
[002] .  .  .  .  "9"
[003] .  .  .  </D_DIGIT>
[005] .  .  .  <D_DIGIT> :
[004] .  .  .  .  "7"
[005] .  .  .  </D_DIGIT>
[007] .  .  .  <D_DIGIT> :
[006] .  .  .  .  "6"
[007] .  .  .  </D_DIGIT>
[008] .  .  </D_YEAR>
[009] .  .  "-"
[014] .  .  <D_MONTH> :
[011] .  .  .  <D_DIGIT> :
[010] .  .  .  .  "1"
[011] .  .  .  </D_DIGIT>
[013] .  .  .  <D_DIGIT> :
[012] .  .  .  .  "1"
[013] .  .  .  </D_DIGIT>
[014] .  .  </D_MONTH>
[015] .  .  "-"
[020] .  .  <D_DAY> :
[017] .  .  .  <D_DIGIT> :
[016] .  .  .  .  "0"
[017] .  .  .  </D_DIGIT>
[019] .  .  .  <D_DIGIT> :
[018] .  .  .  .  "3"
[019] .  .  .  </D_DIGIT>
[020] .  .  </D_DAY>
[021] .  </D_DATE_COMPONENT>
[022] .  " "
[040] .  <D_TIME_COMPONENT> :
[027] .  .  <D_HOUR> :
[024] .  .  .  <D_DIGIT> :
[023] .  .  .  .  "1"
[024] .  .  .  </D_DIGIT>
[026] .  .  .  <D_DIGIT> :
[025] .  .  .  .  "8"
[026] .  .  .  </D_DIGIT>
[027] .  .  </D_HOUR>
[028] .  .  ":"
[033] .  .  <D_MIN> :
[030] .  .  .  <D_DIGIT> :
[029] .  .  .  .  "1"
[030] .  .  .  </D_DIGIT>
[032] .  .  .  <D_DIGIT> :
[031] .  .  .  .  "5"
[032] .  .  .  </D_DIGIT>
[033] .  .  </D_MIN>
[034] .  .  ":"
[039] .  .  <D_SEC> :
[036] .  .  .  <D_DIGIT> :
[035] .  .  .  .  "0"
[036] .  .  .  </D_DIGIT>
[038] .  .  .  <D_DIGIT> :
[037] .  .  .  .  "0"
[038] .  .  .  </D_DIGIT>
[039] .  .  </D_SEC>
[040] .  </D_TIME_COMPONENT>
[041] </D_DATE>

Click here to play with the interactive version
You will see that the parser throws a descriptive error if you try to parse an invalid date.

That’s the concept in a nutshell. Hopefully you can see that this tree structure gives us a powerful and flexible way to describe the anatomy of the input we have parsed. Compare that with simply grabbing the linear sequence of numbers that make up the date string and imagine the potential.

I shall leave this introduction for now. In part 2, I go on to discuss tokenizing and demonstrate the parsing of PHP source.