How does a parser (for example, HTML) work?

Question

For argument\'s sake lets assume a HTML parser.

I\'ve read that it tokenizes everything first, and then parses it.

What does tokenize mean?

Answer 1

HTML and XML syntax (and others based on SGML) are quite hard to parse and they don't fit well into the lexing scenario, because they're not regular. In the parsing theory, a regular grammar is the one with doesn't have any recursion, that is, self-similar, nested patterns, or parentheses-like wrappers which have to match each other. But HTML/XML/SGML-based languages does have nested patterns: tags could be nested. Syntax with nesting patterns is higher in level in the Chomsky's classification: it's context-free or even context-dependent.

But back to your question about lexer:
Each syntax consists of two kinds of symbols: non-terminal symbols (those which unwind into other syntax rules) and terminal symbols (those which are "atomic" - they are leafs of the syntax tree and don't unwind into anything else). Terminal symbols are often just the tokens. Tokens are pumped one by one from the lexer and matched to their corresponding terminal symbols.

Those terminal symbols (tokens) have often regular syntax, which is easier to recognize (and that's why it's factored out to the lexer, which is more specialized for regular grammars and could do it quicker than by using more general approach of non-regular grammars).

So, to write a lexer for HTML/XML/SGML-like language, you need to find parts of the syntax which are atomic enough and regular, to be dealt with easily by the lexer. And here the problem arises, because it's not at first obvious which parts are these. I struggled with this problem for a long time.

But Lie Ryan above have done a very good job in recognizing these parts. Bravo for him for that! The token types are following:

• TagOpener: < lexeme, used for starting tags.
• TagCloser: > lexeme, used for ending tags.
• ClosingTagMarker: / lexeme used in closing tags.
• Name: alphanumeric sequence starting with letter, used for tag names and attribute names.
• Value: Text which can contain variety of different characters, spaces etc. Used for values of attributes.
• Equals: = lexeme, used for separating attribute names from its values.
• Quote: ' lexeme, used for enclosing attribute values.
• DoubleQuote: " lexeme, used for enclosing attribute values.
• PlainText: Any text not containing < character directly and not covered by the above types.

You can also have some tokens for entity references, like   or &. Probably:

• EntityReference: a lexeme consisting of & followed by some alphanumeric characters and ended with ;.

Why I used separate tokens for ' and " and not one token for attribute value? Because regular syntax couldn't recognize which of these characters should end the sequence - it depends on the character which started it (ending character have to match the starting character). This "parenthesizing" is considered non-regular syntax. So I promote it into a higher level - to the Parser. It'd be his job to match these tokens (starting and ending) together (or none at all, for simple attribute values not containing spaces).

Afterthought: Unfortunately, some of these tokens may occur only inside other markup. So the use of lexical contexts is needed, which after all is another state machine controlling the state machines recognizing particular tokens. And that's why I've said that SGML-like languages don't fit well into the schema of lexical analysis.