[sc34wg3] Whitespace agnosticism in CTM and descendants

[Robert Barta]

On Thu, Apr 05, 2007 at 09:49:33PM +1200, Xu?n Baldauf wrote:
> > I think, that we do not have the complete picture yet. In the expression
> >
> >     a(x:y:z)
> >
> > there is actually NO ambiguity:
[...]
> > And prefixes MUST be declared before they are used.

> Yes, because the parsing depends on the context. (The parser needs
> to know wether the identifier is bound or not to decide which type
> of interpretation should be attached to "x:y:z", and this is not
> context-free.) This was one of my concerns, as most parsers
> generators can only generate parsers for context-free grammars.

Parsing of _any useful_ language _always_ is context sensitive. To
make this happen, even the most ancient parser generators (yacc/bison)
allow to define the Lexer (yylex) to be a _general_ function. In that
function one can tokenize according to the context information.

You need this even in C where you better disambiguate on the token
level, what an identifier is

  statement -> identifier '(' parameter_list ')'
            |  identifier '=' expression

You definitely do not want your parser to have a look-ahead when the
identifier clearly is either a defined function or a defined variable.

In this case, the lookup of an identifier whether it is used as
namespace prefix is not a big deal, I think.

\rho