Architectural ideas for LiquidJS v11

[jg-rp]

Looking at your v11 project, I see that you've mentioned splitting tokenization and parsing phases, and an improved operator interface. So here is an overview of my personal preferences for a typical, single-pass lexer/parser architecture, with plain old tokens and distinct syntax tree nodes.

Even if you have a different approach in mind or you prefer to keep things closer to how they are, I hope this provides some inspiration and/or ideas.

---

Tokens are plain objects containing just three numbers.

type Token = {
  kind: TokenKind;
  start: number;
  end: number;
};

type TokenKind = (typeof T)[keyof typeof T];

// You could use an enum if preferred.
const T = {
  ADD: 1,
  AND: 2,
  ARROW: 3,
  ASSIGN: 4,
  COLON: 5,
  COMMA: 6,
  COMMENT: 7,

  // snip ...

  TAG_END: 53,
  TAG_NAME: 54,
  TAG_START: 55,
  TEXT: 56,
  TRUE: 58,
  WC: 59,
  EOI: 60,
  UNKNOWN: 61,
} as const;

With a helper function to get a token's value.

function getTokenValue(token: Token, source: string): string {
  return source.slice(token.start, token.end);
}

A lexer is responsible for building a flat token array. It scans markup and expressions in one pass. (I have a Shopify/liquid compatible implementation if you're interested.)

interface Lexer {
  tokenize(env: Liquid, source: string, startIndex?: number): Token[];
}

A parser transforms a token array into a syntax tree.

interface Parser {
  parse(env: Liquid, source: string, tokens: Token[]): Block;
}

A syntax tree is composed of plain text, tags/output and expressions.

type Node = string | Markup | Expression;
type Block = Array<string | Markup>;

interface Markup {
  render(context: RenderContext, buffer: OutputBuffer): Promise<void>;
  token: Token;
}

interface Expression {
  evaluate(context: RenderContext): Promise<unknown>;
  children(context: StaticContext): Traversable[];
  token: Token;
}

A tag is anything that creates a Markup node given a stream of tokens.

interface Tag {
  parse(token: Token, stream: TokenStream): Markup;
}

Tag would usually be implemented as a static method of a class implementing Markup. For example:

class AssignTag implements Markup {
  constructor(
    readonly token: Token,
    readonly name: Name,
    readonly expression: Expression,
  ) {}

  static parse(token: Token, stream: TokenStream): Markup {
    const name = stream.parseIdent();
    stream.eat(T.ASSIGN, "bad identifier or missing assignment operator");
    const expression = stream.parseFilteredExpression();
    return new AssignTag(token, name, expression);
  }

  async render(context: RenderContext, buffer: OutputBuffer): Promise<void> {
    context.assign(this.name.value, await this.expression.evaluate(context));
  }
}

TokenStream would have methods like parseBlock(end?: Set<string>): Block and parseExpression(precedence?: number): Expression, where parseExpression implements a typical recursive descent parser instead of converting to postfix notation.

All literals and operators are classes implementing Expression. For example:

class OrExpression implements Expression {
  constructor(
    readonly token: Token,
    readonly left: Expression,
    readonly right: Expression,
  ) {}

  async evaluate(context: RenderContext): Promise<unknown> {
    const left = await this.left.evaluate(context);
    return context.env.isTruthy(left)
      ? left
      : await this.right.evaluate(context);
  }

  children(context: StaticContext): Traversable[] {
    return [this.left, this.right];
  }

  toString(): string {
    return `${this.left} or ${this.right}`;
  }
}

// and

class StringLiteral implements Expression {
  constructor(
    readonly token: Token,
    readonly value: string,
  ) {}

  async evaluate(context: RenderContext): Promise<unknown> {
    return this.value;
  }

  children(context: StaticContext): Traversable[] {
    return [];
  }

  toString(): string {
    return this.token.kind === T.DOUBLE_QUOTED
      ? `"${this.value}"`
      : `'${this.value}'`;
  }
}

A template is simply an array of nodes that calls Markup.render for each node.

class Template {
  constructor(
    readonly env: Liquid,
    readonly nodes: Block,
  ) {}

  async render(): Promise<string> {
    const buffer: string[] = [];
    const context = new RenderContext(this);
    
    for (const node of this.nodes) {
      if (isString(node)) {
        buffer.push(node);
      } else {
        await node.render(context, buffer);
      }
    }

    return buffer.join("");
  }
}

Notice that the Liquid environment object is both the high-level API and central configuration object. We pass it around rather than deriving a distinct options object from it (I like the idea of configuration through extension here). Similarly, render-time configuration is a RenderContext object. It has a reference to the environment object because the environment is conceptually part of the render context.

class Liquid {
  public lexer: Lexer = LegacyLexer;
  public parser: Parser = LegacyParser;

  public tags: { [key: string]: Tag };
  public filters: { [key: string]: Filter };

  constructor() {
    this.tags = {};
    this.filters = {};
    this.setupTags();
    this.setupFilters();
  }

  public parse(source: string): Template {
    return new Template(this, this.parser.parse(this, source, this.lexer.tokenize(this, source, 0)));
  }
}

[harttle]

Thank you for the detailed proposal, I haven't started research yet, but I like most of the above ideas. Thank you @jg-rp .