-
Notifications
You must be signed in to change notification settings - Fork 0
Retrospective
(disclosure: written with assistance of Claude/Sonnet)
Looking back, several things landed right — both architecturally and in terms of developer experience.
-
Grammar as C# code. The operator overloading bet paid off. Using
|and+to express BNF rules directly in aGrammarsubclass meant no external DSL, no code generation step, no build pipeline ceremony. The grammar compiled with your project, refactored with your project, and debugged with your project. For developers coming from ANTLR or Lex/Yacc, the first working grammar with zero generated files was a genuine moment of friction removal. -
A set of reusable terminals (basic language elements). Irony turned parser/scanner construction into a truly OOP-based concept - everything is a class/component that can be possibly shared and reused. Irony provided a set of ready-to-use terminals.
NumberLiteral,StringLiteral,IdentifierTerminaland their siblings weren't regex wrappers — they were configurable components covering suffixes, prefixes, escape variants, and numeric forms. Real language quirks mapped to terminal properties rather than custom tokenizer code. -
A solid LALR(1) engine. Penello's algorithm for state graph construction gave correct, direct LALR(1) lookahead computation without the SLR approximation shortcut. Fast at runtime, correct across the full LALR(1) grammar class. State graph construction was fast enough to happen at startup — typically in the tens of milliseconds — making on-the-fly grammar initialization entirely practical.
-
Performant scanning and parsing. The scanner and LALR engine combination held up under real workloads. The scanner avoided regex overhead on the hot path, and the parser's shift-reduce loop was tight table-driven dispatch. Together they delivered throughput well above what most DSL embedding scenarios require.
-
Grammar Explorer made parser debugging much easier. Admittedly ugly UI creature, but very useful. The synchronized state graph, parsing log, parse tree, and source position view turned "shift/reduce conflict in state 47" from an archaeology problem into a direct inspection. Click a log entry, land on the responsible state and source span simultaneously.
-
The demo grammars proved capabilities and ease of adoption. Python's synthetic
INDENT/DEDENTtokens, BASIC's line-number-prefix structure, Refal's pattern-matching syntax, Brainfuck's minimal footprint — together they demonstrated that Irony wasn't secretly assuming C-family conventions. -
A very PRIMITIVE expression evaluator - answered "then what?" with a working evaluator in readable C# over an Irony parse tree.
-
Positive response and adoption. Developers building DSLs, linters, data parsers, and query engines found Irony and adopted it. The grammars contributed, questions asked, and issues filed all confirmed the core abstraction had landed. That community feedback also surfaced the limitations honestly — which we will discuss next.
Not everything ended up well. Here are the most visible troubles.
-
LALR(1)'s single-token lookahead was a hard ceiling. The deterministic single-state-machine model works cleanly for well-behaved grammars, but real-world languages — C# being the obvious example — routinely require more context than one preview token can provide. I attempted to extend Irony toward multi-token lookahead, but couldn't get it to a satisfactory state. What's actually needed is a different architectural posture: path branching at ambiguous decision points with backtracking on failure, rather than forcing everything through a deterministic automaton.
-
No macro or file include support. Textual preprocessing —
#include,#define, macro expansion — sits in front of the parser at the scanner level, and adding it properly would have required deep structural changes to the scanner engine. I never made that investment, which meant any grammar that needed preprocessing was a non-starter. -
Single-stream, single-file only. Irony operates on a single character stream. There was no concept of multi-file parsing, no way to stitch together compilation units, no support for anything resembling a project or module graph. Just text in, parse tree out. This becomes a severe limitation when trying to construct abstract syntax tree (AST) covering multiple files.
-
Very limited, rudimentary AST construction facilities. Enough said. My personal experience with building NGraphQL required a big effort and quite a lot of code to convert parsing tree into a specialized GraphQL object tree, and it felt like the experience should be improved.
-
No language versioning model. Many real languages evolve syntax across versions — C# is again the obvious case, with features gated by
LangVersion. Irony had no mechanism to associate grammar productions with a language version or to configure the parse engine to accept or reject version-specific syntax at runtime. -
No mixed-language input. HTML embedding JavaScript and CSS is the canonical example, but the pattern appears everywhere: template languages, markdown with code fences, SQL with embedded expressions. Irony had no model for switching grammar context mid-stream. One grammar, one language, full stop.
-
Error recovery was effectively primitive. After hitting a syntax error, Irony's recovery was rudimentary — the same class of panic-mode techniques described in the original Lex/Yacc literature from decades ago. For any use case where continuing to parse after an error matters — IDE integration, multi-error reporting, fault-tolerant tooling — this was a real liability.
-
Documentation was nonexistent. The codebase, the demo grammars, and whatever discussion existed in the Github issues were the documentation. For developers trying to do anything beyond the happy path, that was a significant barrier. It limited adoption and made the community's genuine interest harder to convert into productive use.
Irony started as a personal experiment — can BNF be expressed directly in C#, eliminating the Lex/Yacc ceremony entirely? The answer turned out to be yes, and the extras that followed (GrammarExplorer, the terminal factory, the demo grammars) made it genuinely useful. But taking it further was beyond what I could sustain alone.
Open source doesn't pay the bills, and after-hours time has to be allocated carefully. At the time I was deep in Vita, an ORM project that actually did pay the bills — my employer took a risk on it and it delivered. Irony had to wait.
The obvious question is why not lean on the community. The honest answer is that the gaps listed above aren't isolated features — they require re-architecting core internals. That kind of work needs coordination, design oversight, and lots of discussions. I didn't have the capacity to organize it, and I suspect most potential contributors were in the same position. Generic parsing infrastructure doesn't pay anyone's bills, with very few exceptions.
And here is my biggest failure. I tried to pitch the problem to Microsoft directly. Parsing tech is important (see next section). I argued .NET needs a solid, first-party parsing technology the same way it has JSON serialization and database access. Take Irony as a starting point (free, I don't want anything), build it properly, make it part of the platform. I even reached out to AndersH personally. The response was essentially: compilers use hand-written LL parsers for maximum performance, and for everything else the open source ecosystem — Irony included — is good enough; you are doing a great Job! That was generous, but it missed the point entirely. I wasn't saying Irony was good enough or bad. I was saying parsing was important and I couldn't do the job, the task was too large for volunteer hours. So it goes.
The last fifteen years made the case better than I could have argued it at the time. GraphQL, Protocol Buffers, YAML, TOML, and a wave of domain-specific data languages each spawned their own ecosystem of parsers, linters, formatters, and language-server integrations. None of those tools are trivial to build, and the parser is always the hard part.
Look at HotChocolate's GraphQL parser — it's a substantial, standalone solution, the product of significant engineering effort. That effort gets replicated, in some form, every time a new language or protocol needs a .NET implementation.
My own experience with NGraphQL makes the counter-case sharply. With Irony available, I had a working GraphQL parser in a couple of days with a single grammar file. The rest of the implementation took time, but the hardest part of the cold start was gone almost immediately. That's the leverage a solid parsing platform provides: when a new language or protocol is announced, a .NET implementation can appear quickly rather than waiting for someone to finish hand-writing a parser. Without that platform, the ecosystem perpetually plays catch-up.
There's a persistent assumption that a hand-written LL parser is always meaningfully faster than one built on generic infrastructure like Irony. That deserves more scrutiny than it usually gets.
On raw performance: it is well known that 80–90% of parse time is spent in the scanner, processing input character by character. Irony's scanner is a composition of individual Terminal scanners, each of which takes over once its input pattern is recognized — typically keyed on the first character. That per-terminal C# code can be written to be exactly as tight as equivalent code in a hand-written scanner. There's no inherent architectural penalty. A proper benchmark comparing the two remains to be done, but the assumption that Irony loses badly isn't obviously correct.
On whether it matters at all: for most real-world use cases, the answer is no. Consider a GraphQL server. A typical request roundtrip involves parsing the query (sub-millisecond), hitting the database for data (tens of milliseconds), and serializing the response (sub-millisecond). Whether the parse step takes 0.2ms or 0.5ms is irrelevant to end-to-end latency. The performance argument for hand-written parsers is legitimate in a narrow set of contexts — compiler front-ends processing millions of lines under tight latency budgets (but even in this case, I doubt Irony can't challenge it). For DSLs, query languages, configuration formats, and protocol parsers, it's mostly not the right trade-off to optimize for.
In spite of some discouraging statements above, let's try it. Let's reset and rebuild Irony. Parsing still matters. And there's AI to help us in routine tasks, and readily available knowledge of existing programming languages. I think AI can do a great job generating full grammars for existing popular languages, and creating test suites for them.