Add support for JSON comments

[Kira-NT]

Description

Hi! This PR adds optional support for JSON-style comments.

YAML 1.2 is, by design, a superset of JSON. Unfortunately, though, with the widespread adoption of formats like JSONC, the ability to use a single YAML parser for both YAML and JSON has become less relevant over time. In my case, for example, I'm working on a tool that I want to keep as small as possible when compiled with NativeAOT to simplify its distribution. However, it needs to parse both YAML and JSON with comments, which forces me to depend on both System.Text.Json and YamlDotNet - even though I'd happily sacrifice some performance when parsing strictly JSON files if it meant I could trim System.Text.Json away.

This PR should help people in similar situations by allowing YamlDotNet to process comments according to the JSONC specification, if they choose to enable the feature.

Usage

Compatibility mode for JSON-style comments is enabled by setting Scanner.AllowJsonComments to true.

You may notice that this property is marked as internal. My reasoning is that, since this feature has nothing to do with the YAML specification, it's not really fair to ask the YamlDotNet developers to support it. At the same time, making the flag part of the public API would implicitly commit them to doing so. For that reason, I made AllowJsonComments internal, as this leaves room for the maintainers to modify or even remove the feature entirely if it ever becomes too burdensome, without having to treat the change as a breaking one.

Users who want to opt into this feature can do so via:

typeof(Scanner).GetProperty("AllowJsonComments", BindingFlags.NonPublic | BindingFlags.Instance).SetValue(scanner, true);

Or, better yet:

[UnsafeAccessor(UnsafeAccessorKind.Method, Name = "set_AllowJsonComments")]
static extern void SetAllowJsonComments(Scanner scanner, bool allowJsonComments);

// ...

SetAllowJsonComments(scanner, true);

That said, I may be overthinking this a bit, and perhaps simply marking the property with [EditorBrowsable(EditorBrowsableState.Never)] would communicate the same intent without it being so overzealous. Let me know what you think!

Example

When AllowJsonComments is enabled, YamlDotNet becomes capable of parsing any JSONC file (and even JSONC embedded within YAML). Here's an example taken directly from the new tests:

{
    /*
     * Top comment.
     */
    /*0*/ /*1*/"foo"/*2*/ /*3*/:/*4*/ /*5*/false/*6*/ /*7*/,// 8
    // Middle comment.
    /*9*/"bar"/*10*/:/*11*/true//12
    /*
     * Bottom comment.
     */
}

By default, it's parsed as:

{
    "/* * Top comment. */ /*0*/ /*1*/\"foo\"/*2*/ /*3*/:/*4*/ /*5*/false/*6*/ /*7*/": null,
    "// 8 // Middle comment. /*9*/\"bar\"/*10*/:/*11*/true//12 /* * Bottom comment. */": null
}

With AllowJsonComments set to true, however, it becomes:

{
    "foo": false,
    "bar": true
}

Implementation

When AllowJsonComments is false (i.e., the default), the logic of Scanner remains completely unaltered.

However, funnily enough, even when I set it to true by default and ran the test suite to see which YAML features would break in that mode, all tests still passed. Obviously, tests are imperfect, but hopefully this gives you an idea of how unintrusive this feature is.

As for the notable differences in parsing behavior, consider the following example:

{
  foo: https://github.com,
  bar: null/*null*/,
  baz: baz/*baz*/,
  // one: 1,
  // two
}

By default, YamlDotNet parses it as:

{
    "foo": "https://github.com",
    "bar": "null/*null*/",
    "baz": "baz/*baz*/",
    "// one": "1",
    "// two": null
}

With AllowJsonComments enabled, it becomes:

{
    "foo": "https://github.com",
    "bar": null,
    "baz": "baz/*baz*/"
}

This is definitely not something you would typically see in the real world, so it is unsurprising that the current test suite does not catch these differences. Nevertheless, it is still worth noting that they exist.

Remaining Questions

• Should AllowJsonComments remain internal?
• Should its effect be limited to flow collections, given the focus on JSONC compatibility, or should the following example remain valid?

  // First comment.
  - foo // Second comment.

---

Closes #1052
Closes #1112