Zig-style multi-line strings

[antonmedv]

It's time to start thinking about the next v0.2 spec. A lot of feedback came for MAML. And the main criticism
was for multiline strings. This proposal is trying to address the latter.

Initially I declined Zig-style multiline strings, and decided to use
Python style multiline strings.

Current problems with multiline strings:

We can't include another tripple quotes """ inside it. This makes the current style not universal.
For example, we cannot simply put Python code inside, as Python may contain """ (yes, Python can
also use ''' for multiline strings, but this will require replacing """ with ''').
Quotes can't be placed at the end of a raw string """empty string is """"".

Indentation in deeply nested code makes it more difficult to read.

{
  policy: {
    requirements: {
      per-host: {
        template: """
listen 0.0.0.0;
port 8080;

filesystem /var/www;
"""
        vars: [42, 103]
      }
    }
  }
}

MAML uses double quotes for strings with escapes " and for raw multiline strings without escapes """.
This makes it difficult to understand (without knowing spec), what they behave differently, despite the
same character being used.

How do Zig-multiline strings look like?

const hello_world_in_c =
    \\#include <stdio.h>
    \\
    \\int main(int argc, char **argv) {
    \\    printf("hello world\n");
    \\    return 0;
    \\}
;

Multiline strings in Zig are essentially comments. They solve the problem of escaping and indentation.

What problems do Zig-style multiline strings have?

Mixing comments and multiline-strings.

  \\This is a multiline strings.
  //But this is comment,
  \\Keeps formatting as-is,
  //Mixing strings and comments 
  \\nice but possible.

But this is fixable if MAML does NOT allow comments inside multiline strings.

Another point is often brought up is space after \\ chars. Usually all comments have a space after the // chars.

  // Comments have leading space.
  \\No space in multiline strings.

In markup language, any character (beside {}, [], ", ,, #) can be used to denote a multiline raw string.

I'm proposing to adopt Zig-style multiline strings for MAML. But with a different character ` (backtick),
and without comments in between.

{
  # MAML Spec
  tags: [
    "minimal"
    "readable"
  ]

  # A simple nested object
  spec: {
    author: "Anton Medvedev"
  }

  poem:
    `Roses are red
    `Violets are blue,
    `Sugar is sweet
    `And so are you.
}

Backtick character is easy to type; it doesn't require a space after; it is visually light and easy to read.

Alternatives considered:

• Using one of < or >. But it can be confusing and difficult to remember which one to use (left or right).
• Using |. It is more difficult to type (pressing the Shift key on most keyboard layouts is required). And
  visually adding space between | and the string makes it looks better.

Overall I think ` backtick strings are nice. And cover both points of MAML:
easy to write and easy to parse: MAML already contains comments with #. Parsing
raw strings with ` is almost the same, only after each line we need to concatenate
with a previous line.

Raw strings with ` should not allow comments inside, no empty lines inside. If comments, or additional
empty lines separate raw multiline strings, an error should be raised.

Downside: backticks are usually used as strings (i.e., expected closing backtick), this could lead to confusion,
then raw string is used with just one line.

{
  regex: `^([\n"])+$
}

No closing backtick is needed, but if a backtick is added, it will become part of the string.

So I propose one of the following scenarios:

• Add new backtick strings and delete old multiline strings.
• Add new backtick strings and keep old multiline strings.
• Add but with an alternative character.
• Do not add backtick strings.

[DavidSeptimus]

From a user standpoint, I'm not typically a fan of prefixed multi-line strings as I like that I can copy/paste content to or from non-prefixed multi-line strings and not have to deal with cleaning up or adding the prefix (though IDEs can help with this to some extent). That said, the prefix definitely helps with both nicer formatting and unambiguous indentation of the string contents

My main concern with prefix-based multi-line strings is how they interact with arrays:

[
  `Roses are red
  `Violets are blue,
  
  `Sugar is sweet
  `And so are you.
]

All it takes is a misplaced newline to break a multi-line string into multiple array elements.

Another issue with this approach as a replacement for the current raw-string implementation is that prefixed multi-line comments can't be used inline. The following example would not be possible:
{ key: [ "val", """escape examples: \r\n\t""", "val2" ] }

It would have to be written either using a quoted string with escaped backslashes or like this:

{ key: [ "val", 
    `escape examples: \r\n\t
     "val2" ] }

Maybe not a deal-breaker, but the fact that prefixed multi-line strings can't be used inline is something to consider.

Using one of < or >. But it can be confusing and difficult to remember which one to use (left or right).

In my experience, I've only used < in the terminal on occasion, but never anywhere else. > is used in various contexts, often to denote a quote (as in markdown).

Overall I think ` backtick strings are nice. And cover both points of MAML:
easy to write and easy to parse

Harder to type than you're suggesting. > requires two keys, but reaching the backtick in the top-left corner of the keyboard requires hand movement for touch typers without giant fingers. It's kind of a toss-up on ease of typing.

regex: `^([\n\u])+$

What's \u's purpose in this regex?

From my perspective, this proposal is likely to add more challenges than it solves.

[antonmedv]

I'm not typically a fan of prefixed multi-line strings as I like that I can copy/paste content to or from non-prefixed multi-line strings and not have to deal with cleaning up or adding the prefix

Same for me. Copy/pasting is one of mine concerns as well. But you are right IDE can help (we are lucky what you're IDE-expert😁) , or even CLI. I'm planning on building a tool for maml as well. sed 's/^[[:space:]]*`//'.

My main concern with prefix-based multi-line strings is how they interact with arrays:

This is a good point. This is I think allowing extra lines in between (as in Zig is a bad idea). But I think separation is visually noticeable:

[
  `Roses are red
  `Violets are blue,
  
  `Sugar is sweet
  `And so are you.
]

[
  `Roses are red
  `Violets are blue,
  `
  `Sugar is sweet
  `And so are you.
]

Another issue with this approach as a replacement for the current raw-string implementation is that prefixed multi-line comments can't be used inline.

MAML aims to be a minimal configuration format. So minification is not so much important. (I even thinking as maml is for configurations, should the root object always be an object?)

Harder to type than you're suggesting.

Good point. What about '?

What's \u's purpose in this regex?

Bad example, I will update the comment.

From my perspective, this proposal is likely to add more challenges than it solves.

Thanks for the feedback. So you think "Do not add backtick strings." is proper solution? What about supporting of """ and ` strings?

[DavidSeptimus]

My main argument against their inclusion even in addition to """ strings is that I view the empty line separation of prefixed strings as equivalent to inline whitespace delimiting primitive values.

That is to say:

this (keep in mind that the string could be an arbitrary number of lines)

[
  `Roses are red
  `Violets are blue,
  
  `Sugar is sweet
  `And so are you.
]

is roughly equivalent to this:

[12 345 6789]

Both make my skin crawl a bit. In either case, you're one whitespace character away in the same direction as the values flow from an unintended state.

[antonmedv]

What about this?

[
  12
  345
  6789
]

Values are separated by newline. Same as backtick, but they are require a whole line in between.

[antonmedv]

What about requiring comma in between backticks inside arrays?

[
  `Roses are red
  `Violets are blue,
,
  `Sugar is sweet
  `And so are you.
]

But this looks ugly and complicates parser.

What about actually allowing empty lines in between backtick lines! This solves the ambiguity inside arrays if someone accidentally adds an empty line. Good idea!

[ericem]

I'm also not a fan of the prefix notation, as this makes it more difficult to copy/paste between different config formats. Related, if a tool is needed to ease editing, I think this negates the purpose of a format that is easy for humans to edit. I have a couple of ideas to think about. Have you considered Elixir's sigil syntax? It uses a tilde followed by a single character and then a delimiter. So a an inline string could be: ~s(some \nsdata\here\n) and an multi-line string could be:

  ~s'''
       Roses are red
       Violets are blue,
  
       Sugar is sweet
      And so are you.
'''

This would allow users to change the delimiter based on the type of string data they are embedding. For example, you could use triple-backticks for Python code, and triple-quotes if you are embedding a markdown fenced code block.

[antonmedv]

I think Elixir's sigil syntax is pretty close to Python. Same problems: you need to choose different markers.

[DavidSeptimus]

@antonmedv In a previous thread, you had mentioned potentially allowing \""" inside of raw strings. Have you ultimately decided against that?

[antonmedv]

Sorry, I guess I missed it. In this case how to escape \"""? As \\"""? This essentially adds escaping to raw strings.

I like escape-less, same as in Go: raw strings do not support backticks.

[antonmedv]

Another idea, using single quotes '.

{
  # A simple nested object
  spec: {
    author: 'Anton Medvedev
  }

  poem:
    'Roses are red
    'Violets are blue,
    'Sugar is sweet
    'And so are you.
}

[antonmedv]

Also I was thinking about adding a "wrapper" for backticks. To denote beginning and end of string. Although it solves array of backticks problem, I think it looks more ugly and make writing single line raw strings too difficult.

{
  poem: 
    ```
    `Roses are red
    `Violets are blue,
    `Sugar is sweet
    `And so are you.
    ```
}

[arogozhnikov]

I wrote some thoughts similar to this proposal a couple of years ago (search for f""" in the page) :
https://arogozhnikov.github.io/2022/11/29/delimiter-comes-first.html

There is a benefit of having a separate in-line element before multiline strings - you can add all kinds of string modifiers / tag functions. It also allows you to consider multi-line string as object that you can operate on in expressions if neccessary. There is not need to denote end of string.

[lhearachel]

I think either system (a Zig-style prefix token for each line or a Python-style prefix token for the whole block) has merits and the precise choice will come down to preference... 😅

If I can prompt a (potentially dangerous) question: What if there were an adopted "special" character which can placed directly after a block-string leader that specifies that identical leading whitespace from each line is stripped? e.g.,

{
  poem: """-
    Roses are red,
      Violets are blue,
    Sugar is sweet,
    And so are you.
  """
}

This would evaluate to:

Roses are red,
  Violets are blue,
Sugar is sweet,
And so are you.

As maligned as YAML can be, I do feel that its support for multi-line string support feels extremely nice for, e.g., embedding aligned text or code snippets while not disrupting the overall flow of the document. This does come at the cost of some expressiveness in the grammar—I don't know of a way to denote retained vs. discarded whitespace in ABNF—but perhaps the trade-off is worthwhile.

[antonmedv]

Thanks everyone for the thoughtful input. After weighing the trade-offs, we’ve decided to stick with the existing triple-quoted multiline strings. The explicit start/end delimiter is clearer for readers and tooling, and avoids the added complexity and potential confusion of Zig-style line-prefixed strings. Closing this discussion.