fmt formatting string can't be a variable

[IvanScheers]

fmt accepts hard coded strings, but no variable strings.

This works :

var name: string = "Albert"
echo fmt("Hi {name}, this is a test")

While the code below throws a compile error "only works with string literals". Isn't the variable format a string literal ? Even when it is declared as const (before the loop), it doesn't work.

var name: string = "Bernard"
let format = "Hi {name}, this is a test"
echo fmt(format)                                     # error : only works with string literals

Or is it impossible to use a var because the compiler needs to know beforehand ?
However it works with variables as format strings when using a template for sprintf:

proc csnprintf(result: cstring,size:csize_t, formatstr: cstring):cint {.importc: "snprintf", varargs,
                                  header: "<stdio.h>",discardable.}

template sprintf*(formatstr: string,args:varargs[untyped]):string =
  let len = csnprintf(nil,0,formatstr,args)
  var result = newString(len)
  csnprintf(result,(len+1).csize_t,formatstr,args)
  result

var name: string = "Albert"
var temperature: float = 25.731
var formatter: string = "Hello %s it's %.1f degrees outside"
var sentence: string

sentence = sprintf(formatter, name, temperature)
echo sentence

I would like to do this:

formatter: string = "Hi {name}, it is {temperature:.1f} outside"
sentence = fmt(formatter)

Note: this is a very simple example. The format strings would be in tables and some would be constructed on the fly during runtime.

If it is something that can be fixed/implemented and if so will it be in the not too distant future ? I should make design decision now, even if I go the sprintf route for the time being, I could already code in parts with when statements.

The project is a game speech system (TTS based), where the format strings have to be more or less created on the fly based on parameters.

Nim fmt with variables as format strings would have great benefits:

• use {named} curlies in the format string, beats sprintf by miles with it's %s %d %f fixed position parameter passing
• easy to make the app multi-lingual by loading a list of strings at program startup.

The issue has been mentioned before:
https://scripter.co/notes/nim-fmt/#fmt-and-and-formatting-strings-cannot-be-a-variable

$ nim -v
Nim Compiler Version 1.4
# make sure to include the git hash if not using a tagged release

[kaushalmodi]

Here's the discussion on Nim Forum that triggered this issue.

[juancarlospaco]

For JavaScript targets https://juancarlospaco.github.io/nodejs/nodejs/jsstrformat.html

[Araq]

If it is something that can be fixed/implemented and if so will it be in the not too distant future ?

No and no.

I should make design decision now, even if I go the sprintf route for the time being, I could already code in parts with when statements.

What you should do: Use a custom format proc that supports what you need. Feel free to use strutils.%'s implementation as the starting point.

[IvanScheers]

Ok, thank you. I'll look into jsstrformat, that seems promising and if not try and code a custom format proc.

[bluenote10]

A bit of background to support @Araq's comment:

Usually in type safe languages, string interpolation supports type safe formatters. For instance in Scala it is a compile time error to use a formatter that is inappropriate for the type:

This behavior obviously requires to know the formatting literal at compile time. In contrast dynamic languages like Python cannot find such bugs statically anyway, and thus can allow for dynamically generated format strings.

I always felt it is a bit unfortunate that Nim's string formatting has the limitation of not being able to use dynamically generated format strings, while missing the potential of statically checking formatter correctness. For instance, Nim's equivalent to the example above

let s = "foo"
echo &"{s:.3f}"

unfortunately compiles, and throws an exception at runtime instead. Even semi-valid looking formatters like &"{1:.3f}" are runtime errors. This behavior is particularly frustrating if you're running some ad-hoc long-running number crunching algorithm, and you eagerly await its final print statements (kpis, stats, ...) and it crashes exactly at that point 😂

So rather than going in the direction of allowing dynamic format strings (which may be technically impossible anyway), it would make sense to make strformat fully type safe one day.

[Araq]

unfortunately compiles, and throws an exception at runtime instead.

Er, didn't you submit the original implementation of strformat? You could have given us a better initial implementation. ;-)

[bluenote10]

Er, didn't you submit the original implementation of strformat? You could have given us a better initial implementation. ;-)

I'm pretty sure it was a compilation error in my original implementation using the Scala style formatters, because this was a high priority acceptance criteria of mine. The type safety got lost in the re-implementation towards Python style formatters. I had mentioned the issue on the PR/discussion somewhere. I always wanted to have a look how hard it would be to make the Python style formatters type safe as well, but I never got to it...

[Araq]

I'm pretty sure it was a compilation error in my original implementation using the Scala style formatters, because this was a high priority acceptance criteria of mine. The type safety got lost in the re-implementation towards Python style formatters. I had mentioned the issue on the PR/discussion somewhere. I always wanted to have a look how hard it would be to make the Python style formatters type safe as well, but I never got to it...

Bummer, I wasn't aware.

[timotheecour]

This is a very useful feature and I have an implementation for it:

• more ergonomic than strutils.%
• more efficient than strutils.%: this avoids temporaries and substitutions are lazy, an important point when some variables are potentially unused and expensive to render
• works with runtime pattern unlike fmt (parseExpr is not used)
• works inside templates unlike fmt which suffers from "undeclared identifier" error when using fmt from strformat on devel inside a template  #10977
• more flexible, eg allows arbitrary expressions for substitution variables not found
• typesafe unlike sprintf
• can also work with local variables in scope if variable list not provided
• works in all backends (eg c, js, vm)

when true:
  import formatutils
  block: # D20210615T123026
    var x = @[10, 2]
    let y = 123

    doAssert interpIt("{x} and {y}", (x, y)) == "@[10, 2] and 123" # simplest example
    doAssert interpIt("{y} times 2 is {y2}", (y, y2: y*2)) == "123 times 2 is 246" # allow introducing variables on the fly
    doAssert interpIt("{x} and {y} using locals") == "@[10, 2] and 123 using locals" # using locals()

    var s = "foo {x}"
    doAssert interpIt(s, (x, )) == "foo @[10, 2]" # works with runtime strings
    doAssert interpIt(s) == "foo @[10, 2]" # ditto
    doAssertRaises(ValueError): discard interpIt("foo {badvar}")

    doAssertRaises(ValueError): discard interpIt("{y} {bad}", (x, )) #  # substitution variable not found: bad
    doAssert interpIt("{y} {bad} {bad2}", (y, ), it.toUpper) == "123 BAD BAD2" # use arbitrary expression for variables not found (via `it`)

    let z = x[0].addr
    doAssert interpIt("variable is {z}", (z: $z[])).tap == "variable is 10" # custom $

IMO this is clearly stdlib territory; I understand there are lots of open discussions at the moment but eventually this should make its way

note

• the tuple of variables passed in (eg (x, y)) is neither rendered nor codegen'd, instead each variables are rendered lazily, so a variable not referenced in runtime input string will not be stringified

• note that this isn't intended as a replacement for fmt, which exploits fact that input string is known at CT and can use parseExpr, but it could be seen as a replacement for strutils.%

• it has other features i didn't talk about (eg a 0-alloc version which takes input buffer by var, custom formatting instead of {}, escaping of those, etc; adding $a instead of {a} is not implemented but would be easy)

• implementation is compact and optimized

[IvanScheers]

Thank you timothee ! I'm anxious to try it out. Is formatutils going to be in the nimble package directory ?

[timotheecour]

I'd rather it be added to stdlib so it can be used in stdlib and compiler, but maybe I'll make a nimble package in the meantime

[barcharcraz]

I'm pretty sure it was a compilation error in my original implementation using the Scala style formatters, because this was a high priority acceptance criteria of mine. The type safety got lost in the re-implementation towards Python style formatters. I had mentioned the issue on the PR/discussion somewhere. I always wanted to have a look how hard it would be to make the Python style formatters type safe as well, but I never got to it...

Bummer, I wasn't aware.

It looks like this is because strFormatImpl emits a call to formatValue, and the string version binds to everything with a $ but then calls the string only formatValue that raises the exception if the spec type isn't s. strFormatImpl should parse the whole format string, including specs, at compile time, do error checking, and pass a static specs structure to the actual formatValue implementations.