feat(cmd/influx): influx cli can write CSV files

[sranka]

Closes #17003 #17356

This PR

• supports CSV files in 'influx write'
  • transformation of CSV data to line protocol is driven by CSV annotations
  • annotations described https://v2.docs.influxdata.com/v2.0/reference/syntax/annotated-csv/#annotations are all supported, so that flux query result can be written "as-is"
  • #datatype annotation is enhanced:
    • so you can mark non-field data with:
      • measurement, tag, time
      • ignore value in a column simply ignores the column
      • dateTime is either as a long (int64) number or RFC3339 format, this type is always used to represent time of measurement; or you can specify
        • dateTime:RFC3339 for RFC3339 time
        • dateTime:number to expect a long number
      • time value is also supported as an alias for the existing dateTime
    • a field data type can be used to let detect field's data type
    • default datatype for a column is =field= unless _field column is present (ignored then)
    • there can be at most 1 dateTime column
  • https://github.com/bonitoo-io/influxdb-csv-import/blob/master/README.md contains demonstrative CSV examples
• adds new options to 'influx write":

  -f, --file string        The path to the file to import
      --format string      Input format, either lp (Line Protocol) or csv (Comma Separated Values). Defaults to lp unless '.csv' extension

• a new "write dryrun" (i.e. 'write' command with 'dryrun' sub-command) can be used to validate and tune annotations in CSV files; parameters are the as in "write", but line protocol is written to stdout

• keeps the existing functionality (writing line protocol data) without changes

  • the default input format is lp (line protocol) unless a file with ".csv" extension is supplied

• CHANGELOG.md updated with a link to the PR (not the Issue)

• Well-formatted commit messages

• Rebased/mergeable

• Tests pass

• Documentation updated or issue created (provide link to issue/pr)

• Signed CLA (if not already signed)

[kelwang]

with this pr, we can also close #17356

[sanderson]

@sranka If using the linepart annotation in annotated CSV without the datatype annotations, should the time columns should always be an integer (not RFC3339) according to the precision the user specifies (just like line protocol) or will it convert RFC3339 timestamps into an integer for you?

[sanderson]

@sranka Also, how do you handle string and boolean field values with the linepart annotation? Do you have to include a datatype annotation? Or is it smart enough to know?

[sranka]

@sanderson linepart annotation is now independent on datatype annotation, at least conceptually. linepart places column data in the line, datatype is just field data type.

Regarding the data type annotation:

• if no data type is specified for a time column, it is detected from the data, so both number and RFC3339 string is supported, see https://github.com/influxdata/influxdb/blob/17003/writeFromCsv/write/csvTable.go#L349
• if no data type is specified for a field column, the column data are copied to line protocol as-is
• the datatype annotation for measurement and tag columns is ignored, the conversion to line protocol is given for these types

It might be simpler to introduce extra field, measurement or tag (and ignore) values in the datatype annotation so that the user could then supply a single annotation line (#datatype) to specify fields (with types) and non-field columns (measurement, time, tag and ignored). As I write this comment I tend to change the existing implementation this way and remove the linepart annotation. WDYT?

There is a wide area of enhancements that could be done regarding CSV, once this gets merged:

• time formats may vary
• header/annotation lines could be added from cli line without changing the CSV file
• constant tags, field values or measurement name could be supplied as well
• CSV with more time columns (such as https://data.humdata.org/dataset/novel-coronavirus-2019-ncov-cases) can be supported OOTB
• field types could be detected and enforced
• better error reporting, a better indication of progress / success / failure
• detection of data format without supplying a format flag
• ...
  So this is just a first step to support CSV.

[jsteenb2]

CLI looks good on my end. Would like to see @sebito91 or someone from @influxdata/storage-team take a gander at those bits

[jsteenb2]

excellent, CLI is looking good from my end. Thanks for running with feedback here 👍