Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restore the --jvquoteall flag #1083

Merged
merged 4 commits into from
Aug 22, 2022
Merged

Restore the --jvquoteall flag #1083

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ba8fd44
Restore --jvquoteall flag
johnkerl Aug 22, 2022
b17f7c1
make dev: for doc-build artifacts
johnkerl Aug 22, 2022
9e8a187
unit-test file
johnkerl Aug 22, 2022
9e6343e
Merge branch 'main' into kerl/jvquoteall
johnkerl Aug 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 18 additions & 13 deletions docs/src/manpage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ HELP OPTIONS
mlr help file-format-flags
mlr help flatten-unflatten-flags
mlr help format-conversion-keystroke-saver-flags
mlr help json-only-flags
mlr help legacy-flags
mlr help miscellaneous-flags
mlr help output-colorization-flags
Expand Down Expand Up @@ -441,14 +442,27 @@ FORMAT-CONVERSION KEYSTROKE-SAVER FLAGS
-p Keystroke-saver for `--nidx --fs space --repifs`.
-T Keystroke-saver for `--nidx --fs tab`.

JSON-ONLY FLAGS
These are flags which are applicable to JSON output format.

--jlistwrap or --jl Wrap JSON output in outermost `[ ]`. This is the
default for JSON output format.
--jvquoteall Force all JSON values -- recursively into lists and
object -- to string.
--jvstack Put one key-value pair per line for JSON output
(multi-line output). This is the default for JSON
output format.
--no-jlistwrap Wrap JSON output in outermost `[ ]`. This is the
default for JSON Lines output format.
--no-jvstack Put objects/arrays all on one line for JSON output.
This is the default for JSON Lines output format.

LEGACY FLAGS
These are flags which don't do anything in the current Miller version.
They are accepted as no-op flags in order to keep old scripts from breaking.

--jknquoteint Type information from JSON input files is now
preserved throughout the processing stream.
--jlistwrap or --jl Wrap JSON output in outermost `[ ]`. This is the
default for JSON output format.
--jquoteall Type information from JSON input files is now
preserved throughout the processing stream.
--json-fatal-arrays-on-input
Expand All @@ -458,17 +472,8 @@ LEGACY FLAGS
--json-skip-arrays-on-input
Miller now supports arrays as of version 6.
--jsonx The `--jvstack` flag is now default true in Miller 6.
--jvquoteall Type information from JSON input files is now
preserved throughout the processing stream.
--jvstack Put one key-value pair per line for JSON output
(multi-line output). This is the default for JSON
output format.
--mmap Miller no longer uses memory-mapping to access data
files.
--no-jlistwrap Wrap JSON output in outermost `[ ]`. This is the
default for JSON Lines output format.
--no-jvstack Put objects/arrays all on one line for JSON output.
This is the default for JSON Lines output format.
--no-mmap Miller no longer uses memory-mapping to access data
files.
--ojsonx The `--jvstack` flag is now default true in Miller 6.
Expand Down Expand Up @@ -658,7 +663,7 @@ OUTPUT-COLORIZATION FLAGS
`--list-color-names`) for record values.

PPRINT-ONLY FLAGS
These are flags which are applicable to PPRINT output format.
These are flags which are applicable to PPRINT format.

--barred Prints a border around PPRINT output (not available
for input).
Expand Down Expand Up @@ -3282,5 +3287,5 @@ SEE ALSO



2022-08-20 MILLER(1)
2022-08-22 MILLER(1)
</pre>
31 changes: 18 additions & 13 deletions docs/src/manpage.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ HELP OPTIONS
mlr help file-format-flags
mlr help flatten-unflatten-flags
mlr help format-conversion-keystroke-saver-flags
mlr help json-only-flags
mlr help legacy-flags
mlr help miscellaneous-flags
mlr help output-colorization-flags
Expand Down Expand Up @@ -420,14 +421,27 @@ FORMAT-CONVERSION KEYSTROKE-SAVER FLAGS
-p Keystroke-saver for `--nidx --fs space --repifs`.
-T Keystroke-saver for `--nidx --fs tab`.

JSON-ONLY FLAGS
These are flags which are applicable to JSON output format.

--jlistwrap or --jl Wrap JSON output in outermost `[ ]`. This is the
default for JSON output format.
--jvquoteall Force all JSON values -- recursively into lists and
object -- to string.
--jvstack Put one key-value pair per line for JSON output
(multi-line output). This is the default for JSON
output format.
--no-jlistwrap Wrap JSON output in outermost `[ ]`. This is the
default for JSON Lines output format.
--no-jvstack Put objects/arrays all on one line for JSON output.
This is the default for JSON Lines output format.

LEGACY FLAGS
These are flags which don't do anything in the current Miller version.
They are accepted as no-op flags in order to keep old scripts from breaking.

--jknquoteint Type information from JSON input files is now
preserved throughout the processing stream.
--jlistwrap or --jl Wrap JSON output in outermost `[ ]`. This is the
default for JSON output format.
--jquoteall Type information from JSON input files is now
preserved throughout the processing stream.
--json-fatal-arrays-on-input
Expand All @@ -437,17 +451,8 @@ LEGACY FLAGS
--json-skip-arrays-on-input
Miller now supports arrays as of version 6.
--jsonx The `--jvstack` flag is now default true in Miller 6.
--jvquoteall Type information from JSON input files is now
preserved throughout the processing stream.
--jvstack Put one key-value pair per line for JSON output
(multi-line output). This is the default for JSON
output format.
--mmap Miller no longer uses memory-mapping to access data
files.
--no-jlistwrap Wrap JSON output in outermost `[ ]`. This is the
default for JSON Lines output format.
--no-jvstack Put objects/arrays all on one line for JSON output.
This is the default for JSON Lines output format.
--no-mmap Miller no longer uses memory-mapping to access data
files.
--ojsonx The `--jvstack` flag is now default true in Miller 6.
Expand Down Expand Up @@ -637,7 +642,7 @@ OUTPUT-COLORIZATION FLAGS
`--list-color-names`) for record values.

PPRINT-ONLY FLAGS
These are flags which are applicable to PPRINT output format.
These are flags which are applicable to PPRINT format.

--barred Prints a border around PPRINT output (not available
for input).
Expand Down Expand Up @@ -3261,4 +3266,4 @@ SEE ALSO



2022-08-20 MILLER(1)
2022-08-22 MILLER(1)
1 change: 1 addition & 0 deletions docs/src/new-in-miller-6.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,7 @@ JSON support is improved:
* Streamable JSON parsing: Miller's internal record-processing pipeline starts as soon as the first record is read (which was already the case for other file formats). This means that, unless records are wrapped with outermost `[...]`, Miller now handles JSON / JSON Lines in `tail -f` contexts like it does for other file formats.
* Flatten/unflatten -- conversion of JSON nested data structures (arrays and/or maps in record values) to/from non-JSON formats is a powerful new feature, discussed in the page [Flatten/unflatten: JSON vs. tabular formats](flatten-unflatten.md).
* Since types are better handled now, the workaround flags `--jvquoteall` and `--jknquoteint` no longer have meaning -- although they're accepted as no-ops at the command line for backward compatibility.
* Update: `--jvquoteall` was restored shortly after 6he 6.4.0 release.

JSON vs JSON Lines:

Expand Down
1 change: 1 addition & 0 deletions docs/src/new-in-miller-6.md.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ JSON support is improved:
* Streamable JSON parsing: Miller's internal record-processing pipeline starts as soon as the first record is read (which was already the case for other file formats). This means that, unless records are wrapped with outermost `[...]`, Miller now handles JSON / JSON Lines in `tail -f` contexts like it does for other file formats.
* Flatten/unflatten -- conversion of JSON nested data structures (arrays and/or maps in record values) to/from non-JSON formats is a powerful new feature, discussed in the page [Flatten/unflatten: JSON vs. tabular formats](flatten-unflatten.md).
* Since types are better handled now, the workaround flags `--jvquoteall` and `--jknquoteint` no longer have meaning -- although they're accepted as no-ops at the command line for backward compatibility.
* Update: `--jvquoteall` was restored shortly after 6he 6.4.0 release.

JSON vs JSON Lines:

Expand Down
1 change: 1 addition & 0 deletions docs/src/online-help.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ Flags:
mlr help file-format-flags
mlr help flatten-unflatten-flags
mlr help format-conversion-keystroke-saver-flags
mlr help json-only-flags
mlr help legacy-flags
mlr help miscellaneous-flags
mlr help output-colorization-flags
Expand Down
20 changes: 14 additions & 6 deletions docs/src/reference-main-flag-list.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,19 @@ Additionally:
* `-p` is a keystroke-saver for `--nidx --fs space --repifs`.
* `-T` is a keystroke-saver for `--nidx --fs tab`.

## JSON-only flags

These are flags which are applicable to JSON output format.


**Flags:**

* `--jlistwrap or --jl`: Wrap JSON output in outermost `[ ]`. This is the default for JSON output format.
* `--jvquoteall`: Force all JSON values -- recursively into lists and object -- to string.
* `--jvstack`: Put one key-value pair per line for JSON output (multi-line output). This is the default for JSON output format.
* `--no-jlistwrap`: Wrap JSON output in outermost `[ ]`. This is the default for JSON Lines output format.
* `--no-jvstack`: Put objects/arrays all on one line for JSON output. This is the default for JSON Lines output format.

## Legacy flags

These are flags which don't do anything in the current Miller version.
Expand All @@ -228,17 +241,12 @@ They are accepted as no-op flags in order to keep old scripts from breaking.
**Flags:**

* `--jknquoteint`: Type information from JSON input files is now preserved throughout the processing stream.
* `--jlistwrap or --jl`: Wrap JSON output in outermost `[ ]`. This is the default for JSON output format.
* `--jquoteall`: Type information from JSON input files is now preserved throughout the processing stream.
* `--json-fatal-arrays-on-input`: Miller now supports arrays as of version 6.
* `--json-map-arrays-on-input`: Miller now supports arrays as of version 6.
* `--json-skip-arrays-on-input`: Miller now supports arrays as of version 6.
* `--jsonx`: The `--jvstack` flag is now default true in Miller 6.
* `--jvquoteall`: Type information from JSON input files is now preserved throughout the processing stream.
* `--jvstack`: Put one key-value pair per line for JSON output (multi-line output). This is the default for JSON output format.
* `--mmap`: Miller no longer uses memory-mapping to access data files.
* `--no-jlistwrap`: Wrap JSON output in outermost `[ ]`. This is the default for JSON Lines output format.
* `--no-jvstack`: Put objects/arrays all on one line for JSON output. This is the default for JSON Lines output format.
* `--no-mmap`: Miller no longer uses memory-mapping to access data files.
* `--ojsonx`: The `--jvstack` flag is now default true in Miller 6.
* `--quote-all`: Ignored as of version 6. Types are inferred/retained through the processing flow now.
Expand Down Expand Up @@ -350,7 +358,7 @@ those can be joined with a "-", like "red-bold", "bold-170", "bold-underline", e

## PPRINT-only flags

These are flags which are applicable to PPRINT output format.
These are flags which are applicable to PPRINT format.


**Flags:**
Expand Down
108 changes: 64 additions & 44 deletions internal/pkg/cli/option_parse.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ var FLAG_TABLE = FlagTable{
&FileFormatFlagSection,
&FormatConversionKeystrokeSaverFlagSection,
&CSVTSVOnlyFlagSection,
&JSONOnlyFlagSection,
&PPRINTOnlyFlagSection,
&CompressedDataFlagSection,
&CommentsInDataFlagSection,
Expand Down Expand Up @@ -405,11 +406,73 @@ var SeparatorFlagSection = FlagSection{
},
}

// ================================================================
// JSON-ONLY FLAGS

func JSONOnlyPrintInfo() {
fmt.Println("These are flags which are applicable to JSON output format.")
}

func init() { JSONOnlyFlagSection.Sort() }

var JSONOnlyFlagSection = FlagSection{
name: "JSON-only flags",
infoPrinter: JSONOnlyPrintInfo,
flags: []Flag{

{
name: "--jvstack",
help: "Put one key-value pair per line for JSON output (multi-line output). This is the default for JSON output format.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
options.WriterOptions.JSONOutputMultiline = true
*pargi += 1
},
},

{
name: "--no-jvstack",
help: "Put objects/arrays all on one line for JSON output. This is the default for JSON Lines output format.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
options.WriterOptions.JSONOutputMultiline = false
*pargi += 1
},
},

{
name: "--jlistwrap",
altNames: []string{"--jl"},
help: "Wrap JSON output in outermost `[ ]`. This is the default for JSON output format.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
options.WriterOptions.WrapJSONOutputInOuterList = true
*pargi += 1
},
},

{
name: "--no-jlistwrap",
help: "Wrap JSON output in outermost `[ ]`. This is the default for JSON Lines output format.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
options.WriterOptions.WrapJSONOutputInOuterList = false
*pargi += 1
},
},

{
name: "--jvquoteall",
help: "Force all JSON values -- recursively into lists and object -- to string.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
options.WriterOptions.JVQuoteAll = true
*pargi += 1
},
},
},
}

// ================================================================
// PPRINT-ONLY FLAGS

func PPRINTOnlyPrintInfo() {
fmt.Println("These are flags which are applicable to PPRINT output format.")
fmt.Println("These are flags which are applicable to PPRINT format.")
}

func init() { PPRINTOnlyFlagSection.Sort() }
Expand Down Expand Up @@ -489,12 +552,6 @@ var LegacyFlagSection = FlagSection{
parser: NoOpParse1,
},

{
name: "--jvquoteall",
help: "Type information from JSON input files is now preserved throughout the processing stream.",
parser: NoOpParse1,
},

{
name: "--json-fatal-arrays-on-input",
help: "Miller now supports arrays as of version 6.",
Expand Down Expand Up @@ -544,43 +601,6 @@ var LegacyFlagSection = FlagSection{
help: "Ignored as of version 6. Types are inferred/retained through the processing flow now.",
parser: NoOpParse1,
},

{
name: "--jvstack",
help: "Put one key-value pair per line for JSON output (multi-line output). This is the default for JSON output format.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
options.WriterOptions.JSONOutputMultiline = true
*pargi += 1
},
},

{
name: "--no-jvstack",
help: "Put objects/arrays all on one line for JSON output. This is the default for JSON Lines output format.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
options.WriterOptions.JSONOutputMultiline = false
*pargi += 1
},
},

{
name: "--jlistwrap",
altNames: []string{"--jl"},
help: "Wrap JSON output in outermost `[ ]`. This is the default for JSON output format.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
options.WriterOptions.WrapJSONOutputInOuterList = true
*pargi += 1
},
},

{
name: "--no-jlistwrap",
help: "Wrap JSON output in outermost `[ ]`. This is the default for JSON Lines output format.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
options.WriterOptions.WrapJSONOutputInOuterList = false
*pargi += 1
},
},
},
}

Expand Down
1 change: 1 addition & 0 deletions internal/pkg/cli/option_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ type TWriterOptions struct {
// JSON Lines output: --jlistwrap off, --jvstack off
WrapJSONOutputInOuterList bool // --jlistwrap
JSONOutputMultiline bool // --jvstack
JVQuoteAll bool // --jvquoteall
// Not using miller/types enum to avoid package cycle

// When we read things like
Expand Down
7 changes: 7 additions & 0 deletions internal/pkg/mlrval/mlrmap_json.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,3 +168,10 @@ func (entry *MlrmapEntry) JSONParseInPlace() {
entry.Value = ERROR
}
}

// StringifyValuesRecursively is nominally for the `--jvquoteall` flag.
func (mlrmap *Mlrmap) StringifyValuesRecursively() {
for pe := mlrmap.Head; pe != nil; pe = pe.Next {
pe.Value.StringifyValuesRecursively()
}
}
19 changes: 19 additions & 0 deletions internal/pkg/mlrval/mlrval_output.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,3 +103,22 @@ func (mv *Mlrval) setPrintRep() {
mv.printrepValid = true
}
}

// StringifyValuesRecursively is nominally for the `--jvquoteall` flag.
func (mv *Mlrval) StringifyValuesRecursively() {
switch mv.mvtype {

case MT_ARRAY:
for i, _ := range mv.arrayval {
mv.arrayval[i].StringifyValuesRecursively()
}

case MT_MAP:
for pe := mv.mapval.Head; pe != nil; pe = pe.Next {
pe.Value.StringifyValuesRecursively()
}

default:
mv.SetFromString(mv.String())
}
}
Loading