Unparse skips columns

[silvestreh]

Looks like unparsing a JSON array ends up in missing columns if the first Object in the array is missing properties that other Objects in the same Array do have.

This Array is correctly turned into CSV:

[
  {
    date: '2018-12-20T17:04:41.446Z',
    amount: '30.03',
    metadata_customerEmail: 'someone@domain.com'
  },
  {
    date: '2018-12-20T17:01:30.434Z',
    amount: '13.06',
    metadata_customerEmail: 'someone@domain.com'
  },
  {
    date: '2018-12-20T16:49:55.630Z',
    amount: '31.33',
    metadata_customerEmail: 'someone@domain.com'
  },
  {
    date: '2018-12-20T16:33:50.121Z',
    amount: '29'
  }
]

This is the output:

date,amount,metadata_customerEmail
2018-12-20T17:04:41.446Z,30.03,someone@domain.com
2018-12-20T17:01:30.434Z,13.06,someone@domain.com
2018-12-20T16:49:55.630Z,31.33,someone@domain.com
2018-12-20T16:33:50.121Z,29,

Now, if you sort that array so the oldest entry (the one without metadata_customerEmail) is at index 0, then you end up with this CSV:

date,amount
2018-12-20T16:33:50.121Z,29
2018-12-20T16:49:55.630Z,31.33
2018-12-20T17:01:30.434Z,13.06
2018-12-20T17:04:41.446Z,30.03

[silvestreh]

Here's a thought: maybe Papa could make a first pass to determine what the columns should be and then a second one to grab all the data?

[pokoli]

I don't think we should double pass the results as this will be a performance issue but you may probably do it before passing the values to paparse. This way you can perform any additional checks or whatever you need.

Do you have some suggestion about how to improve the unparse of empty values? For me the current behavior is correct.

[MonkeyDZeke]

I would suggest we add a config variable for unparse called something like columns which accepts either an array (['key_1', 'key_2', 'key_3']) or an object ({old_key_1: 'new_key_1', old_key_2: 'new_key_2', old_key_3: 'new_key_3'}) so that the user can tell the unparser what to expect (and conveniently alter the resulting header row if desired).

Thoughts?

[dboskovic]

I agree with @pokoli that this is something that should happen outside of this library. I think the columns option with an array of keys passed to unparse would be valuable, both in determining output order and in solving this problem. Don't think we should do the old/new mapping though, that feels bloaty.

[MonkeyDZeke]

Makes sense. The resulting header row can be edited afterwards easily enough anyway.

[dboskovic]

Flagging this as a feature ready for contribution!

Help wanted summary:

• Add an option to unparse called columns that, when present, replaces the logic that determines the columns out of the keys for the first object.
• Update the documentation to include the details of how to use this option.
• Write tests that show this option being used to include columns not present in the keys of the first object.
• Make sure the serialized value of non-present keys is an empty string, not the word undefined or null.
• Make sure errors do not occur when attempting to serialize a key not present in a given object.

How to contribute

https://github.com/mholt/PapaParse#contributing

[pokoli]

Fixed on #632