Previously valid control-file entries cause 1.5.4 to fail

[levyj]

What is the reason for the "Unknown control file fields were allowed and ignored; we're now restricting control files to have only those fields DataSync supports" change in 1.5.4? This change causes previously functional automated jobs to fail, requiring editing of the control files or not upgrading DataSync. Unless this change serves an important purpose, would you consider rolling it back? Returning a warning but proceeding with the job seems fine.

Even if important for some reason, would you consider revising your numbering scheme and use of warnings about backwards compatibility? A change that breaks existing jobs probably should carry a bold warning and a version number change higher than the third level.

Thank you.

/cc @tomschenkjr

[rjmac]

The reason was to make control files that contained erroneous fields (particularly ones that were being used by automated jobs) demand a human's attention instead of quietly doing a potentially wrong thing. You're probably right that it should have required a major version release, though. The server now also rejects control files with unknown fields, but the DataSync app sends control files that have been processed on the user's side so those fields would have already been removed by the time it gets there.

[levyj]

I can see some logic in that but it does create a problem because it breaks things that used to work. Particularly when the lines were previously valid -- even included in Socrata-provided control file templates -- might that be taking it farther than necessary? Would you consider permitting lines that were previously valid and only failing on lines that were never valid?

-----Original Message-----
From: rjmac [notifications@github.com]
Received: Wednesday, 25 Feb 2015, 7:08PM
To: socrata/datasync [datasync@noreply.github.com]
CC: Levy, Jonathan [Jonathan.Levy@cityofchicago.org]
Subject: Re: [datasync] Previously valid control-file entries cause 1.5.4 to fail (#93)

The reason was to make control files that contained erroneous fields (particularly ones that were being used by automated jobs) demand a human's attention instead of quietly doing a potentially wrong thing. You're probably right that it should have required a major version release, though. The server now also rejects control files with unknown fields, but the DataSync app sends control files that have been processed on the user's side so those fields would have already been removed by the time it gets there.

Reply to this email directly or view it on GitHubhttps://github.com//issues/93#issuecomment-76103561.

---

This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail (or the person responsible for delivering this document to the intended recipient), you are hereby notified that any dissemination, distribution, printing or copying of this e-mail, and any attachment thereto, is strictly prohibited. If you have received this e-mail in error, please respond to the individual sending the message, and permanently delete the original and any copy of any e-mail and printout thereof.

[levyj]

In case this is helpful to anyone, Socrata confirmed to me in a separate conversation on this topic that the only control file entry previously included in the template by Socrata that is now considered unknown (and therefore is not accepted in DataSync 1.5.4 and later) is ignoreServerLatLong.