DICOM Redaction Updates

[naglepuff]

Fix #182

Changes

1. Additional DICOM metadata rules have been added based on tags found in NEMA data that were not previously supported.
2. Less reliance on file extension to try and redact files, more emphasis on checking the file type. Previously, DICOM files without the .dcm extension failed to be redacted.
3. Custom metadata fields found in DICOM images are deleted by default. Rules overrides can change this behavior.

[naglepuff]

Note that a0346cc is necessary due to psf/black#3887

[manthey]

Should we suppress UserWarnings from pydicom, since we aren't going to do anything about them? In the test files, I get pydicom/valuerep.py:443: UserWarning: Invalid value for VR UI: '1.3.6.1.4.1.36533.0106114253531497724413917111421208341787'. Please see <https://dicom.nema.org/medical/dicom/current/output/html/part05.html#table_6.2-1> for allowed values for each VR. Ironically, as far as I can tell from that link this UI is conformant to what they say, though I suspect the chunk with the leading 0 is the complaint.

[manthey]

I get a pydantic error when I try to use the following rule set override:

---
dicom:
  delete_custom_metadata: false

Pydantic says Field required [type=missing, input_value={'dicom': {'delete_custom_metadata': False}}, input_type=dict], but doesn't say what field is required.

[manthey]

I get a pydantic error when I try to use the following rule set override:

---
dicom:
  delete_custom_metadata: false

Pydantic says Field required [type=missing, input_value={'dicom': {'delete_custom_metadata': False}}, input_type=dict], but doesn't say what field is required.

Aside from pydantic having terrible error messages, we should make it so that a custom rule set has as little required as possible (it can be a separate PR).

[manthey]

Should this be slightly later in the plan: That is, if there is a rule for a specific custom metadata tag, that should take priority over the keep/delete custom metadata.

[naglepuff]

I can definitely move it. But a good point that this raises is that we're relying onpydicom to get a tag's name, which doesn't work for custom metadata. This means we'll need to also support checking by tag.

[naglepuff]

I get a pydantic error when I try to use the following rule set override:

---
dicom:
  delete_custom_metadata: false

Pydantic says Field required [type=missing, input_value={'dicom': {'delete_custom_metadata': False}}, input_type=dict], but doesn't say what field is required.

Aside from pydantic having terrible error messages, we should make it so that a custom rule set has as little required as possible (it can be a separate PR).

Was that the entirety of your rules override file? In that case, the errors are looking for all of the other properties of a Rule Set (name, description, tiff, etc.), since none of it is optional.

It would be easy enough to make those things optional as part of this PR.

For what its worth, I did end up seeing exactly what fields pydantic was complaining about.

There was actually a bug that would have caused that property to not get overridden, so I will address that here.

[manthey]

I think this ultimately should have three states, delete custom metadata, keep custom metadata, or do not have auto rules on custom metadata. This could be a future PR, though. Maybe it is a pair of booleans: delete_custom_metadata and keep_custom_metadata which would then make no sense if both true.