flag invalid data

[HolgerPollyNet]

We might add the option to flag invalid periods of data posteriori .

E.g. for Punta Arenas, we found out that all data between 12. and 24. June 2020 is invalid, as the overlap was completely out.

It would be great if this could be flagged somehow, but maybe more an issue for the data base @rico-hengst ?

[ZPYin]

For internal users, you can only have a knowledge of the lidar status when you see the profiles, quicklooks or housekeeping data. Therefore, I think it's still necessary to process these invalid data for Picasso.

So I agree that this issue should be handled by pollyAPP, since it is the window towards external users.

Ps,

We should move this issue to Discussion session.

[HolgerPollyNet]

@ZPYin , I think the discussion in not over yet ;-). For example, @martin-rdz wants to harvest large datasets with LARDA but does not want to interact with the data base itself. So for sure, an additional quality screening is needed so that netcdf files with low quality data are somehow flagged.

[ZPYin]

The systematic issues can hardly be identified by lidar evaluation algorithms, for instance, the overlap issue. That's to say, (so far) we can only tag such low-quality data through the assist of user inputs. Unfortunately, I don't have a decent way to implement this feature yet.

Do you know how such low-quality data were flagged in EARLINET SCC? They should have much more experience in dealing with flawed data...

Or any suggestions? @HolgerPollyNet

[HolgerPollyNet]

There is nothing like this in the SCC.

I still would think a data base based approach would be most convenient, or something like the standardized log book but extended functionalities.