Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

copernicus data provider #972

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open

copernicus data provider #972

wants to merge 24 commits into from

Conversation

michaelmattig
Copy link
Contributor

  • I added an entry to CHANGELOG.md if knowledge of this change could be valuable to users.

Here is a brief summary of what I did:

@coveralls
Copy link
Collaborator

coveralls commented Jul 19, 2024

Pull Request Test Coverage Report for Build 11482121846

Details

  • 742 of 1394 (53.23%) changed or added relevant lines in 20 files are covered.
  • 12 unchanged lines in 8 files lost coverage.
  • Overall coverage decreased (-0.3%) to 90.672%

Changes Missing Coverage Covered Lines Changed/Added Lines %
operators/src/util/gdal.rs 1 2 50.0%
services/src/pro/layers/add_from_directory.rs 0 1 0.0%
services/src/util/mod.rs 26 27 96.3%
datatypes/src/plots/mod.rs 7 9 77.78%
expression/src/dependencies.rs 12 15 80.0%
services/src/util/tests.rs 0 7 0.0%
services/src/pro/datasets/external/mod.rs 2 11 18.18%
services/src/pro/datasets/external/copernicus_dataspace/stac.rs 140 166 84.34%
services/src/pro/datasets/external/copernicus_dataspace/sentinel2.rs 392 434 90.32%
services/src/pro/datasets/external/copernicus_dataspace/ids.rs 35 292 11.99%
Files with Coverage Reduction New Missed Lines %
services/src/api/handlers/layers.rs 1 88.81%
services/src/api/handlers/datasets.rs 1 94.74%
datatypes/src/plots/mod.rs 1 63.64%
services/src/datasets/external/netcdfcf/mod.rs 1 95.43%
services/src/datasets/external/netcdfcf/ebvportal_api.rs 1 96.97%
services/src/datasets/external/netcdfcf/ebvportal_provider.rs 2 93.14%
services/src/pro/api/handlers/machine_learning.rs 2 93.59%
services/src/pro/contexts/postgres.rs 3 96.76%
Totals Coverage Status
Change from base Build 11461162197: -0.3%
Covered Lines: 134402
Relevant Lines: 148229

💛 - Coveralls

@michaelmattig
Copy link
Contributor Author


// TODO: are these bands for all sentinel 2 products?
#[derive(Debug, Clone, Copy, EnumString, strum::Display, EnumIter)]
pub enum Sentinel2Band {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we also need the SCL band. Without that we can't do cloud free

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but which resolution? See my other comment, it is available in 2 subdatasets/resolution.

B08,
B08A,
B09,
// B10,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not include B10?
edit: sooooo B10 is in the L1 product but not in the L12 product beacause it is needed for correction only. Can you add that and maybe separate between products?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

// B10,
B11,
B12,
// TODO: AOT, CLD, etc. but they have multiple resolutions
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for geo engine we need 3 stacked layers for 10m, 20m and 60m. Are there really bands with multiple resolutions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AOT, CLD, SCL, SNW, WVP are contained in both subdataset 2 with 20m resolution and subdataset 3 with 60m resolution. idk what to do about that.


#[allow(clippy::unused_self)] // might need to be used in the future to distinguish between bands
pub fn data_type(self) -> RasterDataType {
RasterDataType::U16
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SCL is u8 but thats not really a problem (yet)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you sure?

Band 5 Block=128x128 Type=UInt16, ColorInterp=Undefined
Description = SCL, Scene Classification
Categories:
0: NODATA
1: SATURATED_DEFECTIVE
2: DARK_FEATURE_SHADOW
3: CLOUD_SHADOW
4: VEGETATION
5: NOT_VEGETATED
6: WATER
7: UNCLASSIFIED
8: CLOUD_MEDIUM_PROBA
9: CLOUD_HIGH_PROBA
10: THIN_CIRRUS
11: SNOW_ICE
Metadata:
BANDNAME=SCL

}

#[derive(Debug, Clone)]
pub struct Sentinel2Metadata {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe name it CopernicusSentinel2Metadata or something like this because there might be more Sentinel2 sources?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the code is in a submodule, so it should be fine? If we have multiple sources we can use the qualified name

self.zone.epsg_code(),
)
.into(),
time: None, // TODO: specify time (2015, open end/current date)?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

)
.into(),
time: None, // TODO: specify time (2015, open end/current date)?
bbox: None, // TODO: exclude parts that are never visited by the satellite?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use the UTM Zone box here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants