Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PathRow and Sensor as Parameters #8

Open
crh3675 opened this issue Jul 23, 2020 · 6 comments
Open

PathRow and Sensor as Parameters #8

crh3675 opened this issue Jul 23, 2020 · 6 comments

Comments

@crh3675
Copy link

crh3675 commented Jul 23, 2020

Allow a call to send a path/row in lieu of a BBox as well as specify sensor (LS4, LS5, LS7, LS8). I built this same exact solution in NodeJS and love that I found a Python alternative. I also added in the Search capability to find the scenes to order from https://earthexplorer.usgs.gov/inventory/json/v/1.4.0/search.

@loicdtx
Copy link
Owner

loicdtx commented Jul 24, 2020

Thanks for the suggestion, I can have a look at how to implement that.
You say you've done it in javascript already; can you send an example of json query with path/row filtering.
Filtering by sensor is already implemented, and even required I think.

@crh3675
Copy link
Author

crh3675 commented Jul 24, 2020

Here is some scene search code:

const request = require('request');
const moment = require('moment');
//require('request-debug')(request);

/**
 * Search scenes from USGS
 * @class
 */
class SceneSearch {
  constructor(outputType = 0) {
    this.outputType = outputType;
    this.authToken = null;
    this.userName = null;
    this.userPass = null;
  }

  setAuth(userName, userPass) {
    this.userName = userName;
    this.userPass = userPass;
  }

  /**
   * Query USGS scene dataset by pathrow for matching Landsast scenes
   * @param {Object} pathRow
   * @param {String} dataSet - LS4, LS5, LS7, LS8
   * @param {String} startDate - YYYY-MM-DD
   * @param {String} endDate = YYYY-MM-DD
   * @param {Number} maxCloudCover - 1 - 100
   * @return {Promise}
   */
  query(pathRow, dataSet, startDate = new Date(), endDate = new Date(), maxCloudCover = 10) {
    return new Promise((resolve, reject) => {
      return this.login().then(() => {
        const searchUrl = 'https://earthexplorer.usgs.gov/inventory/json/v/1.4.0/search';
        const dataset = SceneSearch.datasets[dataSet];

        startDate = moment(Date.parse(startDate)).format('YYYY-MM-DD[T]00:00:00');
        endDate = moment(Date.parse(endDate)).format('YYYY-MM-DD[T]23:59:59');

        if (!dataset) {
          return reject(new Error('Invalid dataset specified'));
        }

        const childFilters = [];

        for (const fieldId in dataset.criteria) {
          const param = dataset.criteria[fieldId];
          const childFilter = {
            filterType: 'value',
            fieldId: +fieldId,
            value: param.value,
            operand: '='
          };

          // Swap out placeholder values
          Object.keys(pathRow).forEach(prop => {
            if (String(childFilter.value).match(/^:/)) {
              childFilter.value = childFilter.value.replace(`:${prop}`, pathRow[prop]);
            }
          });
          childFilters.push(childFilter);
        }

        const payload = {
          apiKey: this.authToken,
          maxResults: 300,
          sortOrder: 'DESC',
          datasetName: dataset.collection,
          temporalFilter: {
            startDate: startDate,
            endDate: endDate
          },
          includeUnknownCloudCover: true,
          additionalCriteria: {
            filterType: 'and',
            childFilters: childFilters
          }
        };

        if (maxCloudCover) {
          payload.maxCloudCover = maxCloudCover;
        }

        const options = {
          url: searchUrl,
          followAllRedirects: true,
          formData: {
            jsonRequest: JSON.stringify(payload)
          },
          json: true
        };

        request.post(options, (err, httpResponse, body) => {
          if (err) {
            return reject(err);
          }
          if (httpResponse.statusCode !== 200) {
            return reject(`API returned ${httpResponse.statusCode} for ${dataSet}`);
          }

          let scenes = [];

          if (body.data && body.data.results) {
            scenes = body.data.results.map(result => result.displayId);
          } else {
            console.error(`Status was 200 but no scenes were available`);
            console.log(httpResponse)
          }

          if (this.outputType === SceneSearch.outputList) {
            resolve(scenes);
          } else if (this.outputType > 0) {
            const calendar = {};

            scenes.forEach(sceneId => {
              const sceneParts = sceneId.split('_');
              const year = sceneParts[3].substr(0, 4);
              const month = +(sceneParts[3].substr(4, 2)) - 1;

              if (!calendar.hasOwnProperty(year)) {
                calendar[year] = [...Array(12)].map(e => []);
              }
              if (this.outputType === SceneSearch.outputOnePerByYearMonth && calendar[year][month].length) {
                return;
              }
              calendar[year][month].push(sceneId);
            });
            resolve(calendar);
          }
        })
      });
    })
  }

  /**
   * Login to earthexplorer
   * @param {String} userName
   * @param {String} userPass
   * @return {Promise}
   */
  login(userName, userPass) {
    userName = userName || this.userName;
    userPass = userPass || this.userPass;

    return new Promise((resolve, reject) => {
      if (this.authToken) {
        return resolve(this.authToken);
      }
      const authUrl = 'https://earthexplorer.usgs.gov/inventory/json/v/1.4.0/login';
      const options = {
        url: authUrl,
        formData: {
          jsonRequest: JSON.stringify({
            username: userName,
            password: userPass
          })
        },
        json: true
      };

      request.post(options, (err, httpResponse, body) => {
        if (err) {
          return reject(err);
        }
        this.authToken = body.data;
        resolve(this.authToken);
      })
    });
  }
}

SceneSearch.outputList = 0;
SceneSearch.outputByYearMonth = 1;
SceneSearch.outputOnePerByYearMonth = 2;

SceneSearch.datasets = {
  LS4: {
    collection: 'LANDSAT_TM_C1',
    ordering: 'tm4_collection',
    criteria: {
      '25173': {
        description: 'satellite',
        value: '4'
      },
      '21989': {
        description: 'path',
        value: ':path'
      },
      '19879': {
        description: 'row',
        value: ':row',
      },
      '19880': {
        description: 'category',
        value: ':tier'
      }
    }
  },
  LS5: {
    collection: 'LANDSAT_TM_C1',
    ordering: 'tm5_collection',
    criteria: {
      '25173': {
        description: 'satellite',
        value: '5'
      },
      '21989': {
        description: 'path',
        value: ':path'
      },
      '19879': {
        description: 'row',
        value: ':row',
      },
      '19880': {
        description: 'category',
        value: ':tier'
      }
    }
  },
  LS7: {
    collection: 'LANDSAT_ETM_C1',
    ordering: 'etm7_collection',
    criteria: {
      '19884': {
        description: 'path',
        value: ':path'
      },
      '19887': {
        description: 'row',
        value: ':row',
      },
      '19890': {
        description: 'category',
        value: ':tier'
      }
    }
  },
  LS8: {
    collection: 'LANDSAT_8_C1',
    ordering: 'olitirs8_collection',
    criteria: {
      '20514': {
        description: 'path',
        value: ':path'
      },
      '20516': {
        description: 'row',
        value: ':row',
      },
      '20510': {
        description: 'category',
        value: ':tier'
      }
    }
  }
}

module.exports = SceneSearch;

You can invoke using something like the following:

const search = new SceneSearch();
search.setAuth(userName, userPass);
search.query(criteria.pathRow, 'LS7', criteria.startDate, criteria.endDate, criteria.maxCloudCover).then(data => {
  // do something with results
}).catch(err => {
  // do something with error
});

As well, might also add in ability to specify multiple products: source_metadata, sr, bt, pixel_qa, toa, stats

@loicdtx
Copy link
Owner

loicdtx commented Jul 24, 2020

Thanks for that, it looks great. Though my understanding of javascript is quite limited. Do you think you could send an example of the json being sent?

@crh3675
Copy link
Author

crh3675 commented Jul 24, 2020

{
  "apiKey": null,
  "maxResults": 300,
  "sortOrder": "DESC",
  "datasetName": "LANDSAT_8_C1",
  "temporalFilter": {
    "startDate": "1969-12-31T00:00:00",
    "endDate": "2020-07-24T23:59:59"
  },
  "includeUnknownCloudCover": true,
  "additionalCriteria": {
    "filterType": "and",
    "childFilters": [
      {
        "filterType": "value",
        "fieldId": 20510,
        "value": "T1",
        "operand": "="
      },
      {
        "filterType": "value",
        "fieldId": 20514,
        "value": "97",
        "operand": "="
      },
      {
        "filterType": "value",
        "fieldId": 20516,
        "value": "65",
        "operand": "="
      }
    ]
  },
  "maxCloudCover": 20
}

Note that the fieldId input parameters are different per sensor so you need a dictionary to map them against. They come from their API but easier to have a dictionary than to query the API for each value.

@crh3675
Copy link
Author

crh3675 commented Jul 24, 2020

Unfortunately that API is having errors with responses for some reason

@loicdtx
Copy link
Owner

loicdtx commented Jul 24, 2020

That's very helpful. Thanks @crh3675 !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants