Skip to content

Latest commit

 

History

History
75 lines (55 loc) · 4.55 KB

bulk_download_using_curl.md

File metadata and controls

75 lines (55 loc) · 4.55 KB
Raw

How to bulk download LP DAAC data using Curl

This guide shows how to bulk download LP DAAC data using cURL from the command line. cURL is a free and open source software developed under Client for URLs (cURL) project as a command line tool for transfering data using URLs.

Requirements:

Step 1: Save the Download Link(s)

Save download links for your data as a text file using Nasa Earthdata Search or Common Metadata Repository (CMR) API. Follow the steps in the Earthdata Search guide to find your data and save the download links. If you prefer to use an API to find your data and save the download links, a tutorial on how to use the CMR API can be found here.

Step 2: Set up a .netrc file for Authentication

Set up a .netrc file in your home directory.

  • Manual set up

    • Download the .netrc template file and save it in your home directory.
      • Open the .netrc file in a text editor and replace <USERNAME> with your NASA Earthdata Login username and <PASSWORD> with your NASA Earthdata Login password.

  • Create .netrc file from the Command Line

    • Enter the following in Terminal:

      • Windows

        To Create a .netrc file, enter the following in the command line, replacing and with your NASA Earthdata username and password. This will create a file in your home directory or append your NASA credentials to an existing file.

        echo machine urs.earthdata.nasa.gov login <USERNAME> password <PASSWORD> >> %userprofile%\.netrc

      • Linux/MacOS:

        To Create a .netrc file, enter the following in the command line, replacing and with your NASA Earthdata username and password. This will create a file in your home directory or append your NASA credentials to an existing file.

        echo "machine urs.earthdata.nasa.gov login <USERNAME> password <PASSWORD>" >>~/.netrc

  • Programmatically:

Step 3: Download LP DAAC Data

You should now be able to run the command to download data directly from the LP DAAC.

  • Navigate to the directory you want to save the data using cd Insert_Your_Directory.

  • To download a single file, replace the Insert_the_Download_Link in the command below with the URL to the data file you wish to download.

    curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n Insert_the_Download_Link

    Example:

    curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T12SWF.2023189T175919.v2.0/HLS.S30.T12SWF.2023189T175919.v2.0.B08.tif

  • To download multiple files, replace Insert_Text_File in the command below with the full path to the text file saved previously in Step 1.

    xargs -n 1 curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n < Insert_Text_File

    Example:

     xargs -n 1 curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n < data/Granule-DownloadLinks.txt

Useful Links:

Contact Info:

Email: LPDAAC@usgs.gov
Voice: +1-866-573-3222
Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹
Website: https://lpdaac.usgs.gov/
Date last modified: 07-11-2023

¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I.