|
| 1 | +# Metadata examples |
| 2 | + |
| 3 | +Custom application looks for new files in particular folders an S3 bucket and interacts with the Metadata API based on the data in the delimited file |
| 4 | + |
| 5 | +## Use cases |
| 6 | + |
| 7 | +1. Obtained list of current OCLC Numbers based on a given OCLC Number |
| 8 | +2. Obtain list of merged OCLC Numbers based on a given OCLC Number |
| 9 | +3. Set Holdings based on a given OCLC Number |
| 10 | +4. Delete Holdings based on a given OCLC Number |
| 11 | +5. Add an LBD based on a given OCLC Number and 500 note data |
| 12 | + |
| 13 | +## Installing Locally |
| 14 | + |
| 15 | +### Step 1: Clone the repository |
| 16 | +Clone this repository |
| 17 | + |
| 18 | +```bash |
| 19 | +$ git clone {url} |
| 20 | +``` |
| 21 | +or download directly from GitHub. |
| 22 | + |
| 23 | +Change into the application directory |
| 24 | + |
| 25 | +### Step 2: Setup Virtual Environment |
| 26 | + |
| 27 | +```bash |
| 28 | +$ python -m venv venv |
| 29 | +$ . venv/bin/activate |
| 30 | +``` |
| 31 | + |
| 32 | +### Step 3: Install python dependencies |
| 33 | + |
| 34 | +```bash |
| 35 | +$ pip install -r requirements.txt |
| 36 | +``` |
| 37 | + |
| 38 | +### Step 4: Run local tests |
| 39 | + |
| 40 | +```bash |
| 41 | +$ python -m pytest |
| 42 | +``` |
| 43 | + |
| 44 | +### Step 5: Run code locally |
| 45 | +```bash |
| 46 | +usage: processSheet.py [-h] --itemFile ITEMFILE --operation |
| 47 | + {getCurrentOCLCNumbers, retrieveMergedOCLCNumbers, setHoldingsbyOCLCNumber, deleteHoldingsbyOCLCNumber, addLBDs} |
| 48 | + --outputDir OUTPUTDIR |
| 49 | + |
| 50 | +optional arguments: |
| 51 | + -h, --help show this help message and exit |
| 52 | + --itemFile ITEMFILE File you want to process |
| 53 | + --operation {getCurrentOCLCNumbers, retrieveMergedOCLCNumbers, setHoldingsbyOCLCNumber, deleteHoldingsbyOCLCNumber, addLBDs} |
| 54 | + Operation to run: getCurrentOCLCNumbers, |
| 55 | + retrieveMergedOCLCNumbers, setHoldingsbyOCLCNumber, |
| 56 | + deleteHoldingsbyOCLCNumber, addLBDs |
| 57 | + --outputDir OUTPUTDIR |
| 58 | + Directory to save output to |
| 59 | + |
| 60 | +``` |
| 61 | + |
| 62 | +#### Example |
| 63 | +```bash |
| 64 | +$ python processSheet.py --itemFile samples/oclc_numbers.csv --operation getCurrentOCLCNumbers --outputDir samples/getCurrentOCLCNumbers.csv |
| 65 | + |
| 66 | +$ python processSheet.py --itemFile samples/oclc_numbers_holdings.csv --operation retrieveMergedOCLCNumbers --outputDir samples/mergedOCLCNumbers.csv |
| 67 | + |
| 68 | +$ python processSheet.py --itemFile samples/sp_holdings.csv --operation setHoldingsbyOCLCNumber --outputDir samples/addedHoldings.csv |
| 69 | + |
| 70 | +$ python processSheet.py --itemFile samples/my_retentions.csv --operation deleteHoldingsbyOCLCNumber --outputDir samples/removedHoldings.csv |
| 71 | + |
| 72 | +$ python processSheet.py --itemFile samples/symbol_retentions.csv --operation addLBDs --outputDir samples/newLBDs.csv |
| 73 | +``` |
| 74 | + |
| 75 | +## Running in AWS Lambda |
| 76 | + |
| 77 | +### Step 1: Use npm to install dependencies needed to deploy code |
| 78 | +Download node and npm and use the `install` command to read the dependencies JSON file |
| 79 | + |
| 80 | +```bash |
| 81 | +$ npm install |
| 82 | +``` |
| 83 | + |
| 84 | +### Step 2: AWS Setup |
| 85 | + |
| 86 | +1. Install AWS Command line tools |
| 87 | +- https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html |
| 88 | +I recommend using pip. |
| 89 | +2. Create an AWS user in IAM console. Give it appropriate permissions. Copy the key and secret for this user to use in the CLI. |
| 90 | +3. Configure the command line tools - https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html |
| 91 | + |
| 92 | +- Make sure you add |
| 93 | +-- key/secret |
| 94 | +-- region |
| 95 | + |
| 96 | +### Step 3: Create an S3 Bucket for the files |
| 97 | +1. Use the AWS Console to create a bucket. Note your bucket name!!! |
| 98 | +2. Create folder metadata_tasks/ |
| 99 | +3. Add a sample csv file named getCurrentOCLCNumbers.csv with data to check for current OCLC Numbers |
| 100 | +4. Add a sample csv file named getMergedOCLCNumbers.csv with data to check for merged OCLC Numbers |
| 101 | +5. Add a sample csv file named addHoldings.csv with data to add holdings by OCLC Number |
| 102 | +6. Add a sample csv file named deleteHoldings.csv with data delete holdings by OCLC Number |
| 103 | +7. Add a sample csv file named LBDsToAdd.csv with data to add LBD records to OCLC Numbers |
| 104 | + |
| 105 | + |
| 106 | +### Step 4: Test application locally |
| 107 | +1. Alter s3-getCurrentOCLCNumbers.json to point to your bucket and your sample txt file. |
| 108 | + |
| 109 | +2. Use serverless to test locally |
| 110 | + |
| 111 | +```bash |
| 112 | +$ serverless invoke local --function getCurrentOCLCNumbers --path s-getCurrentOCLCNumbers.json |
| 113 | +``` |
| 114 | + |
| 115 | +3. Alter s3-getMergedOCLCNumbers.json to point to your bucket and your sample csv file. |
| 116 | + |
| 117 | +4. Use serverless to test locally |
| 118 | + |
| 119 | +```bash |
| 120 | +$ serverless invoke local --function getMergedOCLCNumbers --path s3-getMergedOCLCNumbers.json |
| 121 | +``` |
| 122 | + |
| 123 | +5. Alter s3-addHoldings.json to point to your bucket and your sample csv file. |
| 124 | + |
| 125 | +6. Use serverless to test locally |
| 126 | + |
| 127 | +```bash |
| 128 | +$ serverless invoke local --function addHoldings --path s3-addHoldings.json |
| 129 | +``` |
| 130 | + |
| 131 | +7. Alter s3-deleteHoldings.json to point to your bucket and your sample csv file. |
| 132 | + |
| 133 | +8. Use serverless to test locally |
| 134 | + |
| 135 | +```bash |
| 136 | +$ serverless invoke local --function deleteHoldings --path s3-deleteHoldings.json |
| 137 | +``` |
| 138 | + |
| 139 | +9. Alter s3-addLBDs.json to point to your bucket and your sample csv file. |
| 140 | + |
| 141 | +10. Use serverless to test locally |
| 142 | + |
| 143 | +```bash |
| 144 | +$ serverless invoke local --function addLBDs --path s3-addLBDs.json |
| 145 | +``` |
| 146 | + |
| 147 | +### Step 5: Deploy the code using serverless |
| 148 | + |
| 149 | +```bash |
| 150 | +$ serverless deploy |
| 151 | +``` |
0 commit comments