Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Spike] Harvesting pipeline in SSB #4216

Closed
11 of 17 tasks
nickumia-reisys opened this issue Feb 19, 2023 · 2 comments
Closed
11 of 17 tasks

[Spike] Harvesting pipeline in SSB #4216

nickumia-reisys opened this issue Feb 19, 2023 · 2 comments

Comments

@nickumia-reisys
Copy link
Contributor

nickumia-reisys commented Feb 19, 2023

Purpose

We want to test creating a brokerpak to provsion harvesting micro-services and have an external app facilitate triggers and connections, but we're not sure if it would work.

Given above question, conducting design + investigation + prototyping is needed to provide factual knowledge on future steps.

4 days of effort has been allocated and once compete, findings will be demonstrated and specific future actions will be decided.

Acceptance Criteria

  • GIVEN a queue pipeline has been created
    WHEN 4 days expires
    THEN the pipeline is shown via 0.5 hour presentation
    AND a recommendation is made on this approach for catalog harvesting

Background

Harvesting 2.0 proposed architecture

Sketch

Pre-work assumptions:

  • A new repository is created, specifically for this work (which may be thrown away if we decide later to abandon this approach).
  • Access to ssb-development account (AWS Credentials)

General Workflow:

  • Copy a skeleton of the brokerpak (either from an existing (solr) (eks) (smtp) or the template)
    • Full Disclosure: the template one may be too outdated for it to matter
  • Create an SQS Service (as brokerpak plan)
  • Create a Lambda Function (as brokerpak plan)
  • Option 1:
    • Provision SQS
    • Provision Lambda function
    • Create SQS "binding"
    • Create Lamdba "binding"
    • Create a local script that connects the SQS binding to Lambda
    • Create a local script that puts things into SQS and checks outputs from the Lambda function
  • Option 2:
    • Provision SQS
    • Create SQS "binding"
    • Provision Lambda function (maybe with the SQS Binding as input?)
    • Create Lambda "binding"
    • Create local script that puts things into SQS and checks outputs from the Lambda function

(There are probably more options, but these two seem the most viable at the time of writing)

@nickumia-reisys
Copy link
Contributor Author

Preliminary Notes/Observations:

  • The Proof of Concept works but the design still leaves a lot of questions about implementation..
    • What does a "bind" for an SQS instance look like? (e.g. a restrictive policy on who can talk to it? a mapping/connection to another service? both? something else?)
    • What does a "bind" for a lambda function look like? (e.g. do we want to make an API gateway in front of the lambda function to interact with it and supply credentials there? do we want to have no binding and the function has everything it needs from the input to do its job?)
    • What are all of the inputs/outputs for each service?
      • SQS --> Lambda
      • Lambda --> SQS?
      • S3 --> SQS?
      • S3 --> Lambda?
      • Lambda --> S3
    • What other services do we need to implement?
  • Overall, I think that this paradigm is powerful and robust, but there are quite a few options we'd need to design for and make available for proper customization when actually utilizing these services.

I will post an update shortly with the example test case as a reference.

@nickumia-reisys nickumia-reisys moved this from 🏗 In Progress [8] to 👀 Needs Review [2] in data.gov team board Mar 10, 2023
@nickumia-reisys nickumia-reisys moved this from 👀 Needs Review [2] to 🏗 In Progress [8] in data.gov team board Mar 13, 2023
@nickumia-reisys
Copy link
Contributor Author

General overview of work and place to put additional notes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

No branches or pull requests

1 participant