[Issue]: Ingest forecasted opportunities

[TylerHendrickson]

Why is this issue important?

We want grants-ingest to maintain an index of forecasted opportunities, e.g. so that they can be searched for in downstream applications such as GOST.

Current State

Opportunities are only ingested starting at the point of being published as "Open", when they show up in the grants.gov DB archive as an <OpportunitySynopsisDetail_1_0> node.

Expected State

Forecasted opportunities, represented in the grants.gov DB archive as <OpportunityForecastDetail_1_0> nodes, are indexed as we do with opportunities in other (post-published as "Open") states.

Implementation Plan

1. Update the SplitGrantsGovXMLDB handler to split <OpportunityForecastDetail_1_0> from the grants.gov XML database snapshot.
2. Determine how we want to represent forecast-only fields and their values (there are additional fields prefixed with Estimated* that we don't currently have handling for).
3. Update the PublishGrantEvents handler to incorporate new forecast fields into published messages.

Relevant Code Snippets

Example of a forecasted opportunity, as represented by grants.gov XML database snapshots:

    <OpportunityForecastDetail_1_0>
        <OpportunityID>284765</OpportunityID>
        <OpportunityTitle>PPHF 2015: Immunization Grants-CDC Partnership: Strengthening Public Health Laboratories-fin
anced in part by 2015 Prevention and Public Health Funds</OpportunityTitle>
        <OpportunityNumber>CDC-RFA-OE15-150101SUPP16</OpportunityNumber>
        <OpportunityCategory>D</OpportunityCategory>
        <FundingInstrumentType>CA</FundingInstrumentType>
        <CategoryOfFundingActivity>HL</CategoryOfFundingActivity>
        <CFDANumbers>93.322</CFDANumbers>
        <EligibleApplicants>12</EligibleApplicants>
        <EligibleApplicants>13</EligibleApplicants>
        <EligibleApplicants>22</EligibleApplicants>
        <AdditionalInformationOnEligibility>Single-Source Eligibility for the Association of Public Health Laboratories (APHL).&amp;#160; </AdditionalInformationOnEligibility>
        <AgencyCode>HHS-CDC-CSELS</AgencyCode>
        <AgencyName>Centers for Disease Control - CSELS</AgencyName>
        <PostDate>04142016</PostDate>
        <LastUpdatedDate>04142016</LastUpdatedDate>
        <EstimatedSynopsisPostDate>04152016</EstimatedSynopsisPostDate>
        <FiscalYear>2016</FiscalYear>
        <EstimatedSynopsisCloseDate>05092016</EstimatedSynopsisCloseDate>
        <EstimatedSynopsisCloseDateExplanation>Electronically submitted applications must be submitted no later than 11:59 p.m., ET, on the listed application due date.</EstimatedSynopsisCloseDateExplanation>
        <EstimatedAwardDate>05292015</EstimatedAwardDate>
        <EstimatedProjectStartDate>07012015</EstimatedProjectStartDate>
        <AwardCeiling>0</AwardCeiling>
        <AwardFloor>0</AwardFloor>
        <EstimatedTotalProgramFunding>645000</EstimatedTotalProgramFunding>
        <ExpectedNumberOfAwards>1</ExpectedNumberOfAwards>
        <Description> CDC&amp;apos;s collaboration with the Association of Public Health Laboratories announces two supplemental funding opportunities. These funding opportunities will be awarded for Advanced Molecular Detection and Vision 2020 - Focus Group Discussions. The purpose of the Advanced Molecular Detection Project is to support state laborartory capacity building to improve characterization of the influenza vaccine viruses by a minimum of two weeks. This project will ensure that the prerequisite infrastructure is available for full production of genomic sequencing prior to the 2016 Influenza season. The anticipated funding for this activity is $620,000. The Vision 2020 Focus Group Discussions will support the Laborartory Systems and Standards Program to develop best practices and performance quality standards. Develop guidance and tools that assist Public Health Laboratories with developing and supporting regional networks and consortiums for effective sharing of testing and other services that will improve preparedness through established test-sharing mechanisms. The anticipated funding level for this activity is $25,000. </Description>
        <Version>Forecast 1</Version>
        <CostSharingOrMatchingRequirement>No</CostSharingOrMatchingRequirement>
        <ArchiveDate>05292016</ArchiveDate>
        <GrantorContactEmail>jfl9@cdc.gov</GrantorContactEmail>
        <GrantorContactEmailDescription>jfl9@cdc.gov</GrantorContactEmailDescription>
        <GrantorContactName>John Lisco</GrantorContactName>
        <GrantorContactPhoneNumber>(404) 498-6416</GrantorContactPhoneNumber>
    </OpportunityForecastDetail_1_0>

[TylerHendrickson]

Update: We decided to hold off on including forecasted grants for the moment as more research is needed to confidently verify the work TBD.

Current state: On branch 330-forecasted-grants, commit 431ec05 refactored the SplitGrantsGovXMLDB Lambda handler code to read in data from OpportunityForecastDetail_1_0 XML nodes. A few mentions:

• A common interface, grantRecord, was created to unify the opportunity and (new) forecast types defined in the handler.
• This makes it possible to continue to use a single channel (previously defined as chan opportunity; now as chan grantRecord) to send and receive data parsed from the source XML. It also enables better reuse of existing code responsible for processing the parsed data.
• Tests have been updated to pass, but have not been updated with new test cases that ensure forecasted opportunities are correctly parsed and persisted to S3. Adding test coverage for forecasted opportunities should be considered a requirement for delivery.
• The code also modifies the S3 object keys used to store the parsed data by including the name of the XML node (either OpportunitySynopsisDetail_1_0 or OpportunityForecastDetail_1_0) from which the data was derived. The idea here is to make it possible to retain source records that represent the final state of a forecasted grant opportunity separately from its eventually-non-forecasted form, as well as (and perhaps more importantly) to assist the PersistGrantsGovXMLDB handler in determining which Go type an incoming S3 object's data should be deserialized into for DynamoDB storage.
  • Currently, this constitutes a breaking change as the underlying infrastructure needs to be updated with the new S3 key naming conventions. In particular, IAM policies and S3 bucket triggers specify that object keys must end in /v2.xml – these will need to be updated to /v2.OpportunitySynopsisDetail_1_0.xml and /v2.OpportunityForecastDetail_1_0.xml.
  • Other code updates may be necessary – I recommend searching Terraform and Lambda handler code for occurrences of v2.xml and adjusting accordingly.

Next steps:

1. Update SplitGrantsGovXMLDB unit tests to cover forecasted grant opportunity branches.
2. Update (refactor) Terraform and Lambda handler code as mentioned above to account for new S3 naming conventions.
3. Update the PersistGrantsGovXMLDB handler's code to account for the possibility that S3 object data may need to be deserialized into the OpportunityForecastDetail_1_0 type. The exact nature of these updates likely depends on which details we want to capture from forecasted grants, but assuming we want to fully capture the data (as opposed to, say, massaging the data into the existing OpportunitySynopsisDetail_1_0 data model), a logic branch will be needed to deserialize the S3 data into the correct struct type.
4. Although also dependent on currently-undetermined implementation needs, it's likely that the PublishGrantEvents Lambda handler and/or the pkg/grantsSchemas/usdr library package code will need to be updated to account for forecasted grants in some way (e.g. updating validation logic).

[TylerHendrickson]

In case it's useful, the following is a visual-diff output showing the field name differences between the OpportunitySynopsisDetail_1_0 (left column) and OpportunityForecastDetail_1_0 (right column) data models:

AdditionalInformationOnEligibility                              AdditionalInformationOnEligibility
AdditionalInformationText                                       AdditionalInformationText
AdditionalInformationURL                                        AdditionalInformationURL
AgencyCode                                                      AgencyCode
AgencyName                                                      AgencyName
ArchiveDate                                                     ArchiveDate
AwardCeiling                                                    AwardCeiling
AwardFloor                                                      AwardFloor
CategoryExplanation                                             CategoryExplanation
CategoryOfFundingActivity                                       CategoryOfFundingActivity
CFDANumbers                                                     CFDANumbers
CloseDate                                                     <
CloseDateExplanation                                          <
CostSharingOrMatchingRequirement                                CostSharingOrMatchingRequirement
Description                                                     Description
EligibleApplicants                                              EligibleApplicants
                                                              > EstimatedAwardDate
                                                              > EstimatedProjectStartDate
                                                              > EstimatedSynopsisCloseDate
                                                              > EstimatedSynopsisCloseDateExplanation
                                                              > EstimatedSynopsisPostDate
EstimatedTotalProgramFunding                                    EstimatedTotalProgramFunding
ExpectedNumberOfAwards                                          ExpectedNumberOfAwards
                                                              > FiscalYear
FundingInstrumentType                                           FundingInstrumentType
GrantorContactEmail                                             GrantorContactEmail
GrantorContactEmailDescription                                  GrantorContactEmailDescription
GrantorContactText                                            | GrantorContactName
                                                              > GrantorContactPhoneNumber
LastUpdatedDate                                                 LastUpdatedDate
OpportunityCategory                                             OpportunityCategory
OpportunityCategoryExplanation                                  OpportunityCategoryExplanation
OpportunityID                                                   OpportunityID
OpportunityNumber                                               OpportunityNumber
OpportunityTitle                                                OpportunityTitle
PostDate                                                        PostDate
Version                                                         Version

That is, a forecasted opportunity (OpportunityForecastDetail_1_0)...

• Does not have the following fields (which are present on posted/non-forecasted opportunities):
  • CloseDate
  • CloseDateExplanation
  • GrantorContactText
• Uniquely has the following fields (which are not present on posted/non-forecasted opportunities):
  • EstimatedAwardDate
  • EstimatedProjectStartDate
  • EstimatedSynopsisCloseDate
  • EstimatedSynopsisCloseDateExplanation
  • EstimatedSynopsisPostDate
  • FiscalYear
  • GrantorContactName
  • GrantorContactPhoneNumber

[ClaireValdivia]

this pr is in staging, but blocked by a bug that #874 will address