Spike - Data Dumps: Availability, Access Controls, and Tracking

[erichfi]

Objective: Investigate and enhance the availability, access controls, and tracking mechanisms for Passport XYZ's data dumps to ensure security, accountability, and efficiency.

Key Areas to Explore:

1. Architecture Review:

• Examine current data dump processes and infrastructure.
• Identify areas for improved availability and access.

2. Access Controls:

• Assess existing access control policies and their implementation.
• Evaluate need for role-based access, multi-factor authentication, and other security measures.

3. Tracking and Auditing:

• Analyze current tracking systems for who accesses the data and when.
• Propose methods to improve audit trails for accountability.

4. Technology Solutions:

• Explore tools and technologies to enhance data management.
• Recommend solutions for scalability and future-proofing.

Expected Outcome:

A comprehensive report outlining the strengths, weaknesses, opportunities, and threats (SWOT analysis) in our current data dump management. Provide actionable recommendations for enhancement, a roadmap for implementation, and impact projections on operations and security.

[tim-schultz]

Architecture Review

Areas for Improvement

1. Data Format Optimization

• Currently exporting multiple data types eventhough its mostly parquet
• Recommendation: Transition to Parquet format for all exports
  • More storage efficient
  • Better compression
  • Optimized for analytics workloads

2. AWS Native Export Methods

• Alternative methods available through AWS RDS for PostgreSQL to S3 exports
• Current Status: Not prioritized because:
  • Existing system is stable and reliable
  • Current filtering capabilities meet requirements
  • Scheduled tasks framework is effective
  • No performance issues reported

3. Environment Configuration

• Export jobs currently running across all environments
• Recommendation: Limit exports to production environment only
  • Would conserve resources
  • Affects scheduled tasks defined in infrastructure code
  • Example location: infra/aws/index.ts:1385

[tim-schultz]

Waiting on confirmation of which exports to keep

[Jkd-eth]

@nutrina It looks like we're blocked on this one, let's connect on Monday on what needs to be done to unblock?

[nutrina]

We should only keep:

• registry_passport.parquet
• registry_score.parquet

@NadjibBenlaldj @stefi-says can you please confirm we do not need keep others?

Also would it help to adjust the table structures to have an updated_at column.
This would allow to make delta data pulls, by filtering by the updated_at value (you can pull only data that was updated since the last pull).

[larisa17]

This depends on #3106