1st place at the O'Reilly Architectural Kata Fall 2024 Katas: Diversity Cyber Council - ClearView | O'Reilly Architectural Kata (Autumn 2024)
A (hopefully) pragmatic approach to the O'Reilly Autumn Architectural Kata 2024
- Team
- Introduction
- Event Storming
- Requirements
- Architecture Characteristics
- Architecture style
- Architecture
- Known Limitations
- Diagrams
The Diversity Cyber Council is a Non-Profit that helps under-represented demographics in the tech-industry with education, staffing and training opportunities.
The ClearView program aims to harness AI to facilitate matches between job candidates and open roles using anonymized resumes.
Our solution consists of an architecture that is well-equipped to serve a UX application such as this. One main contribution was enabling the integration of thousands of HR systems in our platform, another was the efficient organisation of the data flows for our reporting and analytics domain.
However, our most important focus was the fulfillment of all AI-related requirements: The emergence of LLMs with their promise of easy and fast solutions to previously hard to solve problems, understandably drives demand for AI based solutions in IT projects. While there is significant potential in LLMs, there are unique risks to be considered when designing a system whose core functionality leverages LLMs: LLMs are recent and evolve quickly, thus experienced engineers in the field are rare. Furthermore, LLM services are expensive to use, however their cost is hard to predict and externally hosted solutions impose rate-limits and short model lifecycles. And finally, the inherent bias in LLM models must not leak into the matching process since bias is the very thing we seek to control. Our architecture navigates all these risks and makes the use of LLMs feasible for a Non-Profit organisation such as the Diversity Cyber Council. We deliberately save cost in other parts of the architecture in order to mitigate the unpredictable price tag of such an AI solution.
-
Develop an AI-Powered Matching Platform: Create an effective system that minimizes bias in the hiring process.
- Our main contributions for this objective are:
- Research, such as interview with an AI-expert and token estimation of a sample prompt leading to many assumptions and requirements,
- Matching algorithm considerations, such as ADR-010 and ADR-011, deterministic matching, and the corresponding visualization
- Other architectural considerations, such as ADR-006, ADR-007
- ADR-025: AI test concept
- Our main contributions for this objective are:
-
Analyze Hiring Disparities: Utilize metrics and surveys to uncover disparities between selected job candidates and those who were not hired.
-
Seamless HR Integration: Implement a streamlined integration with employers' HR systems to ensure a smooth interview experience.
- Our main contributions for this objective are design decisions such as:
- Asynchronous trigger: ADR-016
- Fault-tolerant and scalable HR Integration Service: ADR-023 and its visualization
- Our main contributions for this objective are design decisions such as:
We used Event Storming to explore and map out various domains within our business processes. The session focused on capturing and documenting the flow of events, actors, and interactions between systems. This document summarizes our outcomes and serves as a guide to understanding our approach.
Based on the problem description, our Event Storming, and research, we compiled a list of functional and non-functional requirements complemented by a set of assumptions. The requirements and assumptions are numbered and are referenced in the following way:
- Ri (f. ex. R2), for functional requirements
- Qi (f.ex. Q13), for non-functional requirements
- Ai (f. ex. A22) , for assumptions
We typically link to the file, but due to markdown limitations, the specific entry can not be referenced in the link.
The glossary defines terms we use.
Starting from the business requirements and the Event Storming we determined the architecture characteristics.
The selection of characteristics provides the basis for the selection of an Architectural Type. We selected the following seven characteristics and focused our attention on the three most important characteristics.
These driving characteristics significantly influenced our ADRs. The following ADRs were important to realize our TOP 3 driving characteristics:
- interoperability
- feasibility
- testability
According to the TOP 3 driving characteristics:
- interoperability
- feasibility
- testability
a service-based architecture ADR-002 with event-streaming capabilities where needed, was selected to leverage the optimal balance between feasibility and testability.
By utilizing the C4 approach to visualize software architecture, we can effectively illustrate the dependencies between various components of our application while emphasizing hierarchical relationships. Our primary focus will be on the C1 and C2 views to provide an initial overview, before delving into the C3 view to examine the core components in greater detail.
The context view allows us to get a first grasp of the actors and the external components interacting with the ClearView system.
The full Context diagram with the description of the Actors and Systems can be found here.
The Container diagram shows the high-level shape of the software architecture and how responsibilities are distributed across it. It also shows how the containers communicate with each another.
The full Container diagram with the descriptions of the containers and ADRs describing the decisions can be found here.
To address the main challenges of this system, we created a components diagram:
- Job Candidate Components: This diagram illustrates how we integrate Security and Testability into our architecture.
- Story Components: This diagram illustrates how we integrate Data Integrity and Testability into our architecture.
- Matching Components: This diagram illustrates how we integrate Scalability and Testability into our architecture.
- HR Integration Components: This diagram illustrates how we integrate Interoperability and Fault Tolerance into our architecture.
An interaction of the Containers and Components is documented in the Use Case: Upload and Publish a Resume
The current architecture has the following limitations:
-
Matching Algorithm Limitations:
The matching algorithm utilizes a predefined set of human-readable features, as described in ADR-011. Defining these features comprehensively for all S.M.A.R.T stories and open roles can be challenging. -
Delayed Matching Process:
Matching is executed as a periodic background process rather than in real-time, as detailed in ADR-008. -
External AI Service Rate Limiting:
Since AI services run independently as external entities, as described in ADR-007, and these services introduce hard rate-limits, tracking and control of traffic will be required for reliability as well as cost-control. If the traffic pattern across all LLM clients needs to be known, the combination of all clients into one container will need to be considered. -
User Administration:
User administration through the UI is not yet part of the current concept and is dependent on initial technology decisions, such as the presence of existing Identity Providers (IdPs) at Diversity Cyber Council. For more information, see ADR-022. The current approach is to use a local user account that has access to the Identity Provider's UI.
The diagrams in this project were created using https://www.drawio.com/. They can be viewed and edited directly by importing or opening the corresponding .drawio files.