Move most of https://github.com/cncf/toc/blob/main/docs/projects.md to landscape metadata

[caniszczyk]

I love this document: https://github.com/cncf/toc/blob/main/docs/projects.md

However, most of the data around project acceptance dates, sandbox to incubation dates, security audits etcs should live in the landscape.yml and we can create a nitfy page to show some of this in some fashion: e.g.,
https://github.com/cncf/landscape/blob/master/landscape.yml#L24

[caniszczyk]

I'd like to put this on your plate @jeefy potentially after kubecon EU

[dims]

@jeefy could we do this before kubecon NA? :)

[caniszczyk]

it's almost done I think if you go to https://landscape.cncf.io/card-mode?project=hosted and click "Download as CSV" you almost have everything Jeefy to start double checking on any missing metadata

…

On Fri, Jul 22, 2022 at 11:31 AM Davanum Srinivas ***@***.***> wrote: @jeefy <https://github.com/jeefy> could we do this before kubecon NA? :) — Reply to this email directly, view it on GitHub <#824 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAPSII5OFLPV3LQ2PAU7QDVVLEGZANCNFSM5TZIOGIA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Cheers, Chris Aniszczyk https://aniszczyk.org

[jeefy]

Yep I think (at a glance) things that are missing are links to the TOC decks.

As I start "enriching" the Landscape data I can backfill those in.

[dims]

thanks @jeefy

[halcyondude]

Hi @jeefy, I have a question :)

Context / Background

I'm pulling data (daily) into https://github.com/cncf/landscape-graph (with a GH Action here).

TLDR: it's putting the landscape metadata into a Graph Database, uses GraphQL as an interface definition language (WIP), and hopes to provide a data model that can help to answer interesting questions. Here's a 10k of the schema/ontology/data model:

https://github.com/cncf/landscape-graph#landscape-graph-data-model

This pulls https://landscape.cncf.io/data/items.json, cleans it up (stripping nulls etc), and commits to:

https://github.com/cncf/landscape-graph/blob/main/landscape-items-clean.json

Most of the rest is using that as it's original source.

name: Import landscape.cncf.io dataset, publish to Neo4j Aura
on:
  schedule:
    - cron: 0 0 * * *
  workflow_dispatch: {}
  push:
    branches:
      - main
    paths:
      - .github/workflows/flat.yml
      - clean-landscape-json.ts
jobs:
  scheduled:
    runs-on: ubuntu-latest
    steps:
      - name: Setup deno
        uses: denoland/setup-deno@main
        with:
          deno-version: v1.10.x
      - name: Check out repo
        uses: actions/checkout@v2
      - name: Fetch data
        uses: githubocto/flat@v3
        with:
          http_url: https://landscape.cncf.io/data/items.json
          downloaded_filename: landscape-items.json
          postprocess: ./clean-landscape-json.ts
<snip/>

Questions

How does the landscape-items.json get generated today?

How can I engage with existing efforts around landscape metadata curation / implementation?

Is there a schema, or de facto schema in use?

This is perhaps the most important thing that I hope to understand, since creating a GraphQL, strongly typed interface for the graph is one of the main advantages of the approach. I'm approaching MVP for the GraphQL composition layer (details)

Next steps after the core data model is life will be to generate a formal ontology (OWL, SHACL, etc) so that we can support inferencing, Natural Language Queries (cncf/landscape-graph#90), and other use cases.

Here's a chunk of the cncf sub-graph:

https://github.com/cncf/landscape-graph/blob/4-graphql-endpoint-v1/db/cncf/cncf.graphql

enum MembershipTypes
{
  PLATINUM
  GOLD
  SILVER
  ACADEMIC
  END_USER_SUPPORTER
  NONPROFIT
}

"""
MembershipType: {Silver, Gold, Platinum, Academic, End User Supporter, Non-Profit}
"""
type MembershipType {
  id: ID! @id
  membershipType: MembershipTypes! @unique

  members: [Member!]! @relationship(type: "IS_MLVL", direction: IN)
}

"""
Member: CNCF Member Organization
"""
type Member {
  id: ID! @id
  name: String! @unique
  categories: [Category!]! @relationship(type: "IN_CAT", direction: OUT)

  cncfMemberships: [CncfMembership!]! @relationship(type: "IS_MLVL", direction: OUT)
  endUserGroups: [EUG!]! @relationship(type: "MEMBER_OF", direction: OUT)
  organizations: [Organization!]! @relationship(type: "OWNS", direction: OUT)
}

enum ProjectPhases {
  GRADUATED
  INCUBATING
  SANDBOX
  ARCHIVED
}

"""
ProjectPhase: Sandbox, Incubation, Graduated, Archived
"""
type ProjectPhase {
  id: ID! @id
  projectPhase: ProjectPhases! @unique
  
  projects: [Project!]! @relationship(type: "IS_PROJPHASE", direction: IN)
}

"""
Project: CNCF Open Source Project (Sandbox, Incubating, Graduated, Archived)
"""
type Project {
  id: ID! @id
  name: String! @unique
  categories: [Category!]! @relationship(type: "IN_CAT", direction: OUT)
  
  projectPhases: [ProjectPhase!]! @relationship(type: "IS_PROJPHASE", direction: OUT)
  licenses: [License!]! @relationship(type: "USES", direction: OUT)
  languages: [Language!]! @relationship(type: "USES", direction: OUT)
  tags: [TAG!]! @relationship(type: "IN_SCOPE", direction: OUT)

  projRoles: [ProjRole!]! @relationship(type: "SERVED", direction: IN)
}

"""
Category
TODO: add (maybe?) "union CategorizedEntity = Member | Project"
"""
type Category {
  id: !ID @id
  name String! @unique
  
  members: [Member!]! @relationship(type: "IN_CAT", direction: IN)
  projects: [Project!]! @relationship(type: "IN_CAT", direction: IN)
}

"""
License: TODO doc
"""
type License {
  id: ID! @id
  name: String! @unique

  projects: [Project!]! @relationship(type: "USES", direction: IN)
}

"""
Language: TODO doc
"""
type Language {
  id: ID! @id
  name: String! @unique

  projects: [Project!]! @relationship(type: "USES", direction: IN)
}

enum ROLE_TYPE {
  TOC
  TECHNICAL_ADVISORY_GROUP
  END_USER_GROUP
  PROJECT
}

enum ROLE_POSITION {
  CHAIR
  TECHNICAL_LEAD
}

interface ServedInRole @relationshipProperties {
  roleType: ROLE_TYPE!
  rolePosition: ROLE_POSITION!
  from: Date
  to: Date
}

"""
TOC: Technical Oversight Committee
"""
type TOC  {
  id: ID! @id
  name: String! @unique
  
  rolePersons: [Person!]! @relationship(type: "SERVED_IN_ROLE", properties: "ServedInRole", direction: IN)
  communityPersons: [Person!]! @relationship(type: "IS_COMMUNITY_MEMBER", direction: IN)
}

"""
TAG: Technical Advisory Group: TODO doc
"""
type TAG {
  id: ID! @id
  name: String! @unique
  
  projectsInScope: [Project!]! @relationship(type: "IN_SCOPE", direction: IN)
  
  rolePersons: [Person!]! @relationship(type: "SERVED_IN_ROLE", properties: "ServedInRole", direction: IN)
  communityPersons: [Person!]! @relationship(type: "IS_COMMUNITY_MEMBER", direction: IN)

}

"""
EUG: End User Group
"""
type EUG {
  id: ID! @id
  name: String! @unique
  
  members: Member @relationship(type: "MEMBER_OF", direction: IN)

  rolePersons: [Person!]! @relationship(type: "SERVED_IN_ROLE", properties: "ServedInRole", direction: IN)
}

"""
Person: human
"""
type Person {
  id: ID! @id
  name: String! @unique
  
  tagRoles: [TAG!]! @relationship(type: "SERVED_IN_ROLE", properties: "ServedInRole", direction: OUT)
  eugRoles: [EUG!]! @relationship(type: "SERVED_IN_ROLE", properties: "ServedInRole", direction: OUT)
  tocRoles: [TOC!]! @relationship(type: "SERVED_IN_ROLE", properties: "ServedInRole", direction: OUT)
  projRoles: [Project!]! @relationship(type: "SERVED_IN_ROLE", properties: "ServedInRole", direction: OUT)
  
  boards: [Organization!]! @relationship(type: "IS_BOARD", direction: OUT)
  
  employers: [Organization!]! @relationship(type: "EMPLOYED", direction: IN)
}

Thanks!

[amye]

"How does the landscape-items.json get generated today?
How can I engage with existing efforts around landscape metadata curation / implementation?
Is there a schema, or de facto schema in use?"
You should ask this question in https://github.com/cncf/landscape/issues where @AndreyKozlov1984 will be better able to answer this

[caniszczyk]

This one is done, new metadata is in landscape and appears in the landscapev2 cncf.landscape2.io

[caniszczyk]

We still need to add TAG metadata but that will be tracked here cncf/landscape#3349