Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tools] Add new page for Base UI npm downloads KPIs #102

Merged
merged 4 commits into from
May 25, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions tools-public/toolpad/components/DownloadsTable.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
import * as React from "react";
import { createComponent } from "@mui/toolpad/browser";
import { DataGrid } from '@mui/x-data-grid';
import { getPackages } from "./NpmChart";

export interface DownloadsTableProps {
data: any[]
}

const getColumns = (packages) => ([
{ field: 'date', headerName: 'Month', width: 150 },
...packages.map(packageName => ({
field: packageName, headerName: packageName, width: packageName === '@radix-ui/react-primitive' || packageName === '@headlessui/react' ? 170 : 120
})),
{field: 'ratio', headerName: "Base UI marketshare", width: 150}
]);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const getColumns = (packages) => ([
{ field: 'date', headerName: 'Month', width: 150 },
...packages.map(packageName => ({
field: packageName, headerName: packageName, width: packageName === '@radix-ui/react-primitive' || packageName === '@headlessui/react' ? 170 : 120
})),
{field: 'ratio', headerName: "Base UI marketshare", width: 150}
]);
const getColumns = (packages) => ([
{ field: 'date', headerName: 'Month', width: 150 },
...packages.map(packageName => ({
field: packageName, headerName: packageName, width: 150
})),
{field: 'ratio', headerName: "Base UI marketshare", width: 150}
]);

cc @apedroferreira this is how you can reproduce the issue with the table overflowing.


function DownloadsTable({ data: dataProp }: DownloadsTableProps) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like a component that would be better in the Toolpad editor, getting us a bit closer to the Google Sheet experience.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand, what do you mean in the Toolpad editor?

Copy link
Member

@Janpot Janpot May 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm interested in learning about why you went for a code component to implement this table rather than use the table that is built in Toolpad. Looking at the code it seems like you use the code component mainly to preprocess the incoming data. At a glance, the problems to solve for us are:

  • Computed columns
    Adding columns mapped from the raw rows, but statically known. e.g. "date" and "ratio". I believe we could solve this either by providing a low-code way of remapping data, or by supporting computed columns
  • Dynamically created columns
    Creating new columns dynamically based on incoming data. e.g. adding a column for each package. It's not clear to me yet how we can support this comprehensively, would need to benchmark other tools for ideas
  • reuse mapped data
    This is a problem I don't see directly here, but we need to think about how we can support e.g. showing a graph of the filtered data of the datagrid. Maybe we can think about providing a myGrid.filteredRows or something that other components can bind to. Not fully convinced about this yet as it tightly couples components to each other. (What if I want to delete/replace the datagrid, does the chart become useless? Do I have to redo all data mangling?)
    Perhaps we should offer an option to circumvent the static column definition and always infer from the incoming data?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see the question, initially I was structuring the data in the component itself, but then moved this logic in a serverless function, I will try to do this change and report back if I miss a feature.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated the app to use the DataGrid component directly. The only feature I was missing (or didn't know how to do) is rename the title of the columns

let data = [...(dataProp ?? [])];

// @ts-ignore
let packages = getPackages(data);
packages = packages.filter(function(item) {
return item !== '@mui/material' && item !== '@mui/core'
});

data = data.map(entry => {
let headlessLibrariesDownloads = 0;
Object.keys(entry).forEach(key => {
if(key !== 'date' && key !== '@mui/base') {
headlessLibrariesDownloads += entry[key];
}
})
return {
...entry,
// @ts-ignore
date: entry.date.slice(0, -3),
// @ts-ignore
id: entry.date,
ratio: `${(entry['@mui/base']/headlessLibrariesDownloads * 100).toFixed(2)}%`
}
});

const columns = getColumns(packages);

return <DataGrid rows={data} columns={columns} />;
}

export default createComponent(DownloadsTable, {
argTypes: {
data: {
typeDef: { type: 'array' }
}
},
});
58 changes: 58 additions & 0 deletions tools-public/toolpad/components/NpmChart.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
import * as React from "react";
import { createComponent } from "@mui/toolpad/browser";
import { LineChart, Line, CartesianGrid, XAxis, Tooltip, Legend, YAxis } from 'recharts';
Copy link
Member

@oliviertassinari oliviertassinari May 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Soon https://master--material-ui-x.netlify.app/x/react-charts/lines/#data-format 😁

Actually, the API would be different, it wouldn't work like recharts

<LineChart width={600} height={300} data={data}>


export interface ChartProps {
data: any[]
}

const colors = [
"#1976d2",
"#9c27b0",
"#d32f2f",
"#ed6c02",
"#2f2f2f",
"#2e7d32",
];

export const getPackages = (inData = []) => {
const packages: string[] = [];
if(inData && inData.length > 0) {
Object.keys(inData[0] ?? {}).forEach(packageName => {
if(packageName !== 'date') {
packages.push(packageName);
}
});
}

return packages;
}

function Chart(props: ChartProps) {
const { data } = props;
// @ts-ignore
let packages = getPackages(data);

packages = packages.filter(function(item) {
return item !== '@mui/material' && item !== '@mui/core'
});

return (
<LineChart width={600} height={300} data={data}>
mnajdova marked this conversation as resolved.
Show resolved Hide resolved
{packages.map((packageName, idx) => <Line type="monotone" dataKey={packageName} key={packageName} stroke={colors[idx]} /> )}
<CartesianGrid stroke="#ccc" />
<Tooltip />
<Legend />
<XAxis dataKey="date" />
<YAxis width={100}/>
</LineChart>
);
}

export default createComponent(Chart, {
argTypes: {
data: {
typeDef: { type: 'array' }
}
},
});
30 changes: 30 additions & 0 deletions tools-public/toolpad/pages/baseUiNpmKpis/page.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
apiVersion: v1
kind: page
spec:
id: U0CsCz5
title: baseUiNpmKpis
display: shell
content:
- component: PageRow
name: pageRow
children:
- component: codeComponent.NpmChart
name: codeComponent_NpmChart
props:
data:
$$jsExpression: |
queryHeadlessLibrariesDownloads.data
- component: PageRow
name: pageRow1
children:
- component: codeComponent.DownloadsTable
name: codeComponent_DownloadsTable
props:
data:
$$jsExpression: |
queryHeadlessLibrariesDownloads.data
queries:
- name: queryHeadlessLibrariesDownloads
query:
function: queryHeadlessLibrariesDownloads
kind: local
1 change: 1 addition & 0 deletions tools-public/toolpad/resources/functions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -289,3 +289,4 @@ export * from "./queryMUIXLabels";
export * from "./queryPRs";
export * from "./queryPRs2";
export * from "./queryGender";
export * from "./queryHeadlessLibrariesDownloads";
81 changes: 81 additions & 0 deletions tools-public/toolpad/resources/queryHeadlessLibrariesDownloads.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
// Toolpad queries:

const formatDatePart = (datePart) => {
return `${datePart < 10 ? "0" : ""}${datePart}`;
}

const getDateString = (date: Date) => {
const month = date.getMonth() + 1;
const day = date.getDate();
return `${date.getFullYear()}-${month < 10 ? "0" : ""}${month}-${day < 10 ? "0" : ''}${date.getDate()}`
}

export const getPackages = (inData) => {
const packages: string[] = [];
Object.keys(inData ?? {}).forEach(packageName => {
packages.push(packageName);
});
return packages;
}

const getMonthKey = (date: string) => {
return date.slice(0, -2) + "01";
}

export const prepareData = (inData) => {
const date = new Date(2022, 6, 1, 0, 0, 0, 0);
const today = new Date();
const packages = getPackages(inData);

const monthsData = {};

while(date < today) {
monthsData[getDateString(date)] = {};
packages.forEach(packageName => {
monthsData[getDateString(date)][packageName] = 0;
})
date.setMonth(date.getMonth() + 1);
}

packages.forEach(packageName => {
Object.keys(inData[packageName]).map(date => {
const monthKey = getMonthKey(date);
monthsData[monthKey][packageName] += inData[packageName][date];
})
});
Comment on lines +23 to +43
Copy link
Member

@oliviertassinari oliviertassinari May 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh nice, it looks almost like the logic in https://docs.google.com/spreadsheets/d/1FHxZ456e1t_MNJZg9wrq7pLRFzknZ6jMBhFuW1j1Q8o/edit#gid=0 used to fetch the data (importdownload()).

function importdownload(start = '2023-01-01', end = '2023-06-01', packagesRange = [['react-dom', '@mui/base']]) {
  const packages = packagesRange[0].filter((cell) => cell !== '');
  console.log('URL', `https://npm-stat.com/api/download-counts?${addQuery({
    package: packages
  })}&from=${start}&until=${end}`);

  var jsondata = UrlFetchApp.fetch(`https://npm-stat.com/api/download-counts?${addQuery({
    package: packages
  })}&from=${start}&until=${end}`);

  let npmStateResponse = JSON.parse(jsondata.getContentText());

  const months = {};

  Object.keys(npmStateResponse).forEach((package) => {
    const downloads = npmStateResponse[package];

    Object.keys(downloads).forEach((date) => {
      const month = date.substring(0, 7);

      months[month] = months[month] || {};
      months[month][package] = months[month][package] || 0;
      months[month][package] += downloads[date];
    });
  });

  const output = [['month', ...packages]];

  Object.keys(months).sort().forEach((month) => {
    output.push([month, ...packages.map((package) => {
      if (package === 'react-dom') {
        return months[month][package];
      }
      return months[month][package] / months[month]['react-dom'];
    })]);
  });

  console.log('output', output);
  return output;
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, looks like the same thing 👍 I was adding to add similar page for all products


const data: object[] = [];

Object.keys(monthsData).forEach((date) => {
const entry = {
date,
...monthsData[date],
'@mui/base': monthsData[date]['@mui/base'] + monthsData[date]['@mui/core'] - monthsData[date]['@mui/material'],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A side note, in https://docs.google.com/spreadsheets/d/1FHxZ456e1t_MNJZg9wrq7pLRFzknZ6jMBhFuW1j1Q8o/edit#gid=590909841, I also subtract with @mui/joy, underestimating the actual downloads of Base UI. Soon or later, we can move to @mui/core-downloads-tracker

Screenshot 2023-05-17 at 19 05 21

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will leave it as is for now

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, agree, makes more sense.

}

delete entry['@mui/material'];
delete entry['@mui/core'];
data.push(entry);
})

return data;
}

export async function queryHeadlessLibrariesDownloads() {
const todayDate = new Date();
const today = `${todayDate.getFullYear()}-${formatDatePart(todayDate.getMonth() + 1)}-${formatDatePart(todayDate.getDate())}`;

const baseDownloadsResponse = await fetch(`https://npm-stat.com/api/download-counts?package=%40mui%2Fbase&package=%40mui%2Fmaterial&package=%40mui%2Fcore&from=2022-07-01&until=${today}`);
const baseDownloads = await baseDownloadsResponse.json()

const headlessLibrariesDownloadsResponse = await fetch(`https://npm-stat.com/api/download-counts?package=%40react-aria%2Futils&package=%40headlessui%2Freact&package=reakit&package=%40radix-ui%2Freact-primitive&package=%40reach%2Futils&from=2022-07-01&until=${today}`);
const headlessLibrariesDownloads = await headlessLibrariesDownloadsResponse.json()

const inData = {
...baseDownloads,
...headlessLibrariesDownloads
}
const data = prepareData(inData);

return data;
}