Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(translations/differences): visualize how many commits behind translations are (#8338) #8338

Merged
merged 23 commits into from
Apr 4, 2024
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
fa4bea8
PoC: visualization for how far behind the lastest commit
hochan222 Mar 4, 2023
6541ebc
feat: tag for Metadata does not exist.
hochan222 Mar 4, 2023
19b5733
fix: filter available
hochan222 Mar 6, 2023
6c9af9b
refactor: execSync cwd option
hochan222 Mar 6, 2023
1f50870
fix: filter priority
hochan222 Mar 6, 2023
36e4628
Optimization through cache
hochan222 Mar 6, 2023
9ebd1a6
reduce ja time to 214s with git rev-list
hochan222 Mar 11, 2023
9d3d8f7
feat: source commit error report
hochan222 Mar 11, 2023
b04917e
Optimize source commit logic
hochan222 Mar 11, 2023
528a7e4
Merge branch 'main' into visualization-source-commit
hochan222 Mar 11, 2023
06e74e3
feat: source commit importance color
hochan222 Mar 11, 2023
ad5246d
new approach for traversing git commit graph
hochan222 Mar 25, 2023
a71eae9
remove p tag in table tag
hochan222 Mar 25, 2023
153d681
add details to report message and file name
hochan222 Mar 25, 2023
94eee57
modify source commit filter condition
hochan222 Mar 25, 2023
2d2dba7
fix: consider sourceCommitsBehindCount undefined
hochan222 Mar 25, 2023
032db13
check automatic cache validation
hochan222 Mar 26, 2023
c799cec
refactor: types and naming
hochan222 Mar 26, 2023
08701ad
execSync -> spawn, various fixes for optimization
hochan222 Apr 22, 2023
afcee51
fix: make sourceCommitCache work correctly
hochan222 Apr 22, 2023
cf36eb5
remove log
hochan222 Apr 22, 2023
0e84df6
Merge branch 'main' into visualization-source-commit
hochan222 Mar 25, 2024
9997b2c
Merge branch 'main' into visualization-source-commit
caugner Apr 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions client/src/translations/differences/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ interface DocumentEdits {
parentModified: string;
commitURL: string;
parentCommitURL: string;
sourceCommitsBehindCount: number;
hochan222 marked this conversation as resolved.
Show resolved Hide resolved
sourceCommitURL?: string;
}

interface Document {
Expand Down Expand Up @@ -574,6 +576,13 @@ function DocumentsTable({
const a = A.mdn_url;
const b = B.mdn_url;
return reverse * a.localeCompare(b);
} else if (sort === "sourceCommit") {
const a = A.edits.sourceCommitsBehindCount;
const b = B.edits.sourceCommitsBehindCount;
if (b === undefined) {
return -1;
}
hochan222 marked this conversation as resolved.
Show resolved Hide resolved
return reverse * (b - a);
} else {
throw new Error(`Unrecognized sort '${sort}'`);
}
Expand Down Expand Up @@ -606,6 +615,7 @@ function DocumentsTable({
<TableHead id="popularity" title="Popularity" />
<TableHead id="modified" title="Last modified" />
<TableHead id="differences" title="Differences" />
<TableHead id="sourceCommit" title="Source Commit" />
</tr>
</thead>
<tbody>
Expand Down Expand Up @@ -658,6 +668,14 @@ function DocumentsTable({
<LastModified edits={doc.edits} />
</td>
<td>{doc.differences.total.toLocaleString()}</td>
<td>
<L10nSourceCommitModified
sourceCommitsBehindCount={
doc.edits.sourceCommitsBehindCount
}
sourceCommitURL={doc.edits.sourceCommitURL}
/>
</td>
</tr>
);
})}
Expand All @@ -680,6 +698,17 @@ function DocumentsTable({
);
}

function L10nSourceCommitModified({
sourceCommitsBehindCount,
sourceCommitURL,
}: Pick<DocumentEdits, "sourceCommitsBehindCount" | "sourceCommitURL">) {
if (!sourceCommitURL) return <p>Metadata does not exist.</p>;
hochan222 marked this conversation as resolved.
Show resolved Hide resolved

return (
<a href={sourceCommitURL}>{`${sourceCommitsBehindCount} commits behind`}</a>
);
}

function LastModified({ edits }: { edits: DocumentEdits }) {
const modified = dayjs(edits.modified);
const parentModified = dayjs(edits.parentModified);
Expand Down
53 changes: 43 additions & 10 deletions server/translations.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import path from "node:path";

import express from "express";
import { fdir } from "fdir";
import { execSync } from "node:child_process";

import { getPopularities, Document, Translation } from "../content/index.js";
import {
Expand Down Expand Up @@ -121,6 +122,8 @@ export function findDocuments({ locale }) {
};
}

const commitHashCache = {};

function getDocument(filePath) {
function packagePopularity(document, parentDocument) {
return {
Expand All @@ -141,22 +144,52 @@ function getDocument(filePath) {
};
}

function getCommitBehindFromLatest(filename, commitHash) {
if (commitHashCache[filename] === undefined) {
try {
commitHashCache[filename] = execSync(
`git rev-list --count ${commitHash}..HEAD -- ${filename}`,
Copy link
Member Author

@hochan222 hochan222 Mar 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to reduce 930 seconds to 214 seconds with git rev-list --count ${commitHash}..HEAD -- ${filename} command for ja locale files.

  • AS IS
    • 6840 files (with source commit 1825 files): takes 930 seconds. (15m 30s)
  • TO BE
    • 6840 files (with source commit 1825 files): takes 214 seconds. (3m 34s)

But still we have to consider up to 10000 files.

{
cwd: CONTENT_ROOT,
}
).toString();
} catch (err) {
// console.log(err)
}
}
// console.log(`${filename}: ${commitHashCache[filename]}`)
return commitHashCache[filename];
}

function packageEdits(document, parentDocument) {
const commitURL = getLastCommitURL(
document.fileInfo.root,
document.metadata.hash
);
const parentCommitURL = getLastCommitURL(
parentDocument.fileInfo.root,
parentDocument.metadata.hash
);
const modified = document.metadata.modified;
const parentModified = parentDocument.metadata.modified;
const {
fileInfo: { root: fileRoot },
metadata: { hash: fileHash, modified, l10n },
} = document;
const {
fileInfo: { root: parentFileRoot, path: parentFilePath },
metadata: { hash: parentFileHash, parentModified },
} = parentDocument;

const commitURL = getLastCommitURL(fileRoot, fileHash);
const parentCommitURL = getLastCommitURL(parentFileRoot, parentFileHash);
let sourceCommitURL;
let sourceCommitsBehindCount;
if (l10n?.sourceCommit) {
sourceCommitURL = getLastCommitURL(CONTENT_ROOT, l10n.sourceCommit);
sourceCommitsBehindCount = getCommitBehindFromLatest(
parentFilePath,
l10n.sourceCommit
);
}

return {
commitURL,
parentCommitURL,
modified,
parentModified,
sourceCommitURL,
sourceCommitsBehindCount,
};
}

Expand Down