Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: analytics into atomic operations #296

Merged
merged 7 commits into from
Jun 28, 2023
Merged

refactor: analytics into atomic operations #296

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d4f1955
analytics into atomic operations
brunocorrea23 Jun 23, 2023
ad91d4a
refactor analytics code and add tests
brunocorrea23 Jun 24, 2023
ff67ac0
refactor metrics in atomic ops
brunocorrea23 Jun 27, 2023
3e2a394
reverting some analytics change plus some refactoring
brunocorrea23 Jun 28, 2023
f1b4d04
merging with main
brunocorrea23 Jun 28, 2023
19167fb
better
brunocorrea23 Jun 28, 2023
f04517d
Merge branch 'main' into analytics-into-atomic-ops
brunocorrea23 Jun 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions routes/callback.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ import {
createUser,
deleteUserBySession,
getUser,
incrementAnalyticsMetricPerDay,
newUserProps,
updateUser,
type User,
Expand Down Expand Up @@ -63,7 +62,6 @@ export const handler: Handlers<any, State> = {
...newUserProps(),
};
await createUser(user);
await incrementAnalyticsMetricPerDay("users_count", new Date());
} else {
await deleteUserBySession(sessionId);
await updateUser({ ...user, sessionId });
Expand Down
2 changes: 0 additions & 2 deletions routes/submit.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ import type { State } from "@/routes/_middleware.ts";
import {
createItem,
getUserBySession,
incrementAnalyticsMetricPerDay,
type Item,
newItemProps,
} from "@/utils/db.ts";
Expand Down Expand Up @@ -51,7 +50,6 @@ export const handler: Handlers<State, State> = {
...newItemProps(),
};
await createItem(item);
await incrementAnalyticsMetricPerDay("items_count", new Date());

return redirect(`/item/${item!.id}`);
},
Expand Down
47 changes: 17 additions & 30 deletions utils/db.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,6 @@ export function newItemProps(): Pick<Item, "id" | "score" | "createdAt"> {
* };
*
* await createItem(item);
* await incrementAnalyticsMetricPerDay("items_count", item.createdAt);
* ```
*/
export async function createItem(item: Item) {
Expand All @@ -78,6 +77,7 @@ export async function createItem(item: Item) {
.set(itemsKey, item)
.set(itemsByTimeKey, item)
.set(itemsByUserKey, item)
.sum(analyticsKey("items_count"), 1n)
.commit();

if (!res.ok) throw new Error(`Failed to create item: ${item}`);
Expand Down Expand Up @@ -196,12 +196,11 @@ export async function createVote(vote: Vote) {
.set(itemsByUserKey, vote.item)
.set(votedItemsByUserKey, vote.item)
.set(votedUsersByItemKey, vote.user)
.sum(analyticsKey("votes_count"), 1n)
.commit();

if (!res.ok) throw new Error(`Failed to set vote: ${vote}`);

await incrementAnalyticsMetricPerDay("votes_count", new Date());

return vote;
}

Expand Down Expand Up @@ -281,7 +280,6 @@ export function newUserProps(): Pick<User, "isSubscribed"> {
* ...newUserProps(),
* };
* await createUser(user);
* await incrementAnalyticsMetricPerDay("users_count", new Date());
* ```
*/
export async function createUser(user: User) {
Expand All @@ -308,6 +306,7 @@ export async function createUser(user: User) {
.set(usersKey, user)
.set(usersByLoginKey, user)
.set(usersBySessionKey, user)
.sum(analyticsKey("users_count"), 1n)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm... It might be better to define a const usersCount at the start of the function with the others without using analyticsKey(). It gives a more immediate idea of what's going on, IMO. At most, maybe a toDate() that returns date.toISOString().split("T")[0] might be useful. Ditto for other similar scenarios. WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I.e. check out formatDate() in #289.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Will adjust that!

.commit();

if (!res.ok) throw new Error(`Failed to create user: ${user}`);
Expand Down Expand Up @@ -401,25 +400,24 @@ export async function incrementAnalyticsMetricPerDay(
.commit();
}

export async function incrementVisitsPerDay(date: Date) {
// convert to ISO format that is zero UTC offset
const visitsKey = [
"visits",
`${date.toISOString().split("T")[0]}`,
];
await kv.atomic()
.sum(visitsKey, 1n)
.commit();
}

iuioiua marked this conversation as resolved.
Show resolved Hide resolved
export async function getVisitsPerDay(date: Date) {
export async function getAnalyticsMetricPerDay(metric: string, date: Date) {
return await getValue<bigint>([
"visits",
metric,
`${date.toISOString().split("T")[0]}`,
]);
}

export async function getAnalyticsMetricsPerDay(
export function analyticsKey(
metric: string,
) {
// convert to ISO format that is zero UTC offset
return [
metric,
`${new Date().toISOString().split("T")[0]}`,
];
}

export async function getAnalyticsMetricListPerDay(
metric: string,
options?: Deno.KvListOptions,
) {
Expand All @@ -438,19 +436,8 @@ export async function getManyAnalyticsMetricsPerDay(
options?: Deno.KvListOptions,
) {
const analyticsByDay = await Promise.all(
metrics.map((metric) => getAnalyticsMetricsPerDay(metric, options)),
metrics.map((metric) => getAnalyticsMetricListPerDay(metric, options)),
);

return analyticsByDay;
}

export async function getAllVisitsPerDay(options?: Deno.KvListOptions) {
const iter = await kv.list<bigint>({ prefix: ["visits"] }, options);
const visits = [];
const dates = [];
for await (const res of iter) {
visits.push(Number(res.value));
dates.push(String(res.key[1]));
}
return { visits, dates };
}
44 changes: 34 additions & 10 deletions utils/db_test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
// Copyright 2023 the Deno authors. All rights reserved. MIT license.
import {
analyticsKey,
type Comment,
createComment,
createItem,
createUser,
deleteUserBySession,
getAllItems,
getAnalyticsMetricPerDay,
getCommentsByItem,
getItem,
getItemsByUser,
Expand All @@ -15,8 +17,7 @@ import {
getUserByLogin,
getUserBySession,
getUserByStripeCustomer,
getVisitsPerDay,
incrementVisitsPerDay,
incrementAnalyticsMetricPerDay,
type Item,
kv,
newCommentProps,
Expand Down Expand Up @@ -71,7 +72,14 @@ Deno.test("[db] (get/create)Item()", async () => {

assertEquals(await getItem(item.id), null);

const itemsCount =
(await getAnalyticsMetricPerDay("items_count", new Date()))?.valueOf() ??
0n;
Copy link
Contributor

@iuioiua iuioiua Jun 25, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const itemsCount =
(await getAnalyticsMetricPerDay("items_count", new Date()))?.valueOf() ??
0n;
const itemsCount = await getAnalyticsMetricPerDay("items_count", new Date());
assertEquals(itemsCount, null);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iuioiua , can you pls explain this suggestion here?
I believe that if you do it like this, the assertion is going to fail. When it gets to this point, the itemsCount is probably equal to 2n because of the two createItem() that were run on the test above (" [db] getAllItems()")

We can't assume anything about the value o itemsCount, as we don't know how many tests will use it before in the code. We can only test that the itemsCount_after = itemsCount_before + 1.

Does it make sense? Did I understand correctly your suggestion?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps, we should move the line that defines itemsCount elsewhere such that it doesn't require a default value. WDYT?

await createItem(item);
assertEquals(
(await getAnalyticsMetricPerDay("items_count", new Date()))!.valueOf(),
iuioiua marked this conversation as resolved.
Show resolved Hide resolved
itemsCount + 1n,
);
await assertRejects(async () => await createItem(item));
assertEquals(await getItem(item.id), item);
});
Expand Down Expand Up @@ -139,8 +147,14 @@ Deno.test("[db] user", async () => {
assertEquals(await getUserBySession(user.sessionId), null);
assertEquals(await getUserByStripeCustomer(user.stripeCustomerId!), null);

const usersCount = (await getAnalyticsMetricPerDay("users_count", new Date()))
?.valueOf() ?? 0n;
await createUser(user);
await assertRejects(async () => await createUser(user));
assertEquals(
(await getAnalyticsMetricPerDay("users_count", new Date()))!.valueOf(),
usersCount + 1n,
);
assertEquals(await getUser(user.id), user);
assertEquals(await getUserByLogin(user.login), user);
assertEquals(await getUserBySession(user.sessionId), user);
Expand All @@ -164,17 +178,20 @@ Deno.test("[db] user", async () => {
);
});

Deno.test("[db] visit", async () => {
Deno.test("[db] analytics increment/get", async () => {
const date = new Date("2023-01-01");
const visitsKey = [
"visits",
const dummyKey = [
"example",
`${date.toISOString().split("T")[0]}`,
];
await incrementVisitsPerDay(date);
assertEquals((await kv.get(visitsKey)).key[1], "2023-01-01");
assertEquals((await getVisitsPerDay(date))!.valueOf(), 1n);
await kv.delete(visitsKey);
assertEquals(await getVisitsPerDay(date), null);
await incrementAnalyticsMetricPerDay("example", date);
assertEquals((await kv.get(dummyKey)).key[1], "2023-01-01");
assertEquals(
(await getAnalyticsMetricPerDay("example", date))!.valueOf(),
1n,
);
await kv.delete(dummyKey);
assertEquals(await getAnalyticsMetricPerDay("example", date), null);
});

Deno.test("[db] newCommentProps()", () => {
Expand Down Expand Up @@ -205,3 +222,10 @@ Deno.test("[db] createComment() + getCommentsByItem()", async () => {
await assertRejects(async () => await createComment(comment2));
assertArrayIncludes(await getCommentsByItem(itemId), [comment1, comment2]);
});

Deno.test("[db] analyticsKey()", () => {
assertEquals(analyticsKey("metric_test"), [
"metric_test",
`${new Date().toISOString().split("T")[0]}`,
]);
});