Add cron job to sync pledge signers #3505

hyphenized · 2023-06-28T17:53:38Z

Implemented a new workflow to account for addresses that could've been dropped during the sync with Galxe. This workflow runs every 3 days and synchronizes addresses over a 4 day sliding window, two of which are from the last sync.

Closes #2678

Latest build: extension-builds-3505 (as of Mon, 03 Jul 2023 22:05:59 GMT).

This script should not have it's dependencies checked during the build workflow

Shadowfiend · 2023-06-29T03:36:21Z

.github/workflows/cron/pledge-sync.js

@@ -0,0 +1,116 @@
+// @ts-check
+/* eslint-disable no-console, no-await-in-loop, import/no-unresolved */


Instead of killing import/no-unresolved, let's have the lint job run yarn install in this package as well as the top level. That job is meant to hit the whole codebase, so we should let it do proper lint checks.

no-await-in-loop and related things that airbnb bans due to regenerator-runtime concerns/functional programming concerns we may want to disable generally for stuff in .github/workflows that is meant to be more scripty. Can discuss that for later though.

Either way, please always drop a comment explaining why linting rules are being disabled.

kkosiorowska · 2023-06-29T06:08:35Z

.github/workflows/cron/pledge-sync.js

+    const payload = {
+      operationName: "credentialItems",
+      query: `
+      mutation credentialItems($credId: ID!, $operation: Operation!, $items: [String!]!) 
+        { 
+          credentialItems(input: { 
+            credId: $credId 
+            operation: $operation 
+            items: $items 
+          }) 
+          { 
+            name 
+          } 
+        }
+      `,
+      variables: {
+        credId: "194531900883902464",
+        operation: "APPEND",
+        items: batch,
+      },
+    }


I wonder about one thing. 🤔 If we add a new workflow, shouldn't we remove this part from here? Currently, part of the code is duplicated in two places.

https://github.com/tahowallet/taho.xyz/blob/main/manifesto-dapp-backend/functions/src/index.ts#L325-L356

Sorry, forgot to submit this comment. We need both because the other one is dropping addresses at random intervals.

Shadowfiend · 2023-06-29T14:03:58Z

.github/workflows/sync-pledge.yml

+
+on:
+  schedule:
+    - cron: "0 0 */3 * *"


How about we also add a workflow_dispatch trigger so we can fire it off manually from the Actions tab?

Shadowfiend · 2023-06-29T15:29:47Z

.github/workflows/cron/pledge-sync.js

+import admin from "firebase-admin"
+import fetch from "node-fetch"
+
+/* eslint-disable prefer-destructuring */


Brain fart, this isn't using Webpack 😬

Lordy I forgot that was even a thing with Webpack. Love Webpack! <_<

Shadowfiend · 2023-06-29T15:31:08Z

.github/workflows/cron/pledge-sync.js

+      throw new Error("Unable to sync with galxe")
+    }
+
+    await wait(3000)


Do we get rate limit information from Galxe, or is this just to cross our fingers and hope not to topple their server over?

We do but it's unlikely that we will get rate limited by Galxe, this is more of a safety measure against the random removal of synced addresses

Shadowfiend · 2023-06-29T15:31:30Z

.github/workflows/cron/pledge-sync.js

+
+    console.log("Retrieved:", allDocs.length, "addresses")
+
+    await wait(1500)


Do we need that when hitting our own Firebase stuff?

Unfortunately when testing this, trying to pull in too many records or pull records too fast errors out. That was the case with firebase API access so I assumed the same would hold true with the admin-sdk

Do those errors give us backoff information by any chance? Not really a blocker, just would be nice to go as hard as they'll let us lol.

Shadowfiend · 2023-06-29T15:32:12Z

.github/workflows/cron/package.json

@@ -0,0 +1,12 @@
+{
+  "name": "@tallyho/taho-extension-cron-jobs",


Also let's maybe rename this dir + package. This isn't a generic cron jobs package, this is specifically a pledge sync job right?. We can do another subdir, but IMO just renaming cron to pledge-signer-sync is fine.

Shadowfiend

One question about one of the non-secrets being used. Going to set up the rest.

.github/workflows/pledge-signer-sync/pledge-sync.js

Shadowfiend · 2023-07-03T16:43:46Z

Setting up secrets + getting this merged is on the list today.

Shadowfiend

All right overall this looks good, and I believe I have the setup correct. Will merge now and then try to kick the action off from the Actions tab.

@kkosiorowska

## Highlights - An updated Taho token list! This will start auto-updating Soon™. ## What's Changed * Update Taho token list reference by @kkosiorowska in #3506 * Don't run e2e tests when workflow is triggered by push of tag by @michalinacienciala in #3514 * Fix throwing NFT filters panel by @jagodarybacka in #3513 * Use gas price from 0x quote to setup swap transaction by @jagodarybacka in #3516 * v0.40.0 by @hyphenized in #3507 * Add cron job to sync pledge signers by @hyphenized in #3505 * Fix invalid workflow file by @hyphenized in #3519 * Remove workflow trigger used for testing by @hyphenized in #3520 * Fix migration for cleaning account balances by @jagodarybacka in #3527 * v0.40.1 with op swaps gas price hotfix by @Shadowfiend in #3518 * v0.40.2 - accounts migration fix by @jagodarybacka in #3529 **Full Changelog**: v0.40.2...v0.41.0 Latest build: [extension-builds-3531](https://github.com/tahowallet/extension/suites/14093546910/artifacts/788185513) (as of Thu, 06 Jul 2023 02:05:27 GMT).

Add cron job to sync pledge signers

fdbb8d1

hyphenized requested a review from a team as a code owner June 28, 2023 17:53

Shadowfiend enabled auto-merge June 28, 2023 21:38

Shadowfiend disabled auto-merge June 28, 2023 21:39

hyphenized added 2 commits June 28, 2023 18:32

Set correct working dir before installing deps

e6e93d7

Disable import/no-unresolved on workflow script

d624a4b

This script should not have it's dependencies checked during the build workflow

Shadowfiend requested changes Jun 29, 2023

View reviewed changes

Install workflow deps before linting in CI

374c7e8

kkosiorowska reviewed Jun 29, 2023

View reviewed changes

Shadowfiend reviewed Jun 29, 2023

View reviewed changes

Add workflow_dispatch to allow manual syncs

670bc44

Shadowfiend reviewed Jun 29, 2023

View reviewed changes

hyphenized force-pushed the add-sync-cron branch from dafe7a9 to c4e29b0 Compare June 29, 2023 18:15

hyphenized added 2 commits June 29, 2023 15:08

Refactor script to use firebase api

25cdbb8

Fix hanging bug, naming improvements

89f5130

hyphenized force-pushed the add-sync-cron branch from c4e29b0 to 89f5130 Compare June 29, 2023 20:09

Shadowfiend reviewed Jun 29, 2023

View reviewed changes

.github/workflows/pledge-signer-sync/pledge-sync.js Show resolved Hide resolved

.github/workflows/pledge-signer-sync/pledge-sync.js Outdated Show resolved Hide resolved

Log errors thrown while fetching addresses

dda6c97

Shadowfiend approved these changes Jul 3, 2023

View reviewed changes

Merge branch 'main' into add-sync-cron

d6c360d

Shadowfiend enabled auto-merge July 3, 2023 21:58

Shadowfiend merged commit df74ca6 into main Jul 3, 2023

Shadowfiend deleted the add-sync-cron branch July 3, 2023 22:02

Shadowfiend mentioned this pull request Jul 6, 2023

v0.41.0 #3531

Merged

hyphenized mentioned this pull request Aug 7, 2023

Pledge credential cleanup #3522

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cron job to sync pledge signers #3505

Add cron job to sync pledge signers #3505

hyphenized commented Jun 28, 2023 •

edited by github-actions bot

Loading

Shadowfiend Jun 29, 2023

kkosiorowska Jun 29, 2023

hyphenized Jul 11, 2023

Shadowfiend Jun 29, 2023

Shadowfiend Jun 29, 2023

hyphenized Jun 29, 2023

Shadowfiend Jun 29, 2023

Shadowfiend Jun 29, 2023

hyphenized Jun 29, 2023

Shadowfiend Jun 29, 2023

hyphenized Jun 29, 2023

Shadowfiend Jun 29, 2023

Shadowfiend Jun 29, 2023

Shadowfiend left a comment

Shadowfiend commented Jul 3, 2023

Shadowfiend left a comment

		@@ -0,0 +1,116 @@
		// @ts-check
		/* eslint-disable no-console, no-await-in-loop, import/no-unresolved */


		console.log("Retrieved:", allDocs.length, "addresses")

		await wait(1500)

		@@ -0,0 +1,12 @@
		{
		"name": "@tallyho/taho-extension-cron-jobs",

Add cron job to sync pledge signers #3505

Add cron job to sync pledge signers #3505

Conversation

hyphenized commented Jun 28, 2023 • edited by github-actions bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Shadowfiend left a comment

Choose a reason for hiding this comment

Shadowfiend commented Jul 3, 2023

Shadowfiend left a comment

Choose a reason for hiding this comment

hyphenized commented Jun 28, 2023 •

edited by github-actions bot

Loading