BLUE-280: Add Publisher for Transaction Digest to IPFS via Web3.Storage #86

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

achal-singh wants to merge 2 commits into dev from BLUE-280

Contributor

achal-singh commented Sep 24, 2024 •

edited

Loading

Summary

The Transaction DIgest Cron Server in the Archiver creates Transaction Digests every 10 cycles.
This PR adds an ipfsPublisher (as a part of the Transaction DIgest Cron Server) to upload/publish the same Transaction Digest to the IPFS Network via the Web3.Storage provider.

Linear Task

Merging Notes

It'd be good to merge BLUE-283's PR into BLUE-280 PR first before merging BLUE-280 to dev.

Testing Notes

Before starting with the tests, make sure you register on the web3.storage portal with your email address and subscribe to their free-tier plan (that gives 5 GB of free storage).
Access your account on https://console.web3.storage and create a Space, set any name as per your preference. At the top of the screen you'll see a Root DID string (starts with did:key:...) of your Space.
Copy and paste the same email ID and Root DID of your space to the root_did and admin_email fields in the archiver-config.json file respectively and set the enableSavingToWeb3Storage flag to true.
Run npm run txDigestCronServer OR pm2 start --name txDigestCron npm -- run txDigestCronServer to launch the txDigestCronServer process.


          BLUE-280: IPFS Data publisher added

f29cafe

achal-singh requested a review from jairajdev

September 24, 2024 01:00

achal-singh self-assigned this

Contributor

github-actions bot commented Sep 24, 2024

PR Reviewer Guide 🔍

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Key issues to review Error Handling The error handling in `uploadDigestToIPFS` function (line 55-60) might not provide sufficient information for debugging. Consider enhancing the error messages to include more specific details about the failure context. Logging Practice The use of `console.log` and `console.error` for logging throughout the `ipfsPublisher.ts` file is inconsistent with the use of `mainLogger` in other parts of the application. It is recommended to use `mainLogger` for uniformity and better control over logging levels and outputs. Possible Bug The condition in `uploadDigestToIPFS` function (line 29) seems to be incorrect. The function returns early if `isUploadPossible()` is false, which is the opposite of the expected behavior where it should proceed if the upload is possible.

github-actions bot added the Review effort [1-5]: 4 label

github-advanced-security bot found potential problems

View reviewed changes

src/txDigester/ipfsPublisher.ts Fixed Show fixed Hide fixed

src/txDigester/ipfsPublisher.ts Fixed Show fixed Hide fixed

src/txDigester/ipfsPublisher.ts Fixed Show fixed Hide fixed

src/txDigester/ipfsPublisher.ts Fixed Show fixed Hide fixed

src/txDigester/ipfsPublisher.ts Fixed Show fixed Hide fixed

src/txDigester/ipfsPublisher.ts Fixed Show fixed Hide fixed

src/txDigester/ipfsPublisher.ts Fixed Show fixed Hide fixed

src/txDigester/ipfsPublisher.ts Fixed Show fixed Hide fixed

aniketdivekar reviewed

View reviewed changes

package.json Outdated Show resolved Hide resolved

src/Config.ts Outdated Show resolved Hide resolved

src/Config.ts Outdated Show resolved Hide resolved

src/txDigester/ipfsPublisher.ts Outdated

    
                try {

                  const plan = await account.plan.get()

                  if (!plan.ok) {

                    console.error(`❌ ${admin_email} does not have an active plan subscribed on Web3.Storage.`)

Contributor

aniketdivekar Sep 24, 2024

we should not log email id. let's mask it using industry standard logic or remove it.

Contributor Author

achal-singh Sep 24, 2024

Yeah we can mask it for sure, although it can't be exploited, I consoled it initially just for testing purpose.

src/txDigester/txDigestFunctions.ts Show resolved Hide resolved

src/txDigester/ipfsPublisher.ts Outdated

    
                  isPublisherActive = false

                } catch (error) {

                  isPublisherActive = false

                  console.error(`❌ Error while Uploading Digest (w/ Hash: ${data.hash}) to IPFS: `)

Contributor

aniketdivekar Sep 24, 2024

lets log cycle range also

Contributor Author

achal-singh Sep 24, 2024

I included the hash only to keep the log line short and the hash makes it easier to search in a DB (if someone wants to quickly look it up since its just 1 value).
I'll add in start and end cycles too.

src/txDigester/ipfsPublisher.ts Show resolved Hide resolved

achal-singh force-pushed the BLUE-280 branch from c691bb5 to eb61cc7 Compare

September 24, 2024 18:46

github-advanced-security bot found potential problems

View reviewed changes

src/txDigester/ipfsPublisher.ts

    
                  isPublisherActive = true

                  console.log(`Uploading TX Digest for Cycle Range ${data.cycleStart} to ${data.cycleEnd}`)

                  await client.setCurrentSpace(rootDID as Web3StorageRootDID)

                  console.log(`Uploading Data to Root-DID: ${rootDID}`)

Check warning

Code scanning / CodeQL

Log injection

Log entry depends on a [user-provided value](1).

Copilot Autofix

AI about 1 year ago

To fix the log injection issue, we need to sanitize the rootDID value before logging it. Specifically, we should remove any newline characters from the rootDID to prevent log forgery. This can be done using String.prototype.replace to ensure no line endings are present in the user input.

Suggested changeset 1

src/txDigester/ipfsPublisher.ts

@@ -39,3 +39,4 @@
                 await client.setCurrentSpace(rootDID as Web3StorageRootDID)
-                console.log(`Uploading Data to Root-DID: ${rootDID}`)
+                const sanitizedRootDID = rootDID.replace(/\n|\r/g, "")
+                console.log(`Uploading Data to Root-DID: ${sanitizedRootDID}`)

Copilot is powered by AI and may make mistakes. Always verify output.

src/txDigester/ipfsPublisher.ts Fixed Show fixed Hide fixed

src/txDigester/ipfsPublisher.ts

    
                try {

                  isPublisherActive = true

                  console.log('Initialising IPFS publisher...')

                  if (!adminEmail || !rootDID) {

Check failure

Code scanning / CodeQL

User-controlled bypass of security check

This condition guards a sensitive [action](1), but a [user-provided value](2) controls it. This condition guards a sensitive [action](3), but a [user-provided value](2) controls it.

src/txDigester/ipfsPublisher.ts

    
                try {

                  isPublisherActive = true

                  console.log('Initialising IPFS publisher...')

                  if (!adminEmail || !rootDID) {

Check failure

Code scanning / CodeQL

User-controlled bypass of security check

This condition guards a sensitive [action](1), but a [user-provided value](2) controls it. This condition guards a sensitive [action](3), but a [user-provided value](2) controls it.

src/txDigester/ipfsPublisher.ts Fixed Show fixed Hide fixed

src/txDigester/ipfsPublisher.ts Fixed Show fixed Hide fixed

src/txDigester/ipfsPublisher.ts Fixed Show fixed Hide fixed

src/txDigester/ipfsPublisher.ts Fixed Show fixed Hide fixed

achal-singh force-pushed the BLUE-280 branch 3 times, most recently from f6fc2c2 to f56041b Compare

September 25, 2024 11:57


          fix: Fixed ESM to CJS import issue & tweaks in the publisher logic

0d28777

achal-singh force-pushed the BLUE-280 branch from f56041b to 0d28777 Compare

September 27, 2024 09:34

github-advanced-security bot found potential problems

View reviewed changes

src/txDigester/ipfsPublisher.ts

    
                  }

                  client = await W3SClient.create()

                  console.log(`Logging into Web3.Storage with Account: ${maskedEmail(adminEmail)}`)

Check warning

Code scanning / CodeQL

Log injection

Log entry depends on a [user-provided value](1).

Copilot Autofix

AI about 1 year ago

To fix the log injection issue, we need to sanitize the adminEmail before logging it. Specifically, we should remove any newline characters from the adminEmail to prevent log injection attacks. This can be done using String.prototype.replace to remove newline characters.

Suggested changeset 1

src/txDigester/ipfsPublisher.ts

@@ -78,2 +78,6 @@
+            const sanitizeEmail = (email: string): string => {
+              return email.replace(/\n|\r/g, "");
+            }
             const maskedEmail = (email: string): string => {
@@ -94,3 +98,3 @@
-                console.log(`Logging into Web3.Storage with Account: ${maskedEmail(adminEmail)}`)
+                console.log(`Logging into Web3.Storage with Account: ${maskedEmail(sanitizeEmail(adminEmail))}`)
                 if (!isAuthWithEmail(adminEmail as Web3StorageAdminEmail)) {

Copilot is powered by AI and may make mistakes. Always verify output.

src/txDigester/ipfsPublisher.ts

    
                  console.log(`Logging into Web3.Storage with Account: ${maskedEmail(adminEmail)}`)

                  if (!isAuthWithEmail(adminEmail as Web3StorageAdminEmail)) {

                    console.log(`⏳ Owner of ${adminEmail} needs to approve the Web3.Storage Auth request to proceed.`)

Check warning

Code scanning / CodeQL

Log injection

Log entry depends on a [user-provided value](1).

src/txDigester/ipfsPublisher.ts

    
                  }

                  await client.setCurrentSpace(rootDID as Web3StorageRootDID)

                  console.log(`Current Space set to DID: ${rootDID}`)

Check warning

Code scanning / CodeQL

Log injection

Log entry depends on a [user-provided value](1).

Copilot Autofix

AI about 1 year ago

To fix the log injection issue, we need to sanitize the rootDID value before logging it. Specifically, we should remove any newline characters from the rootDID to prevent log injection. This can be done using the String.prototype.replace method to replace newline characters with an empty string.

Suggested changeset 1

src/txDigester/ipfsPublisher.ts

@@ -102,3 +102,4 @@
                 await client.setCurrentSpace(rootDID as Web3StorageRootDID)
-                console.log(`Current Space set to DID: ${rootDID}`)
+                const sanitizedRootDID = rootDID.replace(/\n|\r/g, "")
+                console.log(`Current Space set to DID: ${sanitizedRootDID}`)
                 isPublisherActive = false

Copilot is powered by AI and may make mistakes. Always verify output.

achal-singh mentioned this pull request

BLUE-283: IPFS-publisher's PM2 process & separate DB added #89

Open

mhanson-github force-pushed the dev branch from e875823 to 330bf6f Compare

April 7, 2025 04:26

mhanson-github force-pushed the dev branch from dcf1122 to 053fd38 Compare

April 26, 2025 04:34

mhanson-github force-pushed the dev branch from 8f3b44a to edc0f4d Compare

May 13, 2025 05:52

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Review effort [1-5]: 4