feat: allow overriding withCDN via env-var #55

SgtPooki · 2025-09-30T12:12:19Z

> npm run build && WITH_CDN=true npx filecoin-pin add ~/Downloads/images/IMG_1341.png


> filecoin-pin@0.6.0 build
> tsc

┌  Filecoin Pin Add
│
◇  ✓ File validated (17.8 MB)
│
◇  ✓ Connected to calibration
│
◇  ✓ Payment capacity verified
│
◇  ✓ File packed with root CID: bafybeierfvvhvb73ntvschtheysvpscpynjuxgsi7jtst4wg3n3jittuca
│
◇  ✓ IPFS content loaded (17.8 MB)
│
◇  ✓ Payment capacity verified
│
◇  ✓ Storage context ready
│
◇  ━━━ Add Complete ━━━
│
│  Network: calibration
│  
│  Add Details
│    File: /Users/sgtpooki/Downloads/js-colo-media/IMG_1341.png
│    Size: 17.8 MB
│    Root CID: bafybeierfvvhvb73ntvschtheysvpscpynjuxgsi7jtst4wg3n3jittuca
│  
│  Filecoin Storage
│    Piece CID: bafkzcibfqh67ubquflgzvzb76hayv6o4yqer2wpa7susa6xna3yfktvb5zxv37nbvqbq
│    Piece ID: 0
│    Data Set ID: 244
│  
│  Storage Provider
│    Provider ID: 3
│    Name: ezpdpz-calib
│    Direct Download URL: https://calib.ezpdpz.net/piece/bafkzcibfqh67ubquflgzvzb76hayv6o4yqer2wpa7susa6xna3yfktvb5zxv37nbvqbq
│  
│  Transaction
│    Hash: 0x88314b2d3f576fb0a248f82be0971cc77a4b68ad67acc3df4db013058ff6538e
│
└  Add completed successfully

> npx filecoin-pin data-set 244

┌  Filecoin Onchain Cloud Data Sets
│
◇  ━━━ Data Sets ━━━
│
│  Data Set #244 • live
│    Managed by Warm Storage: yes
│    CDN add-on: enabled
│    Pieces stored: 1
│    Leaf count: 588804
│    Total size: 18.0 MB
│    Client data set ID: 3
│    PDP rail ID: 485
│    CDN rail ID: 487
│    Cache-miss rail ID: 486
│    Payer: 0x44f08D1beFe61255b3C3A349C392C560FA333759
│    Payee: 0xa3971A7234a3379A1813d9867B531e7EeB20ae07
│    Service provider: 0xa3971A7234a3379A1813d9867B531e7EeB20ae07
│    Provider: ezpdpz-calib (ID 3)
│    Commission: 0.00%
│  
│  Provider Service
│    Service URL: https://calib.ezpdpz.net
│    Min piece size: 1.0 KB
│    Max piece size: 32.0 GB
│    Storage price: < 0.0001 USDFC/TiB/month
│    Min proving period: 30 epochs
│    Location: unknown
│    Payment token: USDFC (native)
│  
│  Metadata
│      source: filecoin-pin
│      withCDN: (empty)
│      withIPFSIndexing: (empty)
│  
│  
│  Pieces
│    Total pieces: 1
│    Unique CommPs: 1
│    Unique root CIDs: 1
│  
│    #0
│      CommP: bafkzcibfqh67ubquflgzvzb76hayv6o4yqer2wpa7susa6xna3yfktvb5zxv37nbvqbq
│      Root CID: bafybeierfvvhvb73ntvschtheysvpscpynjuxgsi7jtst4wg3n3jittuca
│
└  Data set inspection complete

README.md

SgtPooki

self review

jennijuju · 2025-09-30T12:29:55Z

if we were going to add this. 2 TODOs needs to be considered before this is merged is:

the cost of with cdn add on is different, we will need to account for it filecoin-pin payment deposit. I won’t suggest doing this change now because filcdn is changing how they are billing (transit from a fixed fee to an egress based fee)
filecoin-pin get should fetch from filcdn link first if cdn service is purchased

SgtPooki · 2025-09-30T12:51:13Z

@jennijuju I could add a warning in here during add/import to show users that withCDN isn't fully priced throughout the codebase and link to a tracking issue. Would you like me to do that so we don't miss those follow-up actions?

bajtos · 2025-09-30T13:04:40Z

The first version looks great to me! I would like filecoin-pin to print the FilBeam retrieval URL when CDN is enabled, but that can be added later.

I did not realise there are implications for filecoin-pin payment deposit. Showing a warning to users seems reasonable enough to me. 👍🏻

SgtPooki · 2025-09-30T22:19:14Z

opened up todos: #61 and have a warning with e687b57 (#55):

> npm run build && WITH_CDN=true npx filecoin-pin add ~/Downloads/js-colo-media/71713366785__2CBE4B4C-4404-470F-AEAD-DFF56D011409.png 

> filecoin-pin@0.6.0 build
> tsc

┌  Filecoin Pin Add
│
▲  CDN Pricing Notice
│
│  
│  Filecoin-pin currently does not support CDN pricing in payment calculations.
│  
│  This means:
│    • Deposit calculations may not be accurate for CDN storage
│    • You may need additional USDFC deposits for CDN-enabled uploads
│    • Filcdn is transitioning to egress-based billing (from fixed fees)
│  
│
◇  Do you want to proceed with CDN-enabled upload?
│  No
└  Add cancelled

Add failed: CDN pricing limitations warning cancelled

and for car import:

> filecoin-pin@0.6.0 build
> tsc

┌  Filecoin Pin CAR Import
│
▲  CDN Pricing Notice
│
│  
│  Filecoin-pin currently does not support CDN pricing in payment calculations.
│  
│  This means:
│    • Deposit calculations may not be accurate for CDN storage
│    • You may need additional USDFC deposits for CDN-enabled uploads
│    • Filcdn is transitioning to egress-based billing (from fixed fees)
│  
│
◇  Do you want to proceed with CDN-enabled upload?
│  No
└  Import cancelled

Import failed: CDN pricing limitations warning cancelled

BigLep

I think this environment variable option for now is fine. It unblocks being able to test Filecoin Beam with filecoin-pin data.

What we really need though is a followup issue about adding "withCdn" support in general. This means exposing i through the CLI as an config option, spitting out filecoinbeam URLs, etc. plus handling the payment issues highlighted in #61.

I will get that issue created later today.

bajtos

Awesome, thank you @SgtPooki! ❤️

rvagg · 2025-10-06T07:15:43Z

Quick mention in here (without looking at this), that withCDN changes the pricing. So far we've been avoiding the complexity of different prices for storage by just avoiding withCDN entirely. Now that it's in scope we need to think about how to present the pricing info, the optionality and how to adjust to user request to "upgrade".
Ideally this would be a simple add-on, perhaps even a default so you automatically sign up for your own custom IPFS gateway, but we need to prove this out first.

rvagg · 2025-10-06T09:23:05Z

I see I'm kind of duplicating Jen's comment about pricing already .. so my drive-by probably isn't very helpful here!

SgtPooki · 2025-10-06T12:37:25Z

@rvagg so are you good with us merging this so @bajtos can use and test (likely along with provider override)?

rvagg

ok, I don't love the optionality in envToBool but whatever

README.md

Co-authored-by: Rod Vagg <rod@vagg.org>

rvagg · 2025-10-11T04:05:45Z

needs a rebase and can be landed, I reckon @bajtos et. al. are going to like at least having the option on the table to start experimenting with this soon

SgtPooki · 2025-10-17T12:13:47Z

ok, I don't love the optionality in envToBool but whatever

Removed in 5e25fd0 (#55)

bajtos · 2025-10-21T08:37:25Z

I ran this new version today, and encountered two problems:

The dataset is create without the withCDN flag
The process crashes

◇  ━━━ Add Complete ━━━
│
│  Network: calibration
│
│  Add Details
│    File: /Users/bajtos/Downloads/anthony-delanoix-aDxmYZtYj7g-unsplash.jpg
│    Size: 4.4 MB
│    Root CID: bafybeig7sb5kcd3iswkv4tbrgt5hixmto3pdetxiuf723zx2i3pjlv4vae
│
│  Filecoin Storage
│    Piece CID: bafkzcibfr366gaisp6dbaxljjfjn4ecwjclbdo2h4slzudeeamguovwqrlwc4yhlse5a
│    Piece ID: 0
│    Data Set ID: 724
│
│  Storage Provider
│    Provider ID: 22
│    Name: ezpdpz-calib2
│    Direct Download URL: https://calib2.ezpdpz.net/piece/bafkzcibfr366gaisp6dbaxljjfjn4ecwjclbdo2h4slzudeeamguovwqrlwc4yhlse5a
│
│  Transaction
│    Hash: 0xc23db4f7cc65ec86daa76d8841a0d1aa1d8735abf20b280ec204e9eb1460d5fd
│
└  Add completed successfully

file:///Users/bajtos/src/filoz/filecoin-pin/node_modules/ethers/lib.esm/utils/errors.js:132
            error = new Error(message);
                    ^

Error: provider destroyed; cancelled request (operation="eth_unsubscribe", code=UNSUPPORTED_OPERATION, version=6.15.0)
    at makeError (file:///Users/bajtos/src/filoz/filecoin-pin/node_modules/ethers/lib.esm/utils/errors.js:132:21)
    at file:///Users/bajtos/src/filoz/filecoin-pin/node_modules/ethers/lib.esm/providers/provider-jsonrpc.js:282:40
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
  code: 'UNSUPPORTED_OPERATION',
  operation: 'eth_unsubscribe',
  shortMessage: 'provider destroyed; cancelled request'
}

Node.js v24.10.0

rvagg · 2025-10-21T08:40:02Z

oh, this is an ethers cleanup thing from Websockets, I thought we had this caught and ignored cause it'll be random and it doesn't indicate a problem

Ref: #55 (comment)

rvagg · 2025-10-21T08:54:53Z

Attempting more cleanup here, there's some async jobs that do later cleanup inside ethers that we're not catching. I think this will do it: #144

Ref: #55 (comment)

SgtPooki added 2 commits September 30, 2025 08:11

feat: allow overriding withCDN via env-var

14ddf56

chore: fix lint

0b82e20

SgtPooki commented Sep 30, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

SgtPooki commented Sep 30, 2025

View reviewed changes

docs: indicate default value for WITH_CDN

e37c582

fix: Warn users when enabling WITH_CDN

e687b57

BigLep approved these changes Oct 1, 2025

View reviewed changes

BigLep mentioned this pull request Oct 2, 2025

Filecoin Beam ("withCdn") support #61

Open

bajtos approved these changes Oct 3, 2025

View reviewed changes

rvagg approved these changes Oct 7, 2025

View reviewed changes

rvagg reviewed Oct 7, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

docs: update readme

1fd8fab

Co-authored-by: Rod Vagg <rod@vagg.org>

SgtPooki added 3 commits October 17, 2025 08:07

Merge branch 'master' into feat/allow-with-cdn-override

1170fdd

chore: fix bad merge

2351d1b

chore: remove envToBool

5e25fd0

SgtPooki merged commit 0a89ca8 into master Oct 20, 2025
17 of 18 checks passed

SgtPooki deleted the feat/allow-with-cdn-override branch October 20, 2025 12:23

github-actions bot mentioned this pull request Oct 20, 2025

chore(master): release 0.10.0 #138

Open

rvagg added a commit that referenced this pull request Oct 21, 2025

fix: more ethers.js cleanup silencing

d1a9e10

Ref: #55 (comment)

rvagg mentioned this pull request Oct 21, 2025

fix: more ethers.js cleanup silencing #144

Merged

bajtos mentioned this pull request Oct 21, 2025

fix: withCDN data set creation #145

Merged

SgtPooki pushed a commit that referenced this pull request Oct 21, 2025

fix: more ethers.js cleanup silencing (#144)

785af4a

Ref: #55 (comment)

feat: allow overriding withCDN via env-var #55

feat: allow overriding withCDN via env-var #55

Conversation

SgtPooki commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

SgtPooki left a comment

Choose a reason for hiding this comment

Uh oh!

jennijuju commented Sep 30, 2025

Uh oh!

SgtPooki commented Sep 30, 2025

Uh oh!

bajtos commented Sep 30, 2025

Uh oh!

SgtPooki commented Sep 30, 2025

Uh oh!

BigLep left a comment

Choose a reason for hiding this comment

Uh oh!

bajtos left a comment

Choose a reason for hiding this comment

Uh oh!

rvagg commented Oct 6, 2025

Uh oh!

rvagg commented Oct 6, 2025

Uh oh!

SgtPooki commented Oct 6, 2025

Uh oh!

rvagg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rvagg commented Oct 11, 2025

Uh oh!

SgtPooki commented Oct 17, 2025

Uh oh!

Uh oh!

bajtos commented Oct 21, 2025

Uh oh!

rvagg commented Oct 21, 2025

Uh oh!

rvagg commented Oct 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

SgtPooki commented Sep 30, 2025 •

edited

Loading