Skip to content
This repository has been archived by the owner on Jul 18, 2024. It is now read-only.

[DataCap Application] <OriginStorage> <NOAA Geostationary Operational Environmental Satellites (GOES) 16, 17 & 18> - (1/3) #1660

Closed
1 of 2 tasks
Tom-OriginStorage opened this issue Feb 22, 2023 · 62 comments

Comments

@Tom-OriginStorage
Copy link

Data Owner Name

National Oceanic and Atmospheric Administration

Data Owner Country/Region

United States

Data Owner Industry

Life Science / Healthcare

Website

https://registry.opendata.aws/noaa-goes .

Social Media

https://github.com/awslabs/open-data-docs/tree/main/docs/noaa/noaa-goes16
https://registry.opendata.aws/noaa-goes .

Total amount of DataCap being requested

5PiB

Weekly allocation of DataCap requested

450TiB

On-chain address for first allocation

f1gdxwhlgds3padyhj7rptu3ujxalu4ejc6tyv5xi

Custom multisig

  • Use Custom Multisig

Identifier

No response

Share a brief history of your project and organization

NEW GOES-18 Data!!! GOES-18 is now provisional and data has began streaming. Data files will be available between Provisional and the Operational Declaration of the satellite, however, data will have the caveat GOES-18 Preliminary, Non-Operational Data. The exception is during the interleave period when ABI Radiances and Cloud and Moisture Imagery data will be shared operationally via the NOAA Open Data Dissemination Program. GOES satellites (GOES-16, GOES-17, & GOES-18) provide continuous weather imagery and monitoring of meteorological and space environment data across North America. GOES satellites provide the kind of continuous monitoring necessary for intensive data analysis. They hover continuously over one position on the surface. The satellites orbit high enough to allow for a full-disc view of the Earth. Because they stay above a fixed spot on the surface, they provide a constant vigil for the atmospheric "triggers" for severe weather conditions such as tornadoes, flash floods, hailstorms, and hurricanes. When these conditions develop, the GOES satellites are able to monitor storm development and track their movements. SUVI products available in both NetCDF and FITS.

Is this project associated with other projects/ecosystem stakeholders?

Yes

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

NEW GOES-18 Data!!! GOES-18 is now provisional and data has began streaming. Data files will be available between Provisional and the Operational Declaration of the satellite, however, data will have the caveat GOES-18 Preliminary, Non-Operational Data. The exception is during the interleave period when ABI Radiances and Cloud and Moisture Imagery data will be shared operationally via the NOAA Open Data Dissemination Program. GOES satellites (GOES-16, GOES-17, & GOES-18) provide continuous weather imagery and monitoring of meteorological and space environment data across North America. GOES satellites provide the kind of continuous monitoring necessary for intensive data analysis. They hover continuously over one position on the surface. The satellites orbit high enough to allow for a full-disc view of the Earth. Because they stay above a fixed spot on the surface, they provide a constant vigil for the atmospheric "triggers" for severe weather conditions such as tornadoes, flash floods, hailstorms, and hurricanes. When these conditions develop, the GOES satellites are able to monitor storm development and track their movements. SUVI products available in both NetCDF and FITS.

Where was the data currently stored in this dataset sourced from

AWS Cloud

If you answered "Other" in the previous question, enter the details here

No response

How do you plan to prepare the dataset

No response

If you answered "other/custom tool" in the previous question, enter the details here

No response

Please share a sample of the data

2023-01-10 23:46:16    4.7 MiB SUVI-L1b-He303/2023/010/15/OR_SUVI-L1b-He303_G17_s20230101545352_e20230101545362_c20230101545546.fits
2023-01-10 23:46:15    4.8 MiB SUVI-L1b-He303/2023/010/15/OR_SUVI-L1b-He303_G17_s20230101545352_e20230101545362_c20230101545546.nc
2023-01-10 23:46:16    4.7 MiB SUVI-L1b-He303/2023/010/15/OR_SUVI-L1b-He303_G17_s20230101545443_e20230101545443_c20230101546044.fits
2023-01-10 23:46:15    4.8 MiB SUVI-L1b-He303/2023/010/15/OR_SUVI-L1b-He303_G17_s20230101545443_e20230101545443_c20230101546044.nc
2023-01-10 23:47:41    4.7 MiB SUVI-L1b-He303/2023/010/15/OR_SUVI-L1b-He303_G17_s20230101547043_e20230101547043_c20230101547247.fits
2023-01-10 23:47:42    4.8 MiB SUVI-L1b-He303/2023/010/15/OR_SUVI-L1b-He303_G17_s20230101547043_e20230101547043_c20230101547247.nc
2021-09-28 03:48:15   31.6 KiB index.html
Total Objects: 237687110
Total Size: 780.1 TiB

Confirm that this is a public dataset that can be retrieved by anyone on the Network

  • I confirm

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Yearly

For how long do you plan to keep this dataset stored on Filecoin

1.5 to 2 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, North America, South America

How will you be distributing your data to storage providers

Cloud storage (i.e. S3), HTTP or FTP server

How do you plan to choose storage providers

Slack, Big data exchange

If you answered "Others" in the previous question, what is the tool or platform you plan to use

No response

If you already have a list of storage providers to work with, fill out their names and provider IDs below

No response

How do you plan to make deals to your storage providers

Boost client

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

@large-datacap-requests
Copy link

Thanks for your request!

Heads up, you’re requesting more than the typical weekly onboarding rate of DataCap!

@large-datacap-requests
Copy link

Thanks for your request!
Everything looks good. 👌

A Governance Team member will review the information provided and contact you back pretty soon.

@Sunnyiscoming
Copy link
Collaborator

@Tom-OriginStorage It seems that you have submitted 5 applications.
#1661
#1662
#1664
#1681

Can you introduce your organization and Sps you will cooperate with.

@Tom-OriginStorage
Copy link
Author

hi @Sunnyiscoming I'm glad to receive your reply
OriginStorage as a distributed storage and blockchain practitioner, we started the research and application of IPFS in 2018 and caught up with Filecoin again in 2019. After closely studying the Filecoin white paper and related technologies, we believe it was the only business scenario that could be landed since the development of blockchain technology. ORIGIN storage was finally set up, and lots of human and material resources were invested into Filecoin during the continuous operation.

  1. In 2018, some of our codes were merged. For example:
    btfslib/go-libp2p-quic-transport@944de2e

  2. We went through all the test networks of Filecoin (T02417 is still active in the main network: https://filfox.info/en/address/f02417):
    Testnet1: T017267
    Testnet2: T01030
    Spacerace: T02417, T017273

  3. Co-developed a decentralized netdisk with Blockcasting Cloud in slingshot. (Slingshot registration ID: 676715867) (https://github.com/litianc/slingshot/blob/master/participants/BlockcastingCloud.md)

  4. After continuous investment and efforts, the company has become an essential contributor to the Filecoin network, with a total of 600P+ computing power. We also have an integrated Filecoin storage solution covering IDC server room, hardware, and software. To improve data encapsulation and storage efficiency, we conduct deep optimization in the modules of encapsulation efficiency, scheduling, space-time proof, cluster architecture, etc.

  5. We strive to promote the development of IPFS and Filecoin by establishing partnerships with several web 2.0 industry giants and facilitating the storage of valid data in Filecoin.

@Tom-OriginStorage
Copy link
Author

  1. Participated in the entire process of Filecoin's test net/space race/mainnet development(https://filfox.info/en/address/f02417)

  2. Started in 2019, actively promoted IPFS and Filecoin technology. In October 2020, while many miners did not understand the protocol lab, we wrote many articles to explain the Filecoin economic model and spoke in public to support PL.

  3. We developed the decentralized network disk in Slingshot (ID: 676715867) together with Blockcasting Cloud.
    (https://github.com/litianc/slingshot/blob/master/participants/BlockcastingCloud.md)

  4. We continuously participate in the development of Filecoin, issue reporting, and code contribution.
    (parent-grinding fault overcheck lotus#7833.
    fix ticket expiration check lotus#6635)

  5. We became Venus Master to participate in the construction of the community and make contributions: slack: cryptostack-zou (https://filecoinproject.slack.com/archives/CEHHJNJS3/p1635833794026900)

  6. Become the leading SP of Solana Lab to store blockchain snapshot data in Filecoin.

  7. Become an early builder member of fvm.

  8. Actively participate in the construction of the filecoin community and contribute to the command line HD wallet: https://github.com/llifezou/fil-wallet

@Tom-OriginStorage
Copy link
Author

We keep working on Filecoin, and our colleague has proposed several issues and pr involved in the development of Filecoin.
filecoin-project/lotus#6635
filecoin-project/lotus#7844
filecoin-project/lotus#7512
filecoin-project/lotus#8208
filecoin-project/lotus#7132
filecoin-project/lotus#6669

@Tom-OriginStorage
Copy link
Author

@Sunnyiscoming
We have communicated and confirmed the SPs for cooperation. They are mainly located in Southeast Asia, Greater China and North America. At present, we are sure to use
f01914268
f01896036
f02024315
f02013434
f02044834

At present, we are still talking about the remaining SPs. We expect that more than 20 SPs will join in total;

According to the demand, we have applied for 25PiB in total this time. We have made the preliminary preparations. We hope we can get your support. Thank you

@Sunnyiscoming
Copy link
Collaborator

Total Size: 780.1 TiB
You have submitted 3 applications for 15 PB Datacap. How many copies you want to store?

@Tom-OriginStorage
Copy link
Author

Total Size: 780.1 TiB You have submitted 3 applications for 15 PB Datacap. How many copies you want to store?

We are ready to store to 10 SPs, which have been communicated and confirmed, and the node ID is still being prepared. Because the CAR file is larger than 0.5 times the sector size and smaller than the sector size, the 780T is estimated to need nearly 15P of storage space. Because the final actual size is uncertain, I only submitted 2 applications for the time being(#1681), totaling 10P. After the 10P is used, I will decide whether to apply for the final 5P according to the situation. Thank you

@Sunnyiscoming Sunnyiscoming self-assigned this Mar 7, 2023
@Sunnyiscoming
Copy link
Collaborator

Datacap Request Trigger

Total DataCap requested

5PiB

Expected weekly DataCap usage rate

400TiB

Client address

f1gdxwhlgds3padyhj7rptu3ujxalu4ejc6tyv5xi

@large-datacap-requests
Copy link

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f1gdxwhlgds3padyhj7rptu3ujxalu4ejc6tyv5xi

DataCap allocation requested

225TiB

Id

c93750b0-cf73-45dc-ad33-cc1b2ad0b00b

@Bitrise0111
Copy link

@Tom-OriginStorage
I have looked through the explanation, it makes sense. also your account is an active member of the Network, KYC process can be skipped.
One more question: Will you provide us with a list of CIDs after data be uploaded on the network for these four LDNs so that we can verify the data?

Total Size: 780.1 TiB You have submitted 3 applications for 15 PB Datacap. How many copies you want to store?

We are ready to store to 10 SPs, which have been communicated and confirmed, and the node ID is still being prepared. Because the CAR file is larger than 0.5 times the sector size and smaller than the sector size, the 780T is estimated to need nearly 15P of storage space. Because the final actual size is uncertain, I only submitted 2 applications for the time being(#1681), totaling 10P. After the 10P is used, I will decide whether to apply for the final 5P according to the situation. Thank you

@Tom-OriginStorage
Copy link
Author

@Tom-OriginStorage I have looked through the explanation, it makes sense. also your account is an active member of the Network, KYC process can be skipped. One more question: Will you provide us with a list of CIDs after data be uploaded on the network for these four LDNs so that we can verify the data?

Total Size: 780.1 TiB You have submitted 3 applications for 15 PB Datacap. How many copies you want to store?

We are ready to store to 10 SPs, which have been communicated and confirmed, and the node ID is still being prepared. Because the CAR file is larger than 0.5 times the sector size and smaller than the sector size, the 780T is estimated to need nearly 15P of storage space. Because the final actual size is uncertain, I only submitted 2 applications for the time being(#1681), totaling 10P. After the 10P is used, I will decide whether to apply for the final 5P according to the situation. Thank you

Yes, this is no problem. Thank you, notary

Copy link

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacecpvwk23zl3rn2yo5ng4bbvi4k5bzvmy4vs3nzyzn772gdp7vzasq

Address

f1gdxwhlgds3padyhj7rptu3ujxalu4ejc6tyv5xi

Datacap Allocated

225.00TiB

Signer Address

f1nknj7ayq4o43czrtdoauggtwl43fbqatmqis3yy

Id

c93750b0-cf73-45dc-ad33-cc1b2ad0b00b

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacecpvwk23zl3rn2yo5ng4bbvi4k5bzvmy4vs3nzyzn772gdp7vzasq

@graceonline9
Copy link

It seems that the applicant is a notary. In order to avoid private dealings.
@Tom-OriginStorage Can you explain which organizations these sps belong to? Can you support retrieval?
image
The same sps combination appears in several other applications
#1650
It is suspected that these sp's belong to one organization.
@cryptowhizzard Can you check this one?

@Tom-OriginStorage
Copy link
Author

Tom-OriginStorage commented Mar 9, 2023

Hi, @graceonline9 OriginStorgae is a notary. OriginStorgae is not only a notary, but also a SP. For more information, please refer to the information I posted above. We are still looking for other SPs to join. These are listed for the time being. The total amount is expected to reach 15--20 SPs, what is your guess source, which organization do you belong to? If you have any questions, you can find me on slack, or you can communicate at the notary meeting or T&T meeting

@luobin544
Copy link

checker:manualTrigger

@filplus-checker-app
Copy link

DataCap and CID Checker Report Summary1

Retrieval Statistics

  • Overall Graphsync retrieval success rate: 74.79%
  • Overall HTTP retrieval success rate: 0.00%
  • Overall Bitswap retrieval success rate: 0.00%

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients2

✔️ No CID sharing has been observed.

Full report

Click here to view the CID Checker report.
Click here to view the Retrieval Dashboard.
Click here to view the Retrieval report.

Footnotes

  1. To manually trigger this report, add a comment with text checker:manualTrigger

  2. To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

@luobin544
Copy link

Check for excellent success rates and provide support

Copy link

Request Approved

Your Datacap Allocation Request has been approved by the Notary

Message sent to Filecoin Network

bafy2bzacedillw7jmypszcsg4vd2trbpme2xeh2f5wvnkisd2l3efntl6tjpy

Address

f1gdxwhlgds3padyhj7rptu3ujxalu4ejc6tyv5xi

Datacap Allocated

1.71PiB

Signer Address

f1tbd632f6w62glfaf7wjpimacbnjiz26poyoes2q

Id

80a3cc5d-e3a7-40f2-a196-a7a942ae441a

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedillw7jmypszcsg4vd2trbpme2xeh2f5wvnkisd2l3efntl6tjpy

@large-datacap-requests
Copy link

DataCap Allocation requested

Request number 6

Multisig Notary address

f02049625

Client address

f1gdxwhlgds3padyhj7rptu3ujxalu4ejc6tyv5xi

DataCap allocation requested

1.96TiB

Id

805e0c88-0f44-4f9f-a741-733bffd0eb58

@large-datacap-requests
Copy link

Stats & Info for DataCap Allocation

Multisig Notary address

f02049625

Client address

f1gdxwhlgds3padyhj7rptu3ujxalu4ejc6tyv5xi

Rule to calculate the allocation request amount

400% of weekly dc amount requested

DataCap allocation requested

1.96TiB

Total DataCap granted for client so far

1.5925616025924683e+52YiB

Datacap to be granted to reach the total amount requested by the client (5PiB)

1.5925616025924683e+52YiB

Stats

Number of deals Number of storage providers Previous DC Allocated Top provider Remaining DC
95086 33 1.71PiB 12.96 427.82TiB

Copy link

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzaceddyi5xxhcocifqbdc636hq7rpgr32mkur4exwlddszdyo7y2rx5s

Address

f1gdxwhlgds3padyhj7rptu3ujxalu4ejc6tyv5xi

Datacap Allocated

1.96TiB

Signer Address

f1txfsjmix4vlzido4dkildrnbw26owtlbslexmpa

Id

805e0c88-0f44-4f9f-a741-733bffd0eb58

You can check the status of the message here: https://filfox.info/en/message/bafy2bzaceddyi5xxhcocifqbdc636hq7rpgr32mkur4exwlddszdyo7y2rx5s

@filplus-checker-app
Copy link

DataCap and CID Checker Report Summary1

Retrieval Statistics

  • Overall Graphsync retrieval success rate: 74.14%
  • Overall HTTP retrieval success rate: 0.00%
  • Overall Bitswap retrieval success rate: 0.00%

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients2

⚠️ CID sharing has been observed. (Top 3)

Full report

Click here to view the CID Checker report.
Click here to view the Retrieval Dashboard.
Click here to view the Retrieval report.

Footnotes

  1. To manually trigger this report, add a comment with text checker:manualTrigger

  2. To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.