In this article, we will discuss the functions of storage providers in the Filecoin network, the role of the indexer, and the retrieval process for publicly available data.
When a storage deal is originally made, the client can opt to make the data publicly discoverable. If this is the case, the storage provider must publish an advertisement of the storage deal to the Interplanetary Network Indexer (IPNI). IPNI maps a CID to a storage provider (SP). This mapping allows clients to query the IPNI to discover where content is on Filecoin.
The IPNI also tracks which data transfer protocols you can use to retrieve specific CIDs. Currently, Filecoin SPs have the ability to serve retrievals over Graphsync, Bitswap, and HTTP. This is dependent on the SP setup.
If a client wants to retrieve publicly available data from the Filecoin network, then they generally follow this process.
Before the client can submit a retrieval deal to a storage provider, they first need to find which providers hold the data. To do this, the client sends a query to the Interplanetary Network Indexer.
Assuming the IPNI returns more than one storage provider, the client can select which provider they’d like to deal with. Here, they will also get additional details (if needed) based on the retrieval protocol they want to retrieve the content over.
The client then attempts to retrieve the data from the SP over Bitswap, Graphsync, or HTTP. Note that currently, clients can only get full-piece retrievals using HTTP.
Once the client has received the last chunk of data, the connection is closed.
Different retrieval workflows use different tools:
Serving PDP retrievals as a storage provider
Use Curio for PDP retrievals.
Serving PoRep retrievals as a storage provider
Use Boost to serve retrievals. Boost supports Graphsync retrievals by default, and storage providers can run booster-http for HTTP retrievals when configured.
Retrieving data as a client
Use Lassie to fetch content from Filecoin and IPFS using the best available retrieval path.
Retrieving application data with Filecoin Onchain Cloud
See the Filecoin Onchain Cloud retrieval docs for maintained Synapse SDK retrieval flows.
Last updated
