The Filecoin storage model consists of three main components:

• Providers

• Deals

• Sectors

Providers

Providers offer storage and retrieval services to network users. There are two types of providers:

• Storage Providers

• Retrieval Providers

Storage providers

Storage providers, often called SPs, are responsible for storing files and data for clients on the network. They also provide cryptographic proofs to verify that data is stored securely. The majority of providers on the Filecoin network are SPs.

Retrieval providers

Retrieval providers, or RPs, specialize in delivering quick access to data rather than long-term storage. While many storage providers also offer retrieval services, stand-alone RPs are increasingly joining the network to enhance data accessibility.

Deals

In the Filecoin network, SPs and RPs offer storage or retrieval services to clients through deals. These deals are negotiated between two parties and outline terms such as data size, price, duration, and collateral.

The deal-making process initially occurs off-chain. Once both parties agree to the terms, the deal is published on-chain for network-wide visibility and validation.

Sectors

Sectors are the fundamental units of provable storage where storage providers securely store client data and generate PoSt (Proof of Spacetime) for the Filecoin network. Sectors come in standard sizes, typically 32 GiB or 64 GiB, and have a set lifespan that providers can extend before it expires.

Was this page helpful?

Deal making

The lifecycle of a deal within the storage market includes four distinct phases:

• Discovery: The client identifies potential storage providers (SPs) and requests their prices.

• Negotiation: After selecting an SP, both parties agree to the terms of the deal.

• Publishing: The deal is published on-chain.

• Handoff: The deal is added to a sector, where the SP can provide cryptographic proofs of data storage.

Filecoin Plus

Filecoin Plus aims to maximize useful storage on the Filecoin network by incentivizing the storage of meaningful and valuable data. It offers verified clients low-cost or free storage through a system called datacap, a storage quota that boosts incentives for storage providers.

Verified clients use datacap allocated by community-selected allocators to store data on the network. In exchange for storing verified deals, storage providers receive a 10x boost in storage power, which increases their block rewards as an incentive.

• Datacap: A token allocated to verified clients to spend on storage deals, offering a 10x quality multiplier for deals.

• Allocators: Community-selected entities responsible for verifying storage clients and allocating datacap tokens.

• Verified Clients: Active participants with datacap allocations for their data storage needs.

Storage on-ramps

To simplify data storage on the Filecoin network, several tools offer streamlined integration of Filecoin and IPFS storage for applications or smart contracts.

These storage helpers provide libraries that abstract the Filecoin deal-making process into simple API calls. They also store data on IPFS for efficient and fast content retrieval.

Available storage helpers include:

• lighthouse.storage: An SDK for builders, providing tools for storing data from dApps.

• web3.storage: A user-friendly client for accessing decentralized protocols like IPFS and UCAN.

• Akave: A modular L2 solution for decentralized data management, combining Filecoin storage with encryption and easy-to-use interfaces.

• Storacha: A decentralized hot storage network for scalable, user-owned data with decentralized permissions, leveraging Filecoin.

• Curio: A next-gen platform within the Filecoin ecosystem, streamlining storage provider operations.

• boost.filecoin.io: A tool for storage providers to manage data onboarding and retrieval on the Filecoin network.

Was this page helpful?

Native currency

Filecoin’s native currency, FIL, is a utility token that incentivizes persistent storage on the Filecoin network. Storage providers earn FIL by offering reliable storage services or committing storage capacity to the network. With a maximum circulating supply of 2 billion FIL, no more than 2 billion Filecoin will ever exist.

As a utility token aligned with the network’s long-term growth, Filecoin issuance depends on the network’s provable utility and growth. Most of the Filecoin supply is only minted as the network achieves specific growth and utility milestones.

Filecoin uses a dual minting model for block reward distribution:

Baseline minting

Up to 770 million FIL tokens are minted based on network performance. Full release of these tokens would only occur if the Filecoin network reaches a yottabyte of storage capacity within 20 years, approximately 1,000 times the capacity of today’s cloud storage.

Simple minting

An additional 330 million FIL tokens are released on a 6-year half-life schedule, with 97% of these tokens projected to be released over about 30 years.

Additionally, 300 million FIL tokens are held in a mining reserve to incentivize future mining models.

Vesting

Mining rewards are subject to a vesting schedule to support long-term network alignment. For instance, 75% of block rewards earned by miners vest linearly over 180 days, while 25% are immediately accessible, improving miner cash flow and profitability. Further, FIL tokens are vested to Protocol Labs teams and the Filecoin Foundation over six years and to SAFT investors over three years, as outlined in the vesting schedule.

Collateral and slashing

To ensure network security and reliable storage, storage providers must lock FIL as pledge collateral during block reward mining. Pledge collateral is based on projected block rewards a miner could earn. Collateral and all earned rewards are subject to slashing if the storage fails to meet reliability standards throughout a sector’s lifecycle.

Total supply

FIL’s maximum circulating supply is capped at 2 billion FIL. However, this maximum will never be reached, as a portion of FIL is permanently removed from circulation through gas fees, penalties, and other mechanisms.

Was this page helpful?

Filecoin is a peer-to-peer network that enables reliable, decentralized file storage through built-in economic incentives and cryptographic proofs. Users pay storage providers—computers that store and continuously prove file integrity—to securely store their files over time. Anyone can join Filecoin as a user seeking storage or as a provider offering storage services. Storage availability and pricing aren’t controlled by any single entity; instead, Filecoin fosters an open market for file storage and retrieval accessible to all.

Filecoin is built on the same technology as the IPFS protocol. IPFS is a distributed storage network that uses content addressing to provide permanent data references without dependency on specific devices or cloud providers. Filecoin differs from IPFS by introducing an incentive layer that promotes reliable storage and consistent access to data.

Filecoin’s use cases are diverse, ranging from Web3-native NFT storage to metaverse and gaming assets, as well as incentivized, permanent storage. It also offers a cost-effective solution for archiving traditional Web2 datasets, making it a strong alternative to conventional cloud storage.

For instance, NFT.Storage leverages Filecoin for decentralized NFT content and metadata storage. Likewise, organizations like the Shoah Foundation and the Internet Archive use Filecoin for content preservation and backup.

Filecoin is compatible with various data types, including audio and video files. This versatility allows Web3 platforms like Audius and Huddle01 to use Filecoin as a decentralized storage backend for music streaming and video conferencing.

Was this page helpful?

Choose your own path to start exploring Filecoin:

Was this page helpful?

Built-in actors

Built-in actors are how the Filecoin network manages and updates global state. The global state of the network at a given epoch can be thought of as the set of blocks agreed upon via network consensus in that epoch. This global state is represented as a state tree, which maps an actor to an actor state. An actor state describes the current conditions for an individual actor, such as its FIL balance and its nonce. In Filecoin, actors trigger a state transition by sending a message. Each block in the chain can be thought of as a proposed global state, where the block selected by network consensus sets the new global state. Each block contains a series of messages and a checkpoint of the current global state after the application of those messages. The Filecoin Virtual Machine (FVM) is the Filecoin network component that is in charge of the execution of all actor code.

A basic example of how actors are used in Filecoin is the process by which storage providers prove storage and are subsequently rewarded. The process is as follows:

2. The storage provider is awarded storage power based on whether the proof is valid or not.

Blocks

Each block in the Filecoin chain contains the following:

• Inline data such as current block height.

• A pointer to the current state tree.

• A pointer to the set of messages that, when applied to the network, generated the current state tree.

State tree

• Actors, like FIL balance, nonce, and a pointer (CID) to actor state data.

• Messages in the current block

Messages

Like the state tree, a Merkle Directed Acyclic Graph (Merkle DAG) is used to map the set of messages for a given block. Nodes in the messages may contain information on:

• The actor the message was sent to

• The actor that sent the message

• Target method to call on the actor being sent the message

• A cryptographic signature for verification

• The amount of FIL transferred between actors

Actor code

The code that defines an actor in the Filecoin network is separated into different methods. Messages sent to an actor contain information on which method(s) to call and the input parameters for those methods. Additionally, actor code interacts with a runtime object, which contains information on the general state of the network, such as the current epoch, cryptographic signatures, and proof validations. Like smart contracts in other blockchains, actors must pay a gas fee, which is some predetermined amount of FIL to offset the cost (network resources used, etc.) of a transaction. Every actor has a Filecoin balance attributed to it, a state pointer, a code that tells the system what type of actor it is, and a nonce, which tracks the number of messages sent by this actor.

Types of built-in actors

The 11 different types of built-in actors are as follows:

CronActor

The CronActor sends messages to the StoragePowerActor and StorageMarketActor at the end of each epoch. The messages sent by CronActor indicate to StoragePowerActor and StorageMarketActor how they should maintain the internal state and process deferred events. This system actor is instantiated in the genesis block and interacts directly with the FVM.

InitActor

The InitActor can initialize new actors on the Filecoin network. This system actor is instantiated in the genesis block and maintains a table resolving a public key and temporary actor addresses to their canonical ID addresses. The InitActor interacts directly with the FVM.

AccountActor

The AccountActor is responsible for user accounts. Account actors are not created by the InitActor but by sending a message to a public-key style address. The account actor updates the state tree with a new actor address and interacts directly with the FVM.

RewardActor

The RewardActor manages unminted Filecoin tokens and distributes rewards directly to miner actors, where they are locked for vesting. The reward value used for the current epoch is updated at the end of an epoch. The RewardActor interacts directly with the FVM.

StorageMarketActor

The StorageMarketActor is responsible for processing and managing on-chain deals. This is also the entry point of all storage deals and data into the system. This actor keeps track of storage deals and the locked balances of both the client storing data and the storage provider. When a deal is posted on-chain through the StorageMarketActor, the actor will first check if both transacting parties have sufficient balances locked up and include the deal on-chain. Additionally, the StorageMarketActor holds Storage Deal Collateral provided by the storage provider to collateralize deals. This collateral is returned to the storage provider when all deals in the sector successfully conclude. This actor does not interact directly with the FVM.

StorageMinerActor

The StorageMinerActor is created by the StoragePowerActor and is responsible for storage mining operations and the collection of mining proofs. This actor is a key part of the Filecoin storage mining subsystem, which ensures a storage miner can effectively commit storage to Filecoin and handles the following:

• Committing new storage

• Continuously proving storage

• Declaring storage faults

• Recovering from storage faults

This actor does not interact directly with the FVM.

MultisigActor

The MultisigActor is responsible for dealing with operations involving the Filecoin wallet and represents a group of transaction signers with a maximum of 256. Signers may be external users or the MultisigActor itself. This actor does not interact directly with the FVM.

PaymentChannelActor

The PaymentChannelActor creates and manages payment channels, a mechanism for off-chain microtransactions for Filecoin dApps to be reconciled on-chain at a later time with less overhead than a standard on-chain transaction and no gas costs. Payment channels are uni-directional and can be funded by adding to their balance. To create a payment channel and deposit fund, a user calls the PaymentChannelActor. This actor does not interact directly with the FVM.

StoragePowerActor

The StoragePowerActor is responsible for keeping track of the storage power allocated to each storage miner and has the ability to create a StorageMinerActor. This actor does not interact directly with the FVM.

VerifiedRegistryActor

The VerifiedRegistryActor is responsible for managing Filecoin Plus clients. This actor can add a verified client to the Filecoin Plus program, remove and reclaim expired DataCap allocations, and manage claims. This actor does not interact directly with the FVM.

SystemActor

User actors (smart contracts)

A user actor is code defined by any developer that can interact with the FVM, otherwise known as a smart contract.

With the FVM, actors can be written in Solidity. In future updates, any language that compiles to WASM will be supported. With user actors, users can create and enforce custom rules for storing and accessing data on the network. The FVM is responsible for actors and ensuring that they are executed correctly and securely.

Basic retrieval

Currently, Filecoin nodes support direct retrieval from the storage miners who originally stored the data. Clients can send retrieval requests directly to a storage provider and pay a small amount of FIL to retrieve their data.

To request data retrieval, clients need to provide the following information to the storage provider:

• Storage Provider ID: The ID of the storage provider where the data is stored.

• Payload CID: Also known as Data CID.

• Address: The address initially used to create the storage deal.

Saturn

Tipsets

The Filecoin blockchain consists of a chain of tipsets rather than individual blocks. A tipset is a set of blocks with the same height and parent tipset, allowing multiple storage providers to produce blocks in each epoch to increase network throughput.

Each tipset is assigned a weight, enabling the consensus protocol to guide nodes to build on the heaviest chain. This adds a level of security to the Filecoin network by preventing interference from nodes attempting to produce invalid blocks.

Actors

An actor in the Filecoin blockchain is similar to a smart contract in the Ethereum Virtual Machine. It functions as an ‘object’ within the Filecoin network, with a state and a set of methods for interaction.

Built-in actors

Several built-in system actors power the Filecoin network as a decentralized storage network:

• System actor: General system actor.

• Init actor: Initializes new actors and records the network name.

• Cron actor: Scheduler that runs critical functions at every epoch.

• Account actor: Manages user accounts (non-singleton).

• Reward actor: Manages block rewards and token vesting (singleton).

• Storage miner actor: Manages storage mining operations and validates storage proofs.

• Storage power actor: Tracks storage power allocation for each provider.

• Storage market actor: Manages storage deals.

• Multisig actor: Handles Filecoin multi-signature wallet operations.

• Payment channel actor: Sets up and settles payment channel funds.

• Datacap actor: Manages datacap tokens.

• Verified registry actor: Manages verified clients.

• Ethereum Address Manager (EAM) actor: Assigns Ethereum-compatible addresses on Filecoin, including EVM smart contract addresses.

• Ethereum Virtual Machine (EVM) account actor: Represents an external Ethereum identity backed by a secp256k1 key.

User-programmable actors

With the maturity of the FVM, developers can write actors and deploy them on the Filecoin network, similar to other blockchains' smart contracts. User-programmable actors can interact with built-in actors via the exported API from built-in actors.

Distributed randomness

Nodes

Filecoin nodes are categorized by the services they provide to the storage network, including chain verifier nodes, client nodes, storage provider nodes, and retrieval provider nodes. All participating nodes must provide chain verification services.

Filecoin supports multiple protocol implementations to enhance security and resilience. Active implementations include:

Addresses

In the Filecoin network, addresses identify actors in the Filecoin state. Each address encodes information about the corresponding actor, making it easy to use and resistant to errors. Filecoin has five address types. Mainnet addresses start with f, and Testnet addresses start with t.

• f0/t0: ID address for an actor in a human-readable format, such as f0123261 for a storage provider.

• f1/t1: secp256k1 wallet address, generated from an encrypted secp256k1 public key.

• f2/t2: Address assigned to an actor (smart contract) in a way that ensures stability across network forks.

• f3/t3: BLS wallet address, generated from a BLS public key.

• f4/t4: Address created and assigned to user-defined actors by customizable "address management" actors. This address can receive funds before an actor is deployed.

• f410/t410: Address space managed by the Ethereum Address Manager (EAM) actor, allowing Ethereum-compatible addresses to interact seamlessly with the Filecoin network. Ethereum addresses can be cast as f410/t410 addresses and vice versa, enabling compatibility with existing Ethereum tools.

Consensus

Expected consensus

Expected Consensus (EC) is the consensus algorithm underlying Filecoin. EC is a probabilistic, Byzantine fault-tolerant protocol that conducts a leader election among storage providers each epoch to determine which provider submits a block. Similar to proof-of-stake, Filecoin’s leader election relies on proof-of-storage, meaning the probability of being elected depends on how much provable storage power a miner contributes to the network. This storage power is recorded in the storage power table, managed by the Storage Power Actor.

Block production process

The block production process for each epoch is as follows:

• Elect leaders from eligible miners.

• Miners check if they are elected.

• Elected miners generate WinningPoSt using randomness.

• Miners build and propagate a block.

• Verify the winning miner and election.

• Select the heaviest chain to add blocks.

Finality

EC enforces soft finality, where miners at round N reject blocks forking off before round N - F (where F is set to 900). This ensures finality without compromising chain availability.

Proofs

Filecoin operates on proof-of-storage, where miners offer storage space and provide proofs to verify data storage.

Proof of replication

With proof-of-replication (PoRep), storage providers prove they have created a unique copy of the client’s data for the network.

Proof of spacetime

Storage providers must continuously prove that they are storing clients' data throughout the entire duration of the storage deal. The proof-of-spacetime (PoSt) process includes two types of challenges:

• Winning PoSt: Verifies that a storage provider holds a copy of the data at a specific point in time.

• Window PoSt: Confirms that the data has been consistently stored over a defined period.

Slashing

If storage providers fail to maintain reliable uptime or act maliciously, they face penalties through a process called slashing. Filecoin enforces two types of slashing:

• Storage Fault Slashing: Penalizes providers who fail to maintain healthy and reliable storage sectors.

• Consensus Fault Slashing: Penalizes providers attempting to disrupt the security or availability of the consensus process.

Compute-over-data

For example, Bacalhau provides a platform for public, transparent, and verifiable distributed computation, allowing users to run Docker containers and WebAssembly (Wasm) images as tasks on data stored in InterPlanetary File System (IPFS).

Filecoin is uniquely positioned to support large-scale off-chain computation because storage providers have compute resources, such as GPUs and CPUs, colocated with their data. This setup enables a new paradigm where computations occur directly on the data where it resides, reducing the need to move data to external compute nodes.

Filecoin Virtual Machine

The Filecoin Virtual Machine (FVM) is a runtime environment for executing smart contracts on the Filecoin network. These smart contracts allow users to run bounded computations and establish rules for storing and accessing data. The FVM ensures that these contracts are executed securely and reliably.

Initially, the FVM supports smart contracts written in Solidity, with plans to expand to other languages that compile to Wasm, as outlined in the FVM roadmap.

By enabling compute-over-states on the Filecoin network, the FVM unlocks a wide range of potential use cases. Examples include:

Data Organizations

FVM enables a new kind of organization centered around data.

Data DAOs and tokenized datasets

The FVM makes it possible to create and manage decentralized and autonomous organizations (Data DAOs) focused on data curation and preservation. Data DAOs allow groups of individuals or organizations to govern and monetize data access, pooling returns into a shared treasury to fund preservation and growth. These data tokens can also be exchanged among peers or used to request computation services, such as validation, analysis, feature detection, and machine learning.

Perpetual storage

The FVM allows users to store data once and use repair and replication bots to manage ongoing storage deals, ensuring perpetual data storage. Through smart contracts, users can fund a wallet with FIL, allowing storage providers to maintain data storage indefinitely. Repair bots monitor these storage deals and replicate data across providers as needed, offering long-term data permanence.

Financial services for miners

The FVM can facilitate unique financial services tailored for storage providers (SPs) in the Filecoin ecosystem.

Lending and staking protocols

Users can lend Filecoin to storage providers to be used as storage collateral, earning interest in return. Loans may be undercollateralized based on SP performance history, with reputation scores generated from on-chain data. Loans can also be automatically repaid to investors using a multisig wallet, which includes lenders and a third-party arbitrator. New FVM-enabled smart contracts create yield opportunities for FIL holders while supporting the growth of storage services on the network.

Insurance

SPs may require financial products to protect against risks in providing storage solutions. Attributes such as payment history, operational length, and availability can be used to underwrite insurance policies, shielding SPs from financial impacts due to storage faults or token price fluctuations.

Core chain infrastructure

The FVM is expected to achieve feature parity with other persistent EVM chains, supporting critical infrastructure for decentralized exchanges and token bridges.

Decentralized exchanges

To facilitate on-chain token exchange, the FVM may support decentralized exchanges like Uniswap or Sushi, or implement decentralized order books similar to Serum on Solana.

Token bridges

Although not an immediate focus, token bridges will eventually connect Filecoin to EVM, Move, and Cosmos chains, enabling cross-chain wrapped tokens. While Filecoin currently offers unique value without needing to bootstrap liquidity from other chains, long-term integration with other blockchains is anticipated.

If you are interested in building these use cases, the following solution blueprints may be helpful:

Filecoin EVM

Since Filecoin nodes support the Ethereum JSON-RPC API, FEVM is compatible with existing EVM development tools, such as Hardhat, Brownie, and MetaMask. Most smart contracts deployed to Filecoin require minimal adjustments, if any. For example, new ERC-20 tokens can be launched on Filecoin or bridged to other chains.

Developers can choose between deploying actors on the FEVM or native FVM: for optimal performance, actors should be written in languages that compile to Wasm and deployed to the native FVM. For familiarity with Solidity and EVM tools, the FEVM is a convenient alternative.

In summary, the FEVM provides a straightforward path for Web3 developers to begin building on Filecoin using familiar tools and languages, while gaining native access to Filecoin storage deals.

The primary difference between FEVM and EVM contracts is that FEVM contracts can interact directly with Filecoin-specific actors, such as miner actors, which are inaccessible to Ethereum contracts. To enable seamless integration, a Filecoin-Solidity API library has been developed to facilitate interactions with Filecoin-specific actors and syscalls.

Mainnet

Mainnet is the live production network that connects all nodes on the Filecoin network. It operates continuously without resets.

Testnets

Test networks, or testnets, are versions of the Filecoin network that simulate various aspects of the mainnet. They are intended for testing and should not be used for production applications or services.

Calibration

The Calibration testnet offers the closest simulation of the mainnet. It provides realistic sealing performance and hardware requirements due to the use of finalized proofs and parameters, allowing prospective storage providers to test their setups. Storage clients can also store and retrieve real data on this network, participating in deal-making workflows and testing storage/retrieval functionalities. Calibration testnet uses the same sector size as the mainnet.

• Public endpoint

• Blockchain explorer

• Calibration Faucet - Chainsafe

• Calibration Faucet - Zondax

• Calibration Faucet - Forest Explorer

Was this page helpful?

Was this page helpful?

Blocks

In Filecoin, a block consists of:

• A block header

• A list of messages contained in the block

• A signed copy of each message listed

Every block refers to at least one parent block; that is, a block produced in a prior epoch.

A message represents communication between two actors and thus changes in network state. The messages are listed in their order of appearance, deduplicated, and returned in canonical order of execution. So, in other words, a block describes all changes to the network state in a given epoch.

Blocktime

Blocktime is a concept that represents the average time it takes to mine or produce a new block on a blockchain. In Ethereum, for example, the blocktime is approximately 15 seconds on average, meaning that a new block is added to the Ethereum blockchain roughly every 15 seconds.

In the Filecoin network, storage providers compete to produce blocks by providing storage capacity and participating in the consensus protocol. The block time determines how frequently new blocks are added to the blockchain, which impacts the overall speed and responsiveness of the network.

Filecoin has a block time of 30 seconds, and this duration was chosen for two main reasons:

• Hardware requirements: If the block time were faster while maintaining the same gas limit or the number of messages per block, it would lead to increased hardware requirements. This includes the need for more storage space to accommodate the larger chain data resulting from more frequent block production.

• Storage provider operations: The block time also takes into account the various operations that occur during that duration on the storage provider (SP) side. As SPs generate new blocks, the 30-second block time allows for the necessary processes and computations to be carried out effectively. If the blocktime were shorter, SPs would encounter significantly more blocktime failures.

By considering these factors, the Filecoin network has established a block time of 30 seconds, balancing the need for efficient operations and hardware requirements.

Tipsets

As described in Consensus, multiple potential block producers may be elected via Expected Consensus (EC) to create a block in each epoch, which means that more than one valid block may be produced in a given epoch. All valid blocks with the same height and same parent block are assembled into a group called a tipset.

Benefits of tipsets

In other blockchains, blocks are used as the fundamental representation of network state, that is, the overall status of each participant in the network at a given time. However, this structure has the following disadvantages:

• Potential block producers may be hobbled by network latency.

• Not all valid work is rewarded.

• Decentralization and collaboration in block production are not incentivized.

Because Filecoin is a chain of tipsets rather than individual blocks, the network enjoys the following benefits:

• All valid blocks generated in a given round are used to determine network state, increasing network efficiency and throughput.

• All valid work is rewarded (that is, all validated block producers in an epoch receive a block reward).

• All potential block producers are incentivized to produce blocks, disincentivizing centralization and promoting collaboration.

• Because all blocks in a tipset have the same height and parent, Filecoin is able to achieve rapid convergence in the case of forks.

In summary, blocks, which contain actor messages, are grouped into tipsets in each epoch, which can be thought of as the overall description of the network state for a given epoch.

Tipsets in the Ethereum JSON-RPC

Wherever you see the term block in the Ethereum JSON-RPC, you should mentally read tipset. Before the inclusion of the Filecoin EVM runtime, there was no single hash referring to a tipset. A tipset ID was the concatenation of block CIDs, which led to a variable-length ID and poor user experience.

With the Ethereum JSON-RPC, we introduced the concept of the tipset CID for the first time. It is calculated by hashing the former tipset key using a Blake-256 hash. Therefore, when you see the term:

• block hash, think tipset hash.

• block height, think tipset epoch.

• block messages, think messages in all blocks in a tipset, in their order of appearance, deduplicated and returned in canonical order of execution.

Was this page helpful?

This page covers how Drand is used within the Filecoin network. For more information on Drand generally, take a look at the project’s documentation.

Randomness outputs

By polling the appropriate endpoint, a Filecoin node will get back a Drand value formatted as follows:

{
  "round": 367,
  "signature": "b62dd642e939191af1f9e15bef0f0b0e9562a5f570a12a231864afe468377e2a6424a92ccfc34ef1471cbd58c37c6b020cf75ce9446d2aa1252a090250b2b1441f8a2a0d22208dcc09332eaa0143c4a508be13de63978dbed273e3b9813130d5",
  "previous_signature": "afc545efb57f591dbdf833c339b3369f569566a93e49578db46b6586299422483b7a2d595814046e2847494b401650a0050981e716e531b6f4b620909c2bf1476fd82cf788a110becbc77e55746a7cccd47fb171e8ae2eea2a22fcc6a512486d"
}

• signature: the threshold BLS signature on the previous signature value and the current round number round.

• previous_signature: the threshold BLS signature from the previous Drand round.

• round: the index of randomness in the sequence of all random values produced by this Drand network.

The message signed is the concatenation of the round number treated as a uint64 and the previous signature. At the moment, Drand uses BLS signatures on the BLS12-381 curve with the latest v7 RFC of hash-to-curve, and the signature is made over G1.

Polling the network

Filecoin nodes fetch the Drand entry from the distribution network of the selected Drand network.

Drand distributes randomness using multiple distribution channels such as HTTP servers, S3 buckets, gossiping, etc. Simply put, the Drand nodes themselves will not be directly accessible by consumers; rather, highly-available relays will be set up to serve Drand values over these distribution channels.

On initialization, Filecoin initializes a Drand client with chain info that contains the following information:

• Period: the period of time between each Drand randomness generation.

• GenesisTime: at which the first round in the Drand randomness chain is created.

• PublicKey: the public key to verify randomness.

• GenesisSeed: the seed that has been used for creating the first randomness.

It is possible to simply store the hash of this chain info and to retrieve the contents from the Drand distribution network as well on the /info endpoint.

Thereafter, the Filecoin client can call Drand’s endpoints:

• /public/latest to get the latest randomness value produced by the beacon.

• /public/<round> to get the randomness value produced by the beacon at a given round.

Using Drand

Drand is used as a randomness beacon for leader election in Filecoin. While Drand returns multiple values with every call to the beacon (see above), Filecoin blocks need only store a subset of these in order to track a full Drand chain. This information can then be mixed with on-chain data for use in Filecoin.

Edge cases and outages

Any Drand beacon outage will effectively halt Filecoin block production. Given that new randomness is not produced, Filecoin miners cannot generate new blocks. Specifically, any call to the Drand network for a new randomness entry during an outage should be blocked in Filecoin.

After a beacon downtime, Drand nodes will work to quickly catch up to the current round. In this way, the above time-to-round mapping in Drand used by Filecoin remains invariant after this catch-up following downtime.

While Filecoin miners were not able to mine during the Drand outage, they will quickly be able to run leader election thereafter, given a rapid production of Drand values. We call this a catch-up period.

During the catch-up period, Filecoin nodes will backdate their blocks in order to continue using the same time-to-round mapping to determine which Drand round should be integrated according to the time. Miners can then choose to publish their null blocks for the outage period, including the appropriate Drand entries throughout the blocks, per the time-to-round mapping. Or, as is more likely, try to craft valid blocks that might have been created during the outage.

Based on the level of decentralization of the Filecoin network, we expect to see varying levels of miner collaboration during this period. This is because there are two incentives at play: trying to mine valid blocks during the outage to collect block rewards and not falling behind a heavier chain being mined by a majority of miners who may or may not have ignored a portion of these blocks.

In any event, a heavier chain will emerge after the catch-up period and mining can resume as normal.

Was this page helpful?

Overview

In the Filecoin blockchain, network consensus is achieved using the Expected Consensus (EC) algorithm, a probabilistic, Byzantine fault-tolerant consensus protocol. At a high level, EC achieves consensus by running a secret, fair, and verifiable leader election at every epoch where a set number of participants may become eligible to submit a block to the chain based on fair and verifiable criteria.

Properties

Expected Consensus (EC) has the following properties:

• Each epoch has potentially multiple elected leaders who may propose a block.

• A winner is selected randomly from a set of network participants weighted according to the respective storage power they contribute to the Filecoin network.

• All blocks proposed are grouped together in a tipset, from which the final chain is selected.

• A block producer can be verified by any participant in the network.

• The identity of a block producer is anonymous until they release their block to the network.

Steps

In summary, EC involves the following steps at each epoch:

1. A storage provider checks to see if they are elected to propose a block by generating an election proof.

2. Zero, one, or multiple storage providers may be elected to propose a block. This does not mean that an elected participant is guaranteed to be able to submit a block. In the case where:

   • No storage providers are elected to propose a block in a given epoch; a new election is run in the next epoch to ensure that the network remains live.

   • One or more storage providers are elected to propose a block in a given epoch; each must generate a WinningPoSt proof-of-storage to be eligible to actually submit a block.

3. Each potential block producer elected generates a storage proof using WinningPoSt for a randomly selected sector within in short window of time. Potential block producers that fail this step are not eligible to produce a block. In this step, the following could occur:

   • All potential block producers fail WinningPoSt, in which case EC returns to step 1 (described above).

   • One or more potential block producers pass WinningPoSt, which means they are eligible to submit that block to the epochs tipset.

4. Blocks generated by block producers are grouped into a tipset.

5. The tipset that reflects the biggest amount of committed storage on the network is selected.

6. Using the selected tipset, the chain state is propagated.

7. EC returns to step 1 in the next epoch.

Was this page helpful?

Different blockchains use different cryptographic proving systems (proofs) based on the network’s specific purpose, goals, and functionality. Regardless of which method is used, proofs have the following in common:

• All blockchain networks seek to achieve consensus and rely on proofs as part of this process.

• Proofs incentivize network participants to behave in certain ways and allow the network to penalize participants who do not abide by network standards.

• Proofs allow decentralized systems to agree on a network state without a central authority.

Proof-of-Work and Proof-of-Stake are both fairly common proof methods:

• Proof-of-Work: nodes in the network solve complex mathematical problems to validate transactions and create new blocks,

• Proof-of-Stake: nodes in the network are chosen to validate transactions and create new blocks based on the amount of cryptocurrency they hold and “stake” in the network.

The Filecoin network aims to provide useful, reliable storage to its participants. With a traditional centralized entity like a cloud storage provider, explicit trust is placed in the entity itself that the data will be stored in a way that meets some minimum set of standards such as security, scalability, retrievability, or replication. Because the Filecoin network is a decentralized network of storage providers (SPs) distributed across the globe, network participants need an automated, trustless, and decentralized way to validate that an SP is doing a good job of handling the data.

In particular, the Filecoin proof process must verify the data was properly stored at the time of the initial request and is continuing to be stored based on the terms of the agreement between the client and the SP. In order for the proof processes to be robust, the process must:

• Target a random part of the data.

• Occur at a time interval such that it is not possible, profitable, or rational for an SP to discard and re-fetch the copy of data.

In Filecoin, this process is known as Proof-of-Storage, and consists of two distinct types of proofs:

• Proof of Replication (PoRep): a procedure used at the time of initial data storage to validate that an SP has created and stored a unique copy of some piece of data.

• Proof of Spacetime (PoST): a procedure to validate that an SP is continuing to store a unique copy of some piece of data.

Proof-of-Replication (PoRep)

In the Filecoin storage lifecycle process, Proof-of-Replication (PoRep) is used when an SP agrees to store data on behalf of a client and receives a piece of client data. In this process:

1. The data is placed into a sector.

2. The sector is sealed by the SP.

3. A unique encoding, which serves as proof that the SP has replicated a copy of the data they agreed to store, is generated (described in Sealing as proof).

4. The proof is compressed.

5. The result of the compression is submitted to the network as certification of storage.

Sealing as proof

The unique encoding created during the sealing process is generated using the following pieces of information:

• The data is sealed.

• The storage provider who seals the data.

• The time at which the data was sealed.

Because of the principles of cryptographic hashing, a new encoding will be generated if the data changes, the storage provider sealing the data changes, or the time of sealing changes. This encoding is unique and can be used to verify that a specific storage provider did, in fact, store a particular piece of client data at a specific time.

Proof-of-Spacetime (PoSt)

After a storage provider has proved that they have replicated a copy of the data that they agreed to store, the SP must continue to prove to the network that:

• They are still storing the requested data.

• The data is available.

• The data is still sealed.

Because this method is concerned with proving that data is being stored in a particular space for a particular period or at a particular time, it is called Proof-of-Spacetime (PoSt). In Filecoin, the PoSt process is handled using two different sub-methods, each of which serves a different purpose:

• WinningPoSt is used to prove that an SP selected using an election process has a replica of the data at the specific time that they were asked and is used in the block consensus process.

• WindowPoSt is used to prove that, for any and all SPs in the network, a copy of the data that was agreed to be stored is being continuously maintained over time and is used to audit SPs continuously.

WinningPoSt

WinningPoSt is used to prove that an SP selected via election has a replica of the data at the specific time that they were asked and is specifically used in Filecoin to determine which SPs may add blocks to the Filecoin blockchain.

At the beginning of each epoch, a small number of SPs are elected to mine new blocks using the Expected Consensus algorithm, which guarantees that validators will be chosen based on a probability proportional to their power. Each of the SPs selected must submit a WinningPoSt, proof that they have a sealed copy of the data that they have included in their proposed block. The deadline to submit this proof is the end of the current epoch and was intentionally designed to be short, making it impossible for the SP to fabricate the proof. Successful submission grants the SP:

• The block reward .

• The opportunity to charge other nodes fees in order to include their messages in the block.

If an SP misses the submission deadline, no penalty is incurred, but the SP misses the opportunity to mine a block and receive the block reward.

WindowPoSt

WindowPoSt is used to prove that, for any and all SPs in the network, a copy of the data that was agreed to be stored is being continuously maintained over time and is used to audit SPs continuously. In WindowPoSt, all SPs must demonstrate the availability of all sectors claimed every proving period. Sector availability is not proved individually; rather, SPs must prove a whole partition at once, and that sector must be proved by the deadline assigned (a 30-minute interval in the proving period).

The more sectors an SP has pledged to store, the more the partitions of sectors that the SP will need to prove per deadline. As this requires that the SP has access to sealed copies of each of the requested sectors, it makes it irrational for the SP to seal data every time they need to provide a WindowPoSt proof, thus ensuring that SPs on the network are continuously maintaining the data agreed to. Additionally, failure to submit WindowPoSt for a sector will result in the SPs’ pledge collateral being forfeited and their storage power being reduced.

Was this page helpful?

Uses

FIL plays a vital role in incentivizing users to participate in the Filecoin network and ensuring its smooth operation. Here are some ways in which FIL is used on the Filecoin network:

Network payments

When a user wants to store data on the Filecoin network, they pay in FIL to the storage providers who offer their storage space. The payment is made in advance, for a certain amount of time that the data will be stored on the network.

In addition, storage providers choose their own terms and payment mechanisms for providing storage and retrieval services so other options (such as fiat payments) can be available.

Blockchain rewards

Storage providers are also rewarded with FIL for providing their storage space and performing other useful tasks on the network. FIL is used to reward storage providers who validate and add new blocks to the Filecoin blockchain. Providers receive a block reward in FIL for each new block they add to the blockchain and also earn transaction fees in FIL for processing storage and retrieval transactions.

Governance

As members of the Filecoin community, FIL holders are encouraged to participate in the Filecoin governance process. They can do so by proposing, deliberating, designing, and/or contributing to consensus for network changes, alongside other stakeholders in the Filecoin community- including implementers, Core Devs, storage providers, and other ecosystem partners. Learn more about the Filecoin Governance process.

Denominations

FIL, NanoFIL, and PicoFIL are all denominated in the same cryptocurrency unit, but they represent different levels of precision and granularity. For most users, FIL is the main unit of measurement and is used for most transactions and payments on the Filecoin network.

Much like how a US penny represents a fraction of a US dollar, there are many ways to represent value using Filecoin. This is because some actions on the Filecoin network require substantially less value than one whole FIL. The different denominations of FIL you may see referenced across the ecosystem are:

Was this page helpful?

Exchanges

A cryptocurrency exchange is a digital platform where users can buy, sell, and trade cryptocurrencies for other cryptocurrencies or traditional fiat currencies like USD, EUR, or JPY.

Cryptocurrency exchanges provide a marketplace for users to trade their digital assets and are typically run by private companies that facilitate these transactions. These exchanges can differ in terms of fees, security protocols, and the variety of cryptocurrencies they support.

Users can typically sign up for an account with a cryptocurrency exchange, deposit funds into their account, and then use those funds to purchase or sell cryptocurrencies at the current market price. Some exchanges offer advanced trading features like margin trading, stop-loss orders, and trading bots.

It's important to note that while cryptocurrency exchanges can offer convenience and liquidity for traders, they also come with risks like hacking and regulatory uncertainty. Therefore, users should take precautions to protect their funds and do their own research before using any particular exchange.

Supported exchanges

There are many exchanges that allow users to buy, sell, and trade FIL. Websites like coingecko.com and coinmarketcap.com keep track of which exchanges support which cryptocurrencies. You can use these lists to help you decide which exchange to use.

Once you have found an exchange you want to use, you will have to create an account with that exchange. Many exchanges have strict verification and Know-Your-Customer (KYC) processes in place, so it may take a few days to create your account. However, most large exchanges can verify your information in a few minutes.

Purchasing cryptocurrency varies from exchange to exchange, but the process is usually something like this:

1. Add funds to your exchange account in your local currency (USD, EUR, YEN, etc.).

2. Exchange your local currency for FIL at a set price.

Address compatibility

Some exchanges allow users to fund and withdraw FIL using any of the Filecoin address type. However, some exchanges only support one or a handful of the available address types. Most exchanges do not currently support f410 addresses.

If your exchange does not yet support Filecoin Eth-style 0x addresses, you must create a wallet to relay the funds through. Take a look at the Transfer FIL page for details on how to transfer your funds safely.

Fiat on-ramps

A fiat on-ramp is a service or platform that allows individuals to convert traditional fiat currencies such as the US dollar, Euro, or any other government-issued currency into cryptocurrencies. These on-ramps serve as entry points for people who want to start participating in the cryptocurrency ecosystem by purchasing digital currencies with their money but don't want to sign up with a cryptocurrency exchange.

FIL is supported by a number of fiat on-ramps, such as:

• Changelly

• ChangeNow

• MoonPay

• Ramp Network

• Simplex.

Users are cautioned to do their own due diligence with respect to choosing a fiat on-ramp provider.

Crypto ATMs

Crypto ATMs, also known as Bitcoin ATMs, are kiosks that allow individuals to buy and/or sell cryptocurrencies in exchange for fiat currency like the US dollar. They function similarly to traditional ATMs but are not connected to a bank account. Instead, they connect the user directly to a cryptocurrency exchange.

Using a Bitcoin ATM often comes with higher fees than online exchanges. Fees can vary, but they can range anywhere from 5% to 15% or even more per transaction.

Test FIL

If you’re looking to get FIL to test your applications on a testnet like Calibration, then check how to get test tokens! Test FIL is often referred to as tFIL.

Was this page helpful?

All Filecoin addresses begin with an f to indicate the network (Filecoin), followed by any of the address prefix numbers (0, 1, 2, 3, 4) to indicate the address type. There are five address types:

Each of the address types is described below.

Actor IDs

All actors have a short integer assigned to them by InitActor, a unique actor that can create new actors. This integer that gets assigned is the ID of that actor. An ID address is an actor’s ID prefixed with the network identifier and the address type.

Actor ID addresses are not robust in the sense that they depend on chain state and are defined on-chain by the InitActor. Additionally, actor IDs can change for a brief time after creation if the same ID is assigned to different actors on different forks. Actor ID addresses are similar to monotonically increasing numeric primary keys in a relational database. So, when a chain reorganization occurs (similar to a rollback in a SQL database), you can refer to the same ID for different rows. The expected consensus algorithm will resolve the conflict. Once the state that defines a new ID reaches finality, no changes can occur, and the ID is bound to that actor forever.

For example, the mainnet burn account ID address, f099, is structured as follows:

  Address type
  |
f 0 9 9
|    |
|    Actor ID
|
Network identifier

ID addresses are often referred to by their shorthand f0.

Public keys

Actors managed directly by users, like accounts, are derived from a public-private key pair. If you have access to a private key, you can sign messages sent from that actor. The public key is used to derive an address for the actor. Public key addresses are referred to as robust addresses as they do not depend on the Filecoin chain state.

Public key addresses allow devices, like hardware wallets, to derive a valid Filecoin address for your account using just the public key. The device doesn’t need to ask a remote node what your ID address is. Public key addresses provide a concise, safe, human-readable way to reference actors before the chain state is final. ID addresses are used as a space-efficient way to identify actors in the Filecoin chain state, where every byte matters.

Filecoin supports two types of public key addresses:

• secp256k1 addresses that begin with the prefix f1.

• BLS addresses that begin with the prefix f3.

For BLS addresses, Filecoin uses curve bls12-381 for BLS signatures, which is a pair of two related curves, G1 and G2.

Filecoin uses G1 for public keys, as G1 allows for a smaller representation of public keys and G2 for signatures. This implements the same design as ETH2 but contrasts with Zcash, which has signatures on G1 and public keys on G2. However, unlike ETH2, which stores private keys in big-endian order, Filecoin stores and interprets private keys in little-endian order.

Public key addresses are often referred to by their shorthand, f1 or f3.

Actors

Actor addresses provide a way to create robust addresses for actors not associated with a public key. They are generated by taking a sha256 hash of the output of the account creation. The ZH storage provider has the actor address f2plku564ddywnmb5b2ky7dhk4mb6uacsxuuev3pi and the ID address f01248.

Actor addresses are often referred to by their shorthand, f2.

Extensible user-defined actors

Filecoin supports extensible, user-defined actor addresses through the f4 address class, introduced in Filecoin Improvement Proposal (FIP) 0048. The f4 address class provides the following benefits to the network:

• A predictable addressing scheme to support interactions with addresses that do not yet exist on-chain.

• User-defined, custom addressing systems without extensive changes and network upgrades.

• Support for native addressing schemes from foreign runtimes such as the EVM.

An f4 address is structured as f4<address-manager-actor-id>f<new-actor-id>, where <address-manager-actor-id> is the actor ID of the address manager, and <new-actor-id> is the arbitrary actor ID chosen by that actor. An address manager is an actor that can create new actors and assign an f4 address to the new actor.

Currently, per FIP 0048, f4 addresses may only be assigned by and in association with specific, built-in actors called address managers. Once users are able to deploy custom WebAssembly actors, this restriction will likely be relaxed in a future FIP.

As an example, suppose an address manager has an actor ID (an f0 address) 123, and that address manager creates a new actor. Then, the f4 address of the actor created by the address manager is f4123fa3491xyz, where f4 is the address class, 123 is the actor ID of the address manager, f is a separator, and a3491xyz is the arbitrary <new-actor-id> chosen by that actor.

Was this page helpful?

When someone sends cryptocurrency to your wallet address, the transaction is recorded on the blockchain network, and the funds are added to your wallet balance. Similarly, when you send cryptocurrency from your wallet to someone else’s wallet, the transaction is recorded on the blockchain network, and the funds are deducted from your wallet balance.

There are various types of cryptocurrency wallets, including desktop, mobile, hardware, and web-based wallets, each with its own unique features and levels of security. It’s important to choose a reputable and secure wallet to ensure the safety of your digital assets.

Compatible wallets

We do not provide technical support for any of these wallets. Please use caution when researching and using the wallets listed below. Wallets that have conducted third-party audits of their open-source code by a reputable security auditor are marked recommended below.

Hot versus cold

A hot wallet refers to any wallet that is permanently connected to the internet. They can be mobile, desktop, or browser-based. Hot wallets make it faster and easier to access digital assets but could be vulnerable to online attacks. Therefore, it is recommended to keep large balances in cold wallets and only use hot wallets to hold funds that need to be accessed frequently.

Cold wallets most commonly refer to hardware wallet devices shaped like a USB stick. They are typically offline and only connected to the internet for transactions. Accessing a cold wallet requires physical possession of the device plus knowledge of the private key, which makes them more resistant to theft. Cold wallets can be less convenient and are most useful for storing larger balances securely.

Security

Wallets that have gone through an audit have had their codebase checked by a recognized security firm for security vulnerabilities and potential leaks. However, just because a wallet has had an audit does not mean that it’s 100% bug-proof. Be incredibly cautious when using unaudited wallets.

Never share your seed phrase, password, or private keys. Bad actors will often use social engineering tactics such as phishing emails or posing as customer service or tech support to lure users into handing over their private key or seed phrase.

Add a wallet to our list

If you know of a wallet that supports Filecoin, you can submit a pull request to this page and add it!

• If the wallet is a mobile wallet, it must be available on both Android and iOS.

• The wallet must have been audited. The results of this audit must be public.

What is IPC?

What is horizontal scalability and why is it important for dApps?

For decentralized applications (dApps), there are several key motivations to adopt scaling - performance, decentralization, security. The challenge is that these factors are known to be conflicting goals.

How does IPC achieve horizontal scalability?

IPC is a scaling solution intentionally designed to achieve considerable performance, decentralization and security for dApps.

Subnets are organized in a hierarchy, with one parent subnet being able to spawn infinite child subnets. Within a hierarchical subsystem, subnets can seamlessly communicate with each other, reducing the need for cross-chain bridges.

How is IPC unique as a scaling solution?

Earlier, we talked about the challenge of scaling solutions to balance performance, security and decentralization. IPC is a standout framework that strikes a considerable balance between these factors, to achieve breakthroughs in scaling.

• Highly customizable without compromising security. Most L2 scaling solutions today either inherit the L1's security features but don't have their own consensus algorithms (e.g. rollups), or do the reverse (e.g. sidechains). They are also deployed in isolation and require custom bridges or protocols to transfer assets and state between L2s that share a common L1, which are vulnerable to attacks. In contrast, IPC subnets have their own consensus algorithms, inherit security features from the parent subnet and have native cross-net communication, eliminating the need for bridges.

Applications of IPC

Here are some practical examples of how IPC improves the performance of dApps:

• Distributed Computation: Spawn ephemeral subnets to run distributed computation jobs.

• Coordination: Assemble into smaller subnets for decentralized orchestration with high throughput and low fees.

• Localization: Leverage proximity to improve performance and operate with very low latency in geographically constrained settings.

• Partition tolerance: Deploy blockchain substrates in mobile settings or other environments with limited connectivity.

With better performance, lower fees and faster transactions, IPC can rapidly improve horizontal and vertical markets with decentralized technology:

• Decentralized Finance (DeFi): Enabling truly high-frequency trading and traditional backends with verifiability and privacy.

• Big Data and Data Science: Multiple teams are creating global-scale distributed compute networks to enable Data Science analysis on Exabytes of decentralized stored data.

• Metaverse/Gaming: Enabling real-time tracking of player interactions in virtual worlds.

• DAOs: Assemble into smaller subnets for decentralized orchestration with high throughput and low fees. Partition tolerance: Deploy blockchain substrates in mobile settings or other environments with limited connectivity.

Get involved

Lassie

Lassie is a simple retrieval client for IPFS and Filecoin. It finds and fetches your data over the best retrieval protocols available. Lassie makes Filecoin retrieval easy. While Lassie is powerful, the core functionality is expressed in a single CLI command:

Lassie also provides an HTTP interface for retrieving IPLD data from IPFS and Filecoin peers. Developers can use this interface directly in their applications to retrieve the data.

Lassie fetches content in content-addressed archive (CAR) form, so in most cases, you will need additional tooling to deal with CAR files. Lassie can also be used as a library to fetch data from Filecoin from within your application. Due to the diversity of data transport protocols in the IPFS ecosystem, Lassie is able to use the Graphsync or Bitswap protocols, depending on how the requested data is available to be fetched. One prominent use case of Lassie as a library is the Saturn Network. Saturn nodes fetch content from Filecoin and IPFS through Lassie in order to serve retrievals.

Retrieve using Lassie

1. Or download and install Lassie using the Go package manager:

You now have everything you need to retrieve a file with Lassie and extract the contents with go-car.

Retrieve

To retrieve data from Filecoin using Lassie, all you need is the CID of the content you want to download.

The video below demonstrates how Lassie can be used to render content directly from Filecoin and IPFS.

Lassie and go-car can work together to retrieve and extract data from Filecoin. All you need is the CID of the content to download.

This command uses a | to chain two commands together. This will work on Linux or macOS. Windows users may need to use PowerShell to use this form. Alternatively, you can use the commands separately, as explained later on this page.

An example of fetching and extracting a single file, identified by its CID:

Basic progress information, similar to the output shown below, is displayed:

The resulting file is a tar archive:

Lassie CLI usage

Lassie's usage for retrieving data is as follows:

• -p is an optional flag that tells Lassie that you would like to see detailed progress information as it fetches your data.

  For example:

• -o is an optional flag that tells Lassie where to write the output to. If you don’t specify a file, it will append .car to your CID and use that as the output file name.

If you specify -p, the output will be written to stdout so it can be piped to another command, such as go-car, or redirected to a file.

• <CID>/path/to/content is the CID of the content you want to retrieve and an optional path to a specific file within that content. Example:

A CID is always necessary, and if you don’t specify a path, Lassie will attempt to download the entire content. If you specify a path, Lassie will only download that specific file or, if it is a directory, the entire directory and its contents.

go-car CLI usage

The car extract command can be used to extract files and directories from a CAR:

• -f is an optional flag that tells go-car where to read the input from. If omitted, it will read from stdin, as in our example above where we piped lassie fetch -o - output to car extract.

• /path/to/file/or/directory is an optional path to a specific file or directory within the CAR. If omitted, it will attempt to extract the entire CAR.

• <OUTPUT_DIR> is an optional argument that tells go-car where to write the output to. If omitted, it will be written to the current directory.

If you supply -p, as in the above example, it will attempt to extract the content directly to stdout. This will only work if we are extracting a single file.

In the example above, where we fetched a file named lidar-data.tar, the > operator was used to redirect the output of car extract to a named file. This is because the content we fetched was raw file data that did not have a name encoded. In this case, if we didn’t use - and > filename, go-car would write to a file named unknown. In this instance, go-car was used to reconstitute the file from the raw blocks contained within Lassie’s CAR output.

go-car has other useful commands. The first is car ls, which can be used to list the contents of a CAR. The second is car inspect, which can be used to inspect the contents of the CAR and optionally verify the integrity of a CAR.

And there we have it! Downloading and managing data from Filecoin is super simple when you use Lassie and Go-car!

Lassie HTTP daemon

The Lassie HTTP daemon is an HTTP interface for retrieving IPLD data from IPFS and Filecoin peers. It fetches content from peers known to have it and provides the resulting data in CAR format.

Lassie’s CAR format

Using ChainID

ChainID.network is a website that lets users easily connect their wallets to EVM-compatible blockchains. ChainID is the simplest way to add the Filecoin network to your MetaMask wallet.

Manual process

If you can't or don't want to use ChainID, you can add the Filecoin network to your MetaMask manually.

Prerequisites

Before we get started, you’ll need the following:

Steps

The process for configuring MetaMask to use Filecoin is fairly simple but has some very specific variables that you must copy exactly.

1. Open your browser and open the MetaMask plugin. If you haven’t opened the MetaMask plugin before, you’ll be prompted to create a new wallet. Follow the prompts to create a wallet.

2. Click the user circle and select Settings.

3. Select Networks.

4. Click Add a network.

5. Scroll down and click Add a network manually.

6. Enter the following information into the fields:

2. Review the values in the fields and click Save.

3. The Filecoin network should now be shown in your MetaMask window.

4. Done!

You can now use MetaMask to interact with the Filecoin network.

Ledger hardware wallet

Install the Ledger app

Before you can connect MetaMask to your Ledger, you must install the Filecoin Ledger app on your Ledger device.

1. Open Ledger Live and navigate to My Ledger.

2. Connect your Ledger device and unlock it.

3. Confirm that you allow My Ledger to access your Ledger device. You can do that by clicking both buttons on your Ledger device simultaneously.

4. Go back to Ledger Live on your computer.

5. In My Ledger, head over to App catalog and search for Filecoin.

6. Click Install.

Enable expert-mode

MetaMask requires that the Filecoin app on your Ledger device is set to Expert mode.

1. Open the Filecoin app on your Ledger device.
2. Use the buttons on your device to navigate to Expert mode.
3. Press both buttons simultaneously to enable Expert mode.

Connect to MetaMask

Once you have installed the Filecoin app on your Ledger device and enabled expert mode, you can connect your device to MetaMask.

1. Open your browser and open the MetaMask extension.

2. In the Accounts menu, select Add hardware wallet.
3. Select Ledger
4. A list of accounts should appear. Select an 0x... account.
5. Done!

That's it! You've now successfully connected your Ledger device to MetaMask. When you submit any transactions through MetaMask using this account, the Filecoin Ledger app will prompt you for a confirmation on the Ledger device.

You may see a blind signing warning on your MetaMask device. This is expected, and is the reason why Expert Mode must be enabled before you can interact with the Filecoin Ledger app.

Storing NFTs

General data storage

Verifiable storage

Filecoin has built-in processes to check the history of files and verify that they have been stored correctly over time. Every storage provider proves that they are maintaining their files in every 24-hour window. Clients can efficiently scan this history to confirm that their files have been stored correctly, even if the client was offline at the time. Any observer can check any storage provider’s track record and will notice if the provider has been faulty or offline in the past.

Open market

In Filecoin, file storage and retrieval deals are negotiated in open markets. Anybody can join the Filecoin network without needing permission. By lowering the barriers to entry, Filecoin enables a thriving ecosystem of many independent storage providers.

Competitive prices

Prices for storage and retrieval are determined by supply and demand, not corporate pricing departments. Filecoin makes reliable storage available at hyper-competitive prices. Miners compete based on their storage, reliability, and speed rather than through marketing or locking users in.

Reliable storage

Because storage is paid for, Filecoin provides a viable economic reason for files to stay available over time. Files are stored on computers that are reliable and well-connected to the internet.

Reputation, not marketing

In Filecoin, storage providers prove their reliability through their track record published on the blockchain, not through marketing claims published by the providers themselves. Users don’t need to rely on status pages or self-reported statistics from storage providers.

Choice of tradeoffs

Users get to choose their own tradeoffs between cost, redundancy, and speed. Users are not limited to a set group of data centers offered by their provider but can choose to store their files on any storage provider participating in Filecoin.

Censorship resistance

Filecoin resists censorship because no central provider can be coerced into deleting files or withholding service. The network is made up of many different computers run by many different people and organizations. Faulty or malicious actors are noticed by the network and removed automatically.

Useful blockchain

In Filecoin, storage providers are rewarded for providing storage, not for performing wasteful computations. Filecoin secures its blockchain using proof of file replication and proof of storage over time. It doesn’t rely on energy-intensive proof-of-work schemes like other blockchains. Miners are incentivized to amass hard drives and put them to use by storing files. Filecoin doesn’t incentivize the hoarding of graphics cards or application-specific integrated circuits for the sole purpose of mining.

Provides storage to other blockchains

Filecoin’s blockchain is designed to store large files, whereas other blockchains can typically only store tiny amounts of data, very expensively. Filecoin can provide storage to other blockchains, allowing them to store large files. In the future, mechanisms will be added to Filecoin, enabling Filecoin’s blockchain to interoperate with transactions on other blockchains.

Content addressing

Files are referred to by the data they contain, not by fragile identifiers such as URLs. Files remain available no matter where they are hosted or who they are hosted by. When a file becomes popular, it can be quickly distributed by swarms of computers instead of relying on a central computer, which can become overloaded by network traffic.

When multiple users store the same file (and choose to make the file public by not encrypting it), everyone who wants to download the file benefits from Filecoin, keeping it available. No matter where a file is downloaded from, users can verify that they have received the correct file and that it is intact.

Content distribution network

Retrieval providers are computers that have good network connections to lots of users who want to download files. By prefetching popular files and distributing them to nearby users, retrieval providers are rewarded for making network traffic flow smoothly and files download quickly.

Single protocol

Applications implementing Filecoin can store their data on any storage provider using the same protocol. There isn’t a different API to implement for each provider. Applications wishing to support several different providers aren’t limited to the lowest-common-denominator set of features supported by all their providers.

No lock-in

Migrating to a different storage provider is made easier because they all offer the same services and APIs. Users aren’t locked into providers because they rely on a particular feature of the provider. Also, files are content-addressed, enabling them to be transferred directly between providers without the user having to download and re-upload the files.

Traditional cloud storage providers lock users by making it cheap to store files but expensive to retrieve them again. Filecoin avoids this by facilitating a retrieval market where providers compete to give users their files back as fast as possible, at the lowest possible price.

Open source code

The code that runs both clients and storage providers is open-source. Storage providers don’t have to develop their own software for managing their infrastructure. Everyone benefits from improvements made to Filecoin’s code.

Active community

The indexer

When a storage deal is originally made, the client can opt to make the data publicly discoverable. If this is the case, the storage provider must publish an advertisement of the storage deal to the Interplanetary Network Indexer (IPNI). IPNI maps a CID to a storage provider (SP). This mapping allows clients to query the IPNI to discover where content is on Filecoin.

The IPNI also tracks which data transfer protocols you can use to retrieve specific CIDs. Currently, Filecoin SPs have the ability to serve retrievals over Graphsync, Bitswap, and HTTP. This is dependent on the SP setup.

Retrieval process

If a client wants to retrieve publicly available data from the Filecoin network, then they generally follow this process.

Query the IPNI

Before the client can submit a retrieval deal to a storage provider, they first need to find which providers hold the data. To do this, the client sends a query to the Interplanetary Network Indexer.

Select a provider

Assuming the IPNI returns more than one storage provider, the client can select which provider they’d like to deal with. Here, they will also get additional details (if needed) based on the retrieval protocol they want to retrieve the content over.

Initiate retrieval

The client then attempts to retrieve the data from the SP over Bitswap, Graphsync, or HTTP. Note that currently, clients can only get full-piece retrievals using HTTP.

When attempting this retrieval deal using Graphsync, payment channels are used to pay FIL to the storage provider. These payment channels watch the data flow and pay the storage provider after each chunk of data is retrieved successfully.

Finalize the retrieval

Once the client has received the last chunk of data, the connection is closed.

Saturn is a Web3 CDN in Filecoin’s retrieval market. On one side of the network, websites buy fast, low-cost content delivery. On the other side, Saturn node operators earn Filecoin by fulfilling requests.

Saturn is trustless, permissionless, and inclusive. Anyone can run Saturn software, contribute to the network, and earn Filecoin.

Content on Saturn is IPFS content-addressed. Every piece of content is immutable, and every response is verifiable.

Incentives unite, align, and grow the network. Node operators earn Filecoin for accelerating web content, and websites get faster content delivery for less.

Was this page helpful?

Libp2p

A modular network stack, libp2p enables you to run your network applications free from runtime and address services, independently of their location. Learn more at libp2p.io/.

IPLD

IPLD is the data model of the content-addressable web. It allows us to treat all hash-linked data structures as subsets of a unified information space, unifying all data models that link data with hashes as instances of IPLD. Learn more at ipld.io/.

IPFS

IPFS is a distributed system for storing and accessing files, websites, applications, and data. However, it does not have support for incentivization or guarantees of this distributed storage; Filecoin provides the incentive layer. Learn more at ipfs.tech/.

Multiformats

The Multiformats Project is a collection of protocols which aim to future-proof systems through self-describing format values that allow for interoperability and protocol agility. Learn more at multiformats.io/.

ProtoSchool

Interactive tutorials on decentralized web protocols, designed to introduce you to decentralized web concepts, protocols, and tools. Complete code challenges right in your web browser and track your progress as you go. Explore ProtoSchool’s tutorials on Filecoin at proto.school/.

Was this page helpful?

Ways to contribute

Code

Filecoin and its sister-projects are big, with lots of code written in multiple languages. We always need help writing and maintaining code, but it can be daunting to just jump in. We use the label Help Wanted on features or bug fixes that people can help out with. They are an excellent place for you to start contributing code.

The biggest and most active repositories we have today are:

• filecoin-project/venus

• filecoin-project/lotus

• filecoin-project/rust-fil-proofs

If you want to start contributing to the core of Filecoin, those repositories are a great place start. But the Help Wanted label exists in several related projects:

• IPFS

• libp2p

• IPLD

• Multiformats

Documentation

Filecoin is a huge project and undertaking, and with lots of code comes the need for lots of good documentation! However, we need a lot more help to write the awesome docs the project needs. If writing technical documentation is your area, any and all help is welcome!

Before contributing to the Filecoin docs, please read these quick guides; they’ll save you time and help keep the docs accurate and consistent!

1. Style and formatting guide

2. Writing guide

If you have never contributed to an open-source project before, or just need a refresher, take a look at the contribution tutorial.

Community

If interacting with people is your favorite thing to do in this world, join the Filecoin chat and discussion forums to say hello, meet others who share your goals, and connect with other members of the community. You should also consider joining Filecoin Slack.

Build Applications

Filecoin is designed for you to integrate into your own applications and services.

Get started by looking at the list of projects currently built on Filecoin. Build anything you think is missing! If you’re unsure about something, you can join the chat and discussion forums to get help or feedback on your specific problem/idea. You can also join a Filecoin Hackathon, apply for a Filecoin Developer Grant or apply to the Filecoin accelerator program to support the development of your project.

• Filecoin Hackathons

• Filecoin Developer Grants

• Filecoin Accelerator Program

Protocol Design

Filecoin is ultimately about building better protocols, and the community always welcome ideas and feedback on how to improve those protocols.

• filecoin-project/specs

Research

Finally, we see Protocol Labs as a research lab, where YOUR ideas can become technologies that have a real impact on the world. If you’re interested in contributing to our research, please reach out to research@protocol.ai for more information. Include what your interests are so we can make sure you get to work on something fun and valuable.

Writing guide

This guide explains things to keep in mind when writing for Filecoin’s documentation. While the grammar, formatting, and style guide lets you know the rules you should follow, this guide will help you to properly structure your writing and choose the correct tone for your audience.

Walkthroughs

The purpose of a walkthrough is to tell the user how to do something. They do not need to convince the reader of something or explain a concept. Walkthroughs are a list of steps the reader must follow to achieve a process or function.

The vast majority of documentation within the Filecoin documentation project falls under the Walkthrough category. Walkthroughs are generally quite short, have a neutral tone, and teach the reader how to achieve a particular process or function. They present the reader with concrete steps on where to go, what to type, and things they should click on. There is little to no conceptual information within walkthroughs.

Goals

Use the following goals when writing walkthroughs:

Function or process

The end goal of a walkthrough is for the reader to achieve a very particular function. Installing the Filecoin Desktop application is an example. Following this walkthrough isn’t going to teach the reader much about working with the decentralized web or what Filecoin is. Still, by the end, they’ll have the Filecoin Desktop application installed on their computer.

Short length

Since walkthroughs cover one particular function or process, they tend to be quite short. The estimated reading time of a walkthrough is somewhere between 2 and 10 minutes. Most of the time, the most critical content in a walkthrough is presented in a numbered list. Images and GIFs can help the reader understand what they should be doing.

If a walkthrough is converted into a video, that video should be no longer than 5 minutes.

Walkthrough structure

Walkthroughs are split into three major sections:

1. What we’re about to do.

2. The steps we need to do.

3. Summary of what we just did, and potential next steps.

Conceptual articles

Articles are written with the intent to inform and explain something. These articles don’t contain any steps or actions that the reader has to perform right now.

These articles are vastly different in tone when compared to walkthroughs. Some topics and concepts can be challenging to understand, so creative writing and interesting diagrams are highly sought-after for these articles. Whatever writers can do to make a subject more understandable, the better.

Article goals

Use the following goals when writing conceptual articles:

Article structure

Articles are separated into five major sections:

1. Introduction to the thing we’re about to explain.

2. What the thing is.

3. Why it’s essential.

4. What other topics it relates to.

5. Summary review of what we just read.

Tutorials

When writing a tutorial, you’re teaching a reader how to achieve a complex end-goal. Tutorials are a mix of walkthroughs and conceptual articles. Most tutorials will span several pages, and contain multiple walkthroughs within them.

Take the hypothetical tutorial Get up and running with Filecoin, for example. This tutorial will likely have the following pages:

1. A brief introduction to what Filecoin is.

2. Choose and install a command line client.

3. Understanding storage deals.

4. Import and store a file.

Pages 1 and 3 are conceptual articles, describing particular design patterns and ideas to the reader. All the other pages are walkthroughs instructing the user how to perform one specific action.

When designing a tutorial, keep in mind the walkthroughs and articles that already exist, and note down any additional content items that would need to be completed before creating the tutorial.

Grammar and formatting

Here are some language-specific rules that the Filecoin documentation follows. If you use a writing service like Grammarly, most of these rules are turned on by default.

American English

While Filecoin is a global project, the fact is that American English is the most commonly used style of English used today. With that in mind, when writing content for the Filecoin project, use American English spelling. The basic rules for converting other styles of English into American English are:

1. Swap the s for a z in words like categorize and pluralize.

2. Remove the u from words like color and honor.

3. Swap tre for ter in words like center.

The Oxford comma

In a list of three or more items, follow each item except the last with a comma ,:

References to Filecoin

As a proper noun, the name “Filecoin” (capitalized) should be used only to refer to the overarching project, to the protocol, or to the project’s canonical network:

Filecoin [the project] has attracted contributors from around the globe! Filecoin [the protocol] rewards contributions of data storage instead of computation! Filecoin [the network] is currently storing 50 PiB of data!

The name can also be used as an adjective:

The Filecoin ecosystem is thriving! I love contributing to Filecoin documentation!

When referring to the token used as Filecoin’s currency, the name FIL, is preferred. It is alternatively denoted by the Unicode symbol for an integral with a double stroke ⨎:

• Unit prefix: 100 FIL.

• Symbol prefix: ⨎ 100.

The smallest and most common denomination of FIL is the attoFIL (10^-18 FIL).

The collateral for this storage deal is 5 FIL. I generated ⨎100 as a storage provider last month!

Examples of discouraged usage:

Filecoin rewards storage providers with Filecoin. There are many ways to participate in the filecoin community. My wallet has thirty filecoins.

Consistency in the usage of these terms helps keep these various concepts distinct.

References to Lotus

Lotus is the main implementation of Filecoin. As such, it is frequently referenced in the Filecoin documentation. When referring to the Lotus implementation, use a capital L. A lowercase l should only be used when referring to the Lotus executable commands such as lotus daemon. Lotus executable commands should always be within code blocks:

1. Start the Lotus daemon:

   ```shell
   lotus daemon
   ```

2. After your Lotus daemon has been running for a few minutes, use `lotus` to check the number of other peers that it is connected to in the Filecoin network:

   ```shell
   lotus net peers
   ```

Acronyms

If you have to use an acronym, spell the full phrase first and include the acronym in parentheses () the first time it is used in each document. Exception: This generally isn’t necessary for commonly-encountered acronyms like IPFS, unless writing for a stand-alone article that may not be presented alongside project documentation.

Virtual Machine (VM), Decentralized Web (DWeb).

Formatting

How the Markdown syntax looks, and code formatting rules to follow.

Syntax

The Filecoin Docs project follows the GitHub Flavoured Markdown syntax for markdown. This way, all articles display properly within GitHub itself.

Rules

We use the rules set out in the VSCode Markdownlint extension. You can import these rules into any text editor like Vim or Sublime. All rules are listed within the Markdownlint repository.

We highly recommend installing VSCode with the Markdownlint extension to help with your writing. The extension shows warnings within your markdown whenever your copy doesn’t conform to a rule.

Style

The following rules explain how we organize and structure our writing. The rules outlined here are in addition to the rules found within the Markdownlinter extension.

Text

The following rules apply to editing and styling text.

Titles

1. All titles follow sentence structure. Only names and places are capitalized, along with the first letter of the title. All other letters are lower-case:

## This is a title

### Only capitalize names and places

### The capital city of France is Paris

1. Every article starts with a front-matter title and description:

---
title: Example article
description: This is a brief description that shows up in link teasers in services like Twitter and Slack.
---

## This is a subtitle

Example body text.

In the above example title: serves as a <h1> or # tag. There is only ever one title of this level in each article.

1. Titles do not contain punctuation. If you have a question within your title, rephrase it as a statement:

<!-- This title is wrong. -->
## What is Filecoin?

<!-- This title is better. -->
## Filecoin explained

Bold text

Double asterisks ** are used to define boldface text. Use bold text when the reader must interact with something displayed as text: buttons, hyperlinks, images with text in them, window names, and icons.

In the **Login** window, enter your email into the **Username** field and click **Sign in**.

Italics

Underscores _ are used to define italic text. Style the names of things in italics, except input fields or buttons:

Here are some American things:

- The _Spirit of St Louis_.
- The _White House_.
- The United States _Declaration of Independence_.

Try entering them into the **American** field and clicking **Accept**.

Quotes or sections of quoted text are styled in italics and surrounded by double quotes ":

In the wise words of Winnie the Pooh _"People say nothing is impossible, but I do nothing every day."_

Code blocks

Tag code blocks with the syntax of the core they are presenting:

    ```javascript
    console.log(error);
    ```

Output from command-line actions can be displayed by adding another codeblock directly after the input codeblock. Here’s an example telling the use to run go version and then the output of that command in a separate codeblock immediately after the first:

    ```shell 
    go version
    ```

    ```plaintext
    go version go1.19.7 darwin/arm64
    ```

Command-line examples can be truncated with three periods ... to remove extraneous information:

    ```shell
    lotus-miner info
    ```

    ```shell
    Miner: t0103
    Sector Size: 16.0 MiB
    ...
    Sectors:  map[Committing:0 Proving:0 Total:0]
    ```

Inline code tags

Surround directories, file names, and version numbers between inline code tags `.

Version `1.2.0` of the program is stored in `~/code/examples`. Open `exporter.exe` to run the program.

List items

All list items follow sentence structure. Only names and places are capitalized, along with the first letter of the list item. All other letters are lowercase:

1. Never leave Nottingham without a sandwich.

2. Brian May played guitar for Queen.

3. Oranges.

List items end with a period ., or a colon : if the list item has a sub-list:

1. Charles Dickens novels:

   1. Oliver Twist.

   2. Nicholas Nickelby.

   3. David Copperfield.

2. J.R.R Tolkien non-fiction books:

   1. The Hobbit.

   2. Silmarillion.

   3. Letters from Father Christmas.

Unordered lists

Use the dash character - for un-numbered list items:

- An apple.
- Three oranges.
- As many lemons as you can carry.
- Half a lime.

Special characters

Whenever possible, spell out the name of the special character, followed by an example of the character itself within a code block.

Use the dollar sign `$` to enter debug-mode.

Keyboard shortcuts

When instructing the reader to use a keyboard shortcut, surround individual keys in code tags:

Press `ctrl` + `c` to copy the highlighted text.

The plus symbol + stays outside of the code tags.

Images

The following rules and guidelines define how to use and store images.

Alt text

All images contain alt text so that screen-reading programs can describe the image to users with limited sight:

![Screenshot of an image being uploaded through the Filecoin command line.](filecoin-image-upload-screen.png)

Was this page helpful?

Filecoin combines many elements of other file storage and distribution systems. What makes Filecoin unique is that it runs on an open, peer-to-peer network while still providing economic incentives and proofs to ensure files are being stored correctly. This page compares Filecoin against other technologies that share some of the same properties.

• Filecoin vs. Amazon S3, Google Cloud Storage

• Filecoin vs. Bitcoin

Filecoin vs. Amazon S3, Google Cloud Storage

Filecoin tokens (FIL) vs. Bitcoin tokens (BTC)

Was this page helpful?

What are some of the primary use cases for Filecoin?

Filecoin is a protocol that provides core primitives, enabling a truly trustless decentralized storage network. These primitives and features include publicly verifiable cryptographic storage proofs, cryptoeconomic mechanisms, and a public blockchain. Filecoin provides these primitives to solve the really hard problem of creating a trustless decentralized storage network.

On top of the core Filecoin protocol, there are a number of layer 2 solutions that enable a broad array of use cases and applications, many of which also use IPFS, such as Lighthouse or Tableland. Using these solutions, any use case that can be built on top of IPFS can also be built on Filecoin!

Some of the primary areas for development on Filecoin are:

• Additional developer tools and layer-2 solutions and libraries that strengthen Filecoin as a developer platform and ecosystem.

• IPFS apps that rely on decentralized storage solutions and want a decentralized data persistence solution as well.

• Financial tools and services on Filecoin, like wallets, signing libraries, and more.

• Applications that use Filecoin’s publicly verifiable cryptographic proofs in order to provide trustless and timestamped guarantees of storage to their users.

How can a website or app be free if it costs to retrieve data from the Filecoin network?

Most websites and apps make money by displaying ads. This type of income-model could be replaced with a Filecoin incentivized retrieval setup, where users pay small amounts of FIL for whatever files they’re hoping to download. Several large datasets are hosted through Amazon’s pay per download S3 buckets, which Filecoin retrieval could also easily augment or replace.

How will Filecoin attract developers to use Filecoin for storage?

It’s going to require a major shift in how we think about the internet. At the same time, it is a very exciting shift, and things are slowly heading that way. Browser vendors like Brave, Opera, and Firefox are investing into decentralized infrastructure.

We think that the internet must return to its decentralized roots to be resilient, robust, and efficient enough for the challenges of the next several decades. Early developers in the Filecoin ecosystem are those who believe in that same vision and potential for the internet, and we’re excited to work with them to build this space.

What are the detailed parameters of Filecoin’s cryptoeconomics?

We are still finalizing our cryptoeconomic parameters, and they will continue to evolve.

Here is a blog about Filecoin economics from December 2020: Filecoin network economics.

How expensive will Filecoin storage be at launch?

As Filecoin is a free market, the price will be determined by a number of variables related to the supply and demand for storage. It’s difficult to predict before launch. However, a few design elements of the network help support inexpensive storage.

Along with revenue from active storage deals, Storage Miners receive block rewards, where the expected value of winning a given block reward is proportional to the amount of storage they have on the network. These block rewards are weighted heavily towards the early days of the network (with the frequency of block rewards exponentially decaying over time). As a result, Storage Miners are relatively incentivized to charge less for storage to win more deals, which would increase their expected block reward.

Further, Filecoin introduces a concept called Verified Clients, where clients can be verified to actually be storing useful data. Storage Miners who store data from Verified Clients also increase their expected block reward. Anyone running a Filecoin-backed IPFS Pinning Services should qualify as a Verified Client. We do not have the process of verification finalized, but we expect it to be similar to submitting a GitHub profile.

Will it be cheaper to store data on Filecoin than other centralized cloud services?

Filecoin creates a hyper-competitive market for data storage. There will be many storage providers offering many prices, rather than one fixed price on the network. We expect Filecoin’s permissionless model and low barriers to entry to result in some very efficient operations and low-priced storage, but it’s impossible to say what exact prices will be until the network is live.

What happens to the existing content on IPFS once Filecoin launches? What if nodes continue to host content for free and undermine the Filecoin incentive layer?

IPFS will continue to exist as it is, enhanced with Filecoin nodes. There are many use cases that require no financial incentive. Think of it like IPFS is HTTP, and Filecoin is a storage cloud-like S3 – only a fraction of IPFS content will be there.

People with unused storage who want to earn monetary rewards should pledge that storage to Filecoin, and clients who want guaranteed storage should store that data with Filecoin storage providers.

Lotus or Venus, which is better for storage providers?

Lotus is the primary reference implementation for the Filecoin protocol. At this stage, we would recommend most storage providers use lotus to participate in the Filecoin network.

What is your recommendation on the right hardware to use?

While the Filecoin team does not recommend a specific hardware configuration, we document various setups here. Additionally, this guide to storage mining details hardware considerations and setups for storage providers. However, it is likely that there are more efficient setups, and we strongly encourage storage providers to test and experiment to find the best combinations.

We are worried about the ability of our network to handle the additional overhead of running a Filecoin node and still provide fast services for our customers. What are the computational demands of a Lotus node? Are there any metrics for node performance given various requirements?

For information on Lotus requirements, see Prerequisites > Minimal requirements.

For information on Lotus full nodes and lite nodes, see Types of nodes.

We bought a lot of hard drives of data through the Discover project. When will they be shipped to China?

There are a number of details that are still being finalized between the verified deals construction and the associated cryptoeconomic parameters.

Our aim is to allow these details to finalize before shipping, but given timelines, we’re considering enabling teams to take receipt of these drives before the parameters are set. We will publish updates on the status of the Discover project on the Filecoin blog.

Do Filecoin storage providers need a fixed IP?

For mainnet, you will need a public IP address, but it doesn’t need to be fixed (just accessible).

What if we lost a sector accidentally, is there any way to fix that?

If you lost the data itself, then no, there’s no way to recover that, and you will be slashed for it. If the data itself is recoverable, though (say you just missed a WindowPoSt), then the Recovery process will let you regain the sector.

Has Filecoin confirmed the use of the SDR algorithm? Is there any evidence of malicious construction?

SDR (Stacked DRG PoRep) is confirmed and used, and we have no evidence of malicious construction. The algorithm is also going through both internal and external security audits.

If you have any information about any potential security problem or malicious construction, reach out to our team at security@filecoin.org.

How likely is it that the Filecoin protocol will switch to the NSE Proof-of-Replication construction later?

Native storage extension (NSE) is one of the best candidates for a proof upgrade, and teams are working on implementation. But there are other candidates too, which are promising as well. It may be that another algorithm ends up better than NSE – we don’t know yet. Proof upgrades will arrive after the mainnet launch and will coexist.

AMD may be optimal hardware for SDR. You can see this description for more information on why.

How are you working on bootstrapping the demand side of the marketplace? The Discover program is nice, but who is the target market for users, and how do you get them?

In addition to Filecoin Discover, a number of groups are actively building tools and services to support the adoption of the Filecoin network with developers and clients. For example, check out the recordings from our Virtual Community Meetup to see updates about Textile and Starling Storage. You can also read more about some of the teams building on Filecoin through HackFS in our HackFS Week 1 Recap.

Does Filecoin have an implementation of client and storage provider order matching through order books?

There will be off-chain order books and storage provider marketplaces – some are in development now from some teams. They will work mostly off-chain because transactions per second on-chain are not enough for the volume of usage we expect on Filecoin. These order books build on the basic deal-flow on-chain. These order books will arrive in their own development trajectory – most likely around or soon after the mainnet launch.

Why does Filecoin mining work best on AMD?

Currently, Filecoin’s Proof of Replication (PoRep) prefers to be run on AMD processors. See this description of Filecoin sealing for more information. More accurately, it runs much slower on Intel CPUs. It runs competitively fast on some ARM processors, like the ones in newer Samsung phones, but they lack the RAM to seal the larger sector sizes. The main reason that we see this benefit on AMD processors is due to their implementation of the SHA hardware instructions.

What do storage providers have to do to change a committed capacity (CC) sector into a “real-data” sector?

Storage providers will publish storage deals that they will upgrade the CC sector with, announce to the chain that they are doing an upgrade, and prove to the chain that a new sector has been sealed correctly. We expect to evolve and make this cheaper and more attractive over time after the mainnet launch.

What does “terminating a sector” mean?

When a committed capacity sector is added to the chain, it can upgrade to a sector with deals, extend its lifetime, or terminate through either faults or voluntary actions. While we don’t expect this to happen very often on mainnet, a storage provider may deem it rational to terminate their promise to the network and their clients, and accept a penalty for doing so.

Does the committed capacity sector still need to be sealed before it upgrades to one with real data?

For the first iteration of the protocol, yes. We have plans to make it cheaper and more economically attractive after mainnet with no resealing required and other perks.

What’s the minimum time period for the storage contract between the provider and the buyer?

The minimum duration for a deal is set in the storage provider’s ask. There’s also a practical limitation because sectors have a minimum duration (currently 180 days).

After I made a deal with a storage provider and sent my data to them, how exactly is the data supposed to be recoverable and healable if that storage provider goes down?

Automatic repair of faulted data is a feature we’ve pushed off until after the mainnet launch. For now, the way to ensure resiliency is to store your data with multiple storage providers, to gain some level of redundancy. If you want to learn more about how we are thinking about repair in the future, here are some notes.

How do I know that my storage provider will not charge prohibitively high costs for data retrieval?

To avoid extortion, always ensure you store your data with a fairly decentralized set of storage providers (and note: it’s pretty difficult for a storage provider to be sure they are the only person storing a particular piece of data, especially if you encrypt the data).

Storage providers currently provide a ‘dumb box’ interface and will serve anyone any data they have. Maybe in the future, storage providers will offer access control lists (ACLs) and logins and such, but that requires that you trust the storage provider. The recommended (and safest) approach here is to encrypt data you don’t want others to see yourself before storing it.

How do you update data stored on Filecoin?

We have some really good ideas around ‘warm’ storage (that is mutable and provable) that we will probably implement in the near future. But for now, your app will have to treat Filecoin as an append-only log. If you want to change your data, you just write new data.

‘Warm’ storage can be done with a small amount of trust, where you make a deal with a storage provider with a start date quite far in the future. The storage provider can choose to store your data in a sector now (but they won’t get paid for proving it until the actual start date), or they can hold it for you (and even send you proofs of it on request), and you can then send them new data to overwrite it, along with a new storage deal that overwrites the previous one.

There’s a pretty large design space here, and we can do a bunch of different things depending on the levels of trust involved, the price sensitivity, and the frequency of updates clients desire.

Who will be selected to be verifiers to verify clients on the network?

Allocators, selected through an application process, serve as fiduciaries for the Filecoin network and are responsible for allocating DataCap to clients with valuable storage use cases.

See Filecoin Plus.

Will the existence of Filecoin mining pools lead to centralized storage and away from the vision of distributed storage?

No – Filecoin creates a decentralized storage network in part by massively decreasing the barrier to entry to becoming a storage provider. Even if there were some large pools, anyone can join the network and provide storage with just a modest hardware purchase, and we expect clients to store their files with many diverse storage providers.

Also, note that world location matters for mining: many clients will prefer storage providers in specific regions of the world, so this enables lots of storage providers to succeed across the world, where there is storage demand.

Even though Filecoin will be backed up to our normal IPFS pinning layer, we still need to know how quickly we can access data from the Filecoin network. How fast will retrieval be from the Filecoin network?

If you are retrieving your data from IPFS or a remote pinning layer, retrieval should take on the order of milliseconds to seconds in the worst case. Our latest tests for retrieval from the Filecoin network directly show that a sealed sector holding data takes ~1 hour to unseal. 1-5 hours is our best real-world estimate to go from sector unsealing to delivery of the data. If you need faster data retrieval for your application, we recommend building on IPFS.

Was this page helpful?

Chat

For shorter-lived discussions, our community chat open to all on both Slack and Matrix, with bridged channels allowing you to participate in the same conversations from either platform:

• Slack

• Matrix

Discussion Forums

For long-lived discussions and for support, please use the discussion tab on GitHub instead of Slack or Matrix. It’s easy for complex discussions to get lost in a sea of new messages on those chat platforms, and posting longer discussions and support requests on the forums helps future visitors, too.

Was this page helpful?

The Filecoin network provides decentralized data storage and makes sure data is verified, always available, and immutable. Storage providers in the Filecoin network are in charge of storing, providing content and issuing new blocks.

To become a storage provider in the Filecoin network you need a range of technical, financial and business skills. We will explain all the key concepts you need to understand in order to design a suitable architecture, make the right hardware investments, and run a profitable storage provider business.

Follow these steps to begin your storage provider journey:

1. Understand Filecoin economics

2. Plan your business

3. Make sure you have the right skills

4. Build the right infrastructure

Understand Filecoin economics

To understand how you can run a profitable business as a Filecoin storage provider, it is important to make sure you understand the economics of Filecoin. Once you understand all core concepts, you can build out a strategy for your desired ROI.

Storage providers can also add additional value to clients when they offer certain certifications. These can enable a storage provider to charge customers additional fees for storing data in compliance with those standards, for example, HIPAA, SOC2, PCI, GDPR and others.

Filecoin economics ->

Plan your business

The hardware and other requirements for running a Filecoin storage provider business are significantly higher than regular blockchain mining operations. The mechanisms are designed this way because, in contrast to some other blockchain solutions, where you can simply configure one or more nodes to “mine” tokens, the Filecoin network’s primary goal is to provide decentralized storage for humanity’s most valuable data.

You need to understand the various earning mechanisms in the Filecoin network.

Filecoin deals ->

Make sure you have the right skills

As will become clear, running a storage operation is a serious business, with client data and pledged funds at stake. You will be required to run a highly-available service, and there are automatic financial penalties if you cannot demonstrate data availability to the network. There are many things that can go wrong in a data center, on your network, on your OS, or at an application level.

You will need skilled people to operate your storage provider business. Depending on the size and complexity of your setup this can be 1 person with skills across many different domains, or multiple dedicated people or teams.

People and skills ->

Build the right infrastructure

At the lowest level, you will need datacenter infrastructure. You need people capable of architecting, racking, wiring and operating infrastructure components. Alternatively, you can get it collocated, or even entirely as a service from a datacenter provider.

Take availability and suitable redundancy into consideration when choosing your datacenter or collocation provider. Any unavailability of your servers, network or storage can result in automatic financial penalties on the Filecoin network.

Software architecture ->

Infrastructure ->

Was this page helpful?

Was this page helpful?

As a storage provider on the network, you will have to create FIL wallets and add FIL to them. This is used to send messages to the blockchain but is also used for collateral. Providing storage capacity to the network requires you to provide FIL as collateral, which goes into a locked wallet on your Lotus instance. The Lotus documentation details the process of setting up your wallets and funding wallets for the initial setup. Filecoin uses upfront token collateral, as in proof-of-stake protocols, proportional to the storage hardware committed. This gets the best of both worlds to protect the network: attacking the network requires both acquiring and running the hardware, but it also requires acquiring large quantities of the token.

Types of collateral

To satisfy the varied collateral needs of storage providers in a minimally burdensome way, Filecoin includes three different collateral mechanisms:

• Initial pledge collateral, an initial commitment of FIL that a miner must provide with each sector.

• Block rewards as collateral, a mechanism to reduce the initial token commitment by vesting block rewards over time.

• Storage deal provider collateral, which aligns incentives between storage provider and client and can allow storage providers to differentiate themselves in the market.

For more detailed information about how collateral requirements are calculated, see the miner collateral section in the Filecoin spec.

When a storage provider fails to answer to the WindowsPoSt challenges within the 30-minute deadline (see Storage Proving), storage is taken offline, or any storage deal rules are broken, the provider is penalized against the provided collateral. This penalty is called slashing and means that a portion of the pledged collateral is forfeited to the f099 address from your locked or available rewards, and your storage power is reduced. The f099 address is the address where all burned FIL goes.

Commit Pledge

The amount of required collateral depends on the amount of storage pledged to the Filecoin network. The bigger volume you store, the more collateral is required. Additionally, Filecoin Plus uses a QAP multiplier to increase the collateral requirement. See Verified Deals with Filecoin Plus for more information.

The formula for the required collateral is as follows:

Collateral needed for X TiB = (Current Sector Initial Pledge) x (32) x (X TiB)

For instance, for 100 TiB at 0.20 FIL / 32 GiB sector, this means:

0.20 FIL x 32 x 100 = 640 FIL

The “Current Sector Initial Pledge" can be found on blockchain explorers like Filfox and on the Starboard dashboards.

Gas fees

Another cost factor in the network is gas. Storage providers not only pledge collateral for the capacity they announce on-chain. The network also burns FIL in the form of gas fees. Most activity on-chain has some level of gas involved. For storage providers, this is the case for committing sectors.

The gas fees fluctuate over time and can be followed on various websites like Filfox - Gas Statistics and Beryx - Gas Estimator.

FIL lending programs

The ecosystem does have FIL Lenders who can provide you FIL (with interest) to get you started, which you can pay back over time and with the help of earned block rewards. Every lender, though, will still require you to supply up to 20% of the required collateral. The Filecoin Virtual Machine, introduced in March 2023, enables the creation of new lending mechanisms via smart contracts.

Was this page helpful?

Storage proving, known as Proof-of-Spacetime (“PoSt”), is the mechanism that the Filecoin blockchain uses to validate that storage providers are continuously providing the storage they claim. Storage providers earn block rewards each time they successfully answer a PoSt challenge.

Proving deadlines

As a storage provider, you must preserve the data for the duration of the deal, which are on-chain agreements between a client and a storage provider. As of March 2023, deals must have a minimum duration of 180 days, and maximum duration of 540 days. The latter value was chosen to balance long deal length with cryptographic security. Storage providers must be able to continuously prove the availability and integrity of the data they are storing. Every storage sector of 32 GiB or 64 GiB gets verified once in each 24 hour period. This period is called a proving period. Every proving period of 24 hours is broken down into a series of 30 minute, non-overlapping deadlines. This means there are 48 deadlines per day. Storage sectors are grouped in a partition, and assigned to a proving deadline. All storage sectors in a given partition will always be verified during the same deadline.

WindowPoSt

The cryptographic challenge for storage proving is called Window Proof-of-Spacetime (WindowPoSt). Storage providers have a deadline of 30 minutes to respond to this WindowPoSt challenge via a message on the blockchain containing a zk-SNARK proof of the verified sector. Failure to submit this proof within the 30 minute deadline, or failure to submit it at all, results in slashing. Slashing means a portion of the collateral will be forfeited to the f099 burn address and the storage power of the storage provider gets reduced. Slashing is a way to penalize storage providers who fail to meet the agreed upon standards of storage.

Was this page helpful?

YouTube

The Filecoin YouTube channel is home to a wealth of information about the Filecoin project — everything from developer demos to recordings of mining community calls — so you can explore playlists and subscribe to ones that interest and inform you.

Blog

Explore the latest news, events and other happenings on the official Filecoin Blog.

Newsletter

Subscribe to the Filecoin newsletter for official project updates sent straight to your inbox.

Twitter

Get your Filecoin news in tweet-sized bites. Follow these accounts for the latest:

• @Filecoin for news and other updates from the Filecoin project

• @ProtoSchool for updates on ProtoSchool workshops and tutorials

WeChat

Follow FilecoinOfficial on WeChat for project updates and announcements in Chinese.

Was this page helpful?

Get ready to dive into the valuable resources of the storage provider documentation. This comprehensive guide offers a wealth of information about the role of storage providers in the Filecoin ecosystem, including insights into the economic aspects. You’ll also gain knowledge about the software architecture, hardware infrastructure, and the necessary skills for success.

To run a successful storage provider business, it’s crucial to understand the concept of Return on Investment (ROI) and the significance of collateral. By planning ahead and considering various factors, such as CAPEX, OPEX, network variables, and collateral requirements, you can make informed decisions that impact your business’s profitability and desired capacity.

One of the truly enriching elements of the Filecoin ecosystem lies in its vibrant community. Meet the community on the Filecoin Slack. Within this dynamic network, you’ll find a treasure trove of individuals who are eager to share their experiences and offer invaluable solutions to the challenges they’ve encountered along the way. Whether it’s navigating the intricacies of storage provider operations or overcoming hurdles on the blockchain, this supportive community stands ready to lend a helping hand. Embrace the spirit of collaboration and tap into this remarkable network.

Get ready to dive into the heart of the Filecoin network with Lotus, the leading reference implementation. As the most widely used software stack for interacting with the blockchain and operating a storage provider setup, Lotus holds the key to unlocking a world of possibilities. Seamlessly navigate the intricacies of this powerful tool and leverage its capabilities to propel your journey forward.

It’s time to roll up your sleeves and embark on a hands-on adventure. With a multitude of options at your disposal, setting up a local devnet environment is the easiest and most exciting way to kickstart your Filecoin journey. Immerse yourself in the captivating world of sealing sectors and witness firsthand how this critical process works. Feel the thrill of experimentation as you delve deeper into the inner workings of this remarkable technology.

Congratulations on taking the next leap in becoming a full-fledged storage provider! Now is the time to determine your starting capacity and architect a tailored solution to accommodate it. Equip yourself with the necessary hardware to kickstart your journey on the mainnet. Test your setup on the calibration testnet to fine-tune your skills and ensure seamless operations. Once you’re ready, brace yourself for the excitement of joining the mainnet.

As you step into the vibrant realm of the mainnet, it’s time to supercharge your storage provider capabilities with Boost. Discover the immense potential of this powerful software designed to help you secure storage deals and offer efficient data retrieval services to data owners. Unleash the full force of Boost and witness the transformative impact it has on your Filecoin journey.

Within the Filecoin network there are many programs and tools designed to enhance your storage provider setup. Uncover the power of these tools as you dive into the documentation, gaining valuable insights and expanding your knowledge. Make the best use of data programs on your path to success.

Was this page helpful?

What are block rewards?

WinningPoSt (short for Winning Proof of SpaceTime) is the cryptographic challenge through which storage providers are rewarded for their contributions to the network. At the beginning of each epoch (1 epoch = 30 seconds), a small number of storage providers are elected by the network to mine new blocks. Each elected storage provider who successfully creates a block is granted Filecoin tokens by means of a block reward. The amount of FIL per block reward varies over time and is listed on various blockchain explorers like Filfox.

The election mechanism of the Filecoin network is based on the “storage power” of the storage providers. A minimum of 10 TiB in storage power is required to be eligible for WinningPoSt, and hence to earn block rewards. The more storage power a storage provider has, the more likely they will be elected to mine a block. This concept becomes incredibly advantageous in the context of Filecoin Plus verified deals.

Filecoin’s storage capacity

The Filecoin network is composed of storage providers who offer storage capacity to the network. This capacity is used to secure the network, as it takes a significant amount of storage to take part in the consensus mechanism. This large capacity makes it impractical for a single party to reach 51% of the network power, since an attacker would need 10 EiB in storage to control the network. Therefore, it is important that the raw capacity also referred to as raw byte power, remains high. The Filecoin spec also included a baseline power above which the network yields maximum returns for the storage providers.

The graph below shows the evolution of network capacity on the Filecoin network. As can be seen, the baseline power goes up over time (and becomes exponential). This means from May 2021 to February 2023 the network yielded maximum returns for storage providers. However, in recent history, Quality Adjusted Power (QAP) has taken over as a leading indicator of relevance for the Filecoin network. QAP is the result of the multiplier when storing verified deals:

Check out the Starboard dashboard for the most up-to-date Network Storage Capacity.

Impact of storage capacity on block rewards

As mentioned before, when the Raw Byte Power is above the Baseline Power, storage providers yield maximum returns. When building a business plan as a storage provider, it is important not to rely solely on block rewards. Block rewards are an incentive mechanism for storage providers. However, they are volatile and depend on the state of the network, which is largely beyond the control of storage providers.

The amount of FIL that is flowing to the storage provider per earned block reward is based on a combination of simple minting and baseline minting. Simple minting is the minimum amount of FIL any block will always have, which is 5.5. Baseline minting is the extra FIL on top of the 5.5 that comes from how close the Raw Byte Power is to the Baseline Power.

The below graph shows the evolution of FIL per block reward over time:

There is a positive side to releasing less FIL per block reward too. As Filecoin has a capped maximum token supply of 2 billion FIL, the slower minting rate allows for minting over a longer period. A lower circulating supply also has a positive effect on the price of FIL.

See the Crypto Economics page of this documentation and the Filecoin spec for more information.

Was this page helpful?

The real purpose of Filecoin is to store humanity’s most important information. As a storage provider, that means accepting storage deals and storing deal sectors with real data in it. As before, those sectors are either 32 GiB or 64 GiB in size and require that the data be prepared as a content archive; that is, as a CAR file..

Data preparation

Data preparation, which includes packaging files into size appropriate CAR files, is either done by a separate Data Preparer actor, or by storage providers acting as Data Preparers. The latter option is common for new storage providers, as they normally only have a few files that need preparation.

Data preparation can be done in various ways, depending on your use-case. Here are some valuable sources of information:

See the following video for a demonstration on Singularity:

Deal Market

The storage provider can (and should) keep unsealed data copies available for retrieval requests from the client. It is the same software component, Boost, that is responsible for HTTP retrievals from the client and for setting the price for retrievals.

Availability requirements

What’s next?

Storage fault slashing

This term encompasses a broad set of penalties which are to be paid by storage providers if they fail to provide sector reliability or decide to voluntarily exit the network. These include:

• Fault fees are incurred for each day a storage provider’s sector is offline (fails to submit Proofs-of-Spacetime to the chain). Fault fees continue until the associated wallet is empty and the storage provider is removed from the network. In the case of a faulted sector, there will be an additional sector penalty added immediately following the fault fee.

• Sector penalties are incurred for a faulted sector that was not declared faulted before a WindowPoSt check occurs. The sector will pay a fault fee after a Sector Penalty once the fault is detected.

• Termination fees are incurred when a sector is voluntarily or involuntarily terminated and is removed from the network.

Consensus fault slashing

This penalty is incurred when committing consensus faults. This penalty is applied to storage providers that have acted maliciously against the network’s consensus functionality.

Web3.storage runs on “Elastic IPFS” as the inbound storage protocol offering scalability, performance and reliability as the platform grows. It guarantees the user (typically developers) that the platform will always serve your data when you need it. In the backend the data is uploaded onto the Filecoin Network for long-term storage.

Spade automates the process of renewing storage deals on the Filecoin network, ensuring the longevity of data stored on the blockchain. This is particularly useful for datasets that need to be preserved for extended periods, far beyond the standard deal duration. By using Spade, organizations and individuals can manage and maintain their data storage deals more efficiently, guaranteeing that valuable data remains accessible and secure over time.

Partner tools and programs

Many other programs and tools exist in the Filecoin community, developed by partners or storage providers. We list some examples below.

Swan is a provider of cross-chain cloud computing solutions. Developers can use its suite of tools to access resources across multiple chains.

Swan Cloud provides decentralized cloud computing solutions for Web3 projects by integrating storage, computing, and payment into one suite.

Open Panda was a platform for data researchers, analysts, students, and enthusiasts to interact with the largest open datasets in the world. Data available through the platform was stored on Filecoin, a decentralized storage network comprised of thousands of independent Storage Providers around the world.

Former programs and tools

Here is a comprehensive list of deprecated tools and projects.

Evergreen

CO2.Storage

CO2.Storage was a decentralized storage solution for structured data based on content-addressed data schemas. CO2.Storage primarily focused on structured data for environmental assets, such as Renewable Energy Credits, Carbon Offsets, and geospatial datasets, and mapped inputs to base data schemas (IPLD DAGs) for off-chain data (like metadata, images, attestation documents, and other assets) to promote the development of standard data schemas for environmental assets. This project was in alpha, and while many features could be considered stable, it was waiting until being feature complete to fully launch. The Filecoin Green team was actively working on this project and welcomed contributions from the community.

Filecoin Tracker

Filecoin Tracker was deprecated on April 20, 2024.

Here are great existing and working Filecoin dashboards that cover similar topics:

Slingshot

Dataprograms.org

dataprograms.org listed tools, products, and incentive programs designed to drive growth and make data storage on Filecoin more accessible. It was discontinued in April 2024.

Moonlanding

Moon Landing was designed to ramp up storage providers in the Filecoin network by enabling them to serve verified deals at scale.

Filecoin Dataset Explorer

Filecoin Dataset Explorer showcased data stored on the Filecoin network between 2020 and 2022, including telemetry, historical archives, Creative Commons media, entertainment archives, scientific research, and machine learning datasets. It highlighted Filecoin's capability to store large datasets redundantly, ensuring availability from multiple Storage Providers worldwide. Each dataset is identified by a unique content identifier (CID). The platform aimed to make diverse datasets accessible to users globally.

See also: Legacy Explorer (legacy.datasets.filecoin.io)

Estuary

Estuary was an experimental software platform designed for sending public data to the Filecoin network, facilitating data retrieval from anywhere. It integrated IPFS and Filecoin technologies to provide a seamless end-to-end example for data storage and retrieval. When a file was uploaded, Estuary immediately made multiple storage deals with different providers to ensure redundancy and security. The software automated many aspects of deal making and retrieval, offering tools for managing connections, block storage, and deal tracking. Estuary aimed to simplify the use of decentralized storage networks for developers and users.

Estuary was discontinued in July 2023, and the website shut down in April 2024.

Big Data Exchange

Big Data Exchange was a program that allowed storage providers easy access to Filecoin+ deals through an auction where Storage Providers could bid on datasets by offering to pay clients FIL to choose the bidder as their Storage Provider.

Instead of destroying a previously sealed sector and recreating a new sector that needs to be sealed, Snap Deals allow data to be ingested into CC-sectors without the requirement of re-sealing the sector.

Why would you do snap deals?

There are two main reasons why a storage provider could be doing Snap Deals, also known as “snapping up their sectors” in the Filecoin community:

• The first reason is that the 10x storage power on the same volume of data stored is a strong incentive to upgrade to verified deals for those storage providers who started out on CC-sectors and wish to upgrade to verified deals with Filecoin Plus.

• The second reason applies to storage providers who decide to start sealing CC-sectors, but later then fill them with verified deals. When you start as a storage provider or when you expand your storage capacity, it might be a good idea to fill your capacity with CC-sectors in the absence of verified deals. Not only do you start earning block rewards over that capacity, but more importantly, you can plan the sealing throughput, and balance your load over the available hardware. If your sealing rate is 3 TiB/day, it makes no sense to feed 5 TiB/day into the pipeline. This creates congestion and possibly negative performance. If you are sealing 3 TiB/day for 33 days in a row, you end up with 99 TiB of sealed sectors that were sealed evenly and consistently. If you then take on a 99 TiB verified deal (accounting for 1 PiB QAP), the only thing required is to snap up the sectors.

Snapping up sectors with snap deals puts a lot less stress on the storage provider’s infrastructure. The only task that is executed from the sealing pipeline is the replica-update and prove-replica-update phase, which is similar to the PC2 process. The CPU-intensive PreCommit 1 phase is not required in this process.

Do not forget to provide the collateral funds when snapping up a verified deal. The same volume requires more collateral when it counts as Filecoin Plus data, namely 10x the collateral compared to raw storage power.

Was this page helpful?

Saturn

One of the additional services is participation in Saturn retrieval markets. Saturn is a Web3 CDN (“content delivery network”), and will launch in stages in 2023. Saturn aims to be the biggest Web3 CDN, and biggest CDN overall with the introduction of Saturn, data stored on Filecoin is no longer limited to archive or cold storage, but can also be cached into a CDN layer for fast retrieval. Data that needs to be available quickly can then be stored on Filecoin and retrieved through Saturn. Saturn comes with 2 layers of caching, L1 and L2. L1 nodes typically run in data centers, require high availability and 10 GBs minimum connectivity. The L1 Saturn provider earns FIL through caching and serving data to clients. L2 nodes can be run via an app on desktop hardware.

FVM

Other new opportunities are emerging since the launch of FVM (Filecoin Virtual Machine) in March 2023. The FVM allows smart contracts to be executed on the Filecoin blockchain. The FVM is Ethereum-compatible (also called the FEVM) and allows for entire new use cases to be developed in the Filecoin ecosystem. Think of on-chain FIL lending as an example, but the opportunities are countless.

Bacalhau

A next step after the introduction of FVM is Bacalhau), which will be offering Compute over Data (COD). After the introduction of a compute layer on Filecoin, Bacalhau’s COD promises to run compute jobs over the data where the data resides, at the storage provider. Today, data scientists have to transfer their datasets to compute farms in order for their AI, ML or other data processing activities to run. Bacalhau will allow them to run compute activities on the data where the data is located, thereby removing the expensive requirement to move data around. Storage providers will be able to offer - and get rewarded for - providing compute power to data scientists and other parties who want to execute COD.

Storage tiering

Another potential service to offer is storage tiers with various performance profiles. For example, storage providers can offer hot/online storage by keeping an additional copy of the unsealed data available for immediate retrieval, as well as the sealed that has been stored on the Filecoin Network.

Was this page helpful?

Was this page helpful?

Charging for data stored on your storage provider network is an essential aspect of running a sustainable business. While block rewards from the network can provide a source of income, they are highly dependent on the volatility of the price of FIL, and cannot be relied on as the sole revenue stream.

To build a successful business, it is crucial to develop a pricing strategy that is competitive, yet profitable. This will help you attract and retain customers, as well as ensure that your business succeeds in the long term. While some programs may require storage providers to accept deals for free, or bid in auctions to get a deal, it is generally advisable to charge customers for most client deals.

When developing your pricing strategy, it is important to consider the cost of sales associated with acquiring new customers. This cost consideration should include expenses related to business development, marketing, and sales, which you should incorporate into your business’ return-on-investment (ROI) calculation.

In addition to sales costs, other factors contribute to your business’ total cost of ownership. These include expenses related to backups of your setup and data, providing an access layer to ingest data and for retrievals, preparing the data when necessary, and more. Investigating these costs is essential to ensure your pricing is competitive, yet profitable.

By charging for data stored on your network, you can create a sustainable business model that allows you to invest in hardware and FIL as collateral, as well as grow your business over time. This requires skilled people capable of running a business at scale and interacting with investors, venture capitalists, and banks to secure the necessary funding for growth.

Next to the sales cost, there are other things that contribute to the total cost of ownership of your storage provider business. Think of backups of your setup and the data, providing an access layer to ingest data and for retrievals, preparing the data (if not done already), and more.

Was this page helpful?

Cost

When setting up their business, storage providers must determine how fast they should seal and, thus, how much sealing hardware they should buy. In other words, the cost is an important factor in determining a storage provider’s sealing rate. For example, suppose you have an initial storage capacity of 100 TiB, which would account for 1 PiB QAP if all the sectors contain Filecoin Plus verified deals. If your sealing capacity is 2.5 TiB per day, you will seal your full 100 TiB in 40 days. Is it worth investing in double the sealing capacity to fill your storage in just 20 days? It might be if you are planning to grow way beyond 100 TiB. This is an example of the sort of cost considerations storage providers must factor in when tuning the sealing rate.

Customer expectations

A common reason that a storage provider may want or need a faster sealing rate is customer expectations. When you take on a customer deal, there are often requirements to seal a dataset of a certain size within a certain time window. If you are a new storage provider with 2.5 TiB per day in sealing capacity, you cannot take on a deal of 2 PiB that needs to be on-chain in 1 month; at the very least, you could not take the deal using your own sealing infrastructure. Instead, you can use a Sealing-as-a-service provider, which can help you scale your sealing capabilities.

Balancing the sealing pipeline

When designing their sealing pipeline, storage providers should consider bottlenecks, the grouping of similar tasks, and scaling out.

Bottlenecks

The art of building a well-balanced sealing pipeline means having the bottlenecks where you expect them to be; any non-trivial piece of infrastructure always contains some kind of bottleneck. Ideally, you should design your systems so that the PC1 process is the bottleneck. By doing this, all other components are matched to the capacity required to perform PC1. With PC1 being the most resource-intensive task in the pipeline, it makes the most sense to architect a solution around this bottleneck. Knowing exactly how much sealing capacity you can get from your PC1 servers is vital so you can match the rest of your infrastructure to this throughput.

Assuming you obtain maximum hardware utilization from your PC1 server to seal 15 sectors in parallel, and PC1 takes 3 hours on your infrastructure, that would mean a sealing rate of 3.75 TiB per day. The calculation is described below:

15 sectors x 32 GiB / 3 hours PC1 runtime x 24 hours / 1024 = 3.75 TiB /day

Grouping similar tasks

While a Lotus worker can run all of the various tasks in the sealing pipeline, different storage provider configurations may split tasks between workers. Because some tasks are similar in behavior and others are insignificant in terms of resource consumption, it makes sense to group like-tasks together on the same worker.

A common grouping is AddPiece (AP) and PreCommit1 (PC1) because AP essentially prepares the data for the PC1 task. If you have dedicated hardware for PreCommit2 (PC2), your scratch content will move to that other server. If you are grouping PC1 and PC2 on the same server, you won’t have the sealing scratch copied, but you will need a larger NVMe volume. Eventually, you may run out of sealing scratch space and not be able to start sealing additional sectors. Consider a very large bandwidth (40Gb or even 100Gb) between the servers that copy over the sealing space.

As PC1 is CPU-bound and PC2 is GPU-bound, this is another good reason to separate those tasks into dedicated hardware, especially if you are planning to scale. Because PC2 is GPU-bound, it makes sense to have PC2, C1, and C2 collocated on the same worker.

Another rule of thumb is to have two PC2 workers for every PC1 worker in your setup. The WaitSeed phase occurs after PC2, which locks the scratch space for a sector until C1 and C2. In order to keep sealing sectors in PC1, PC2 must have sufficient capacity. You can easily host multiple PC2 workers on a single server though, ideally with separate GPU's.

Scaling out

A storage provider’s sealing capacity scales linearly with the hardware you add to it. For example, if your current setup allows for a sealing rate of 3 TiB per day, doubling the number of workers could bring you to 6 TiB per day. This requires that all components of your infrastructure are able to handle this additional throughput. Using Sealing-as-a-Service providers allows you to scale your sealing capacity without adding more hardware.

Was this page helpful?

Why this automation?

It can be rather overwhelming for new storage providers to learn everything about Filecoin and the various software components. In order to help with the learning process, we provide a fully automated installation of the Lotus and Boost stack. This automation should allow anyone to go on mainnet or the Calibration testnet in no time.

What are we automating?

This automation is still evolving and will receive more features and capabilities over time. In its current state, it lets you:

• Install and configure Lotus Daemon to interact with the blockchain.

• Initialize and configure Lotus Miner to join the network as a storage provider.

• Install and configure Boost to accept storage deals from clients.

• Install and configure Booster-HTTP to provide HTTP-based retrievals to clients.

Sealing configuration

The initial use case of this automation is to use sealing-as-a-service instead of doing your own sealing. As such, there is no Lotus Worker configured for the setup. It is possible to extend the setup with a remote worker. However, this Lotus Worker will require dedicated and custom hardware.

Composable deployment

One of the next features coming to this automation is a composable deployment method. Today Lotus Daemon, Lotus Miner, and Boost are all installed on a single machine. Many production setups, however, will split out those services into their own dedicated hardware. A composable deployment will allow you to deploy singular components on separate servers.

Prerequisites

Read the README carefully on the GitHub repo to make sure you have all the required prerequisites in place.

Was this page helpful?

Was this page helpful?

Calculating the Return-on-Investment (ROI) of your storage provider business is essential to determine the profitability and sustainability of your operations. The ROI indicates the return or profit on your investment relative to the cost of that investment. There are several factors to consider when calculating the ROI of a storage provider business.

First, the cost of the initial hardware investment and the collateral in FIL required to participate in the network must be considered. These costs are significant and will likely require financing from investors, venture capitalists, or banks.

Second, the income generated from the block rewards must be factored into the ROI calculation. However, this income is subject to the volatility of the FIL token price, which can be highly unpredictable.

Third, it is important to consider the cost of sales when calculating the ROI. Sales costs include the cost of acquiring new customers, marketing, and any fees associated with payment processing. These costs can vary depending on the sales strategy and the size of the business.

Fourth, the total cost of ownership must be considered. This includes the cost of backups, providing access to ingest and retrieve data, preparing the data, and any other costs associated with operating a storage provider business.

Finally, the forecasted growth of the network and the demand for storage will also impact the ROI calculation. If the network and demand for storage grow rapidly, the ROI may increase. However, if the growth is slower than anticipated, the ROI may decrease.

Overall, calculating the ROI of a storage provider business is complex and requires a thorough understanding of the costs and income streams involved. The storage provider Forecast Calculator can assist in determining the ROI by accounting for various factors such as hardware costs, token price, and expected growth of the network.

Calculating the ROI of your storage provider business is important. Check out the Storage Provider Forecast Calculator for more details.

For more information and context see the following video:

It takes more variables than the cost vs. the income. In summary, the factors that influence your ROI are:

• Verified Deals:

  How much of your total sealed capacity will be done with Verified Deals (Filecoin Plus)? Those deals give a far higher return because of the 10x multiplier that is added to your storage power and block rewards.

• Committed Capacity:

  How much of your total sealed capacity will be just committed capacity (CC) sectors (sometimes also called pledged capacity)? These deals give a lower return compared to verified deals but are an easy way to get started in the network. Relying solely on this to generate income is challenging though, especially when the price of FIL is low.

• Sealing Capacity:

  How fast can you seal sectors? Faster sealing means you can start earning block rewards earlier and add more data faster. The downside is that it requires a lot of hardware.

• Deal Duration:

  How long do you plan to run your storage provider? Are you taking short-term deals only, or are you in it for the long run? Taking long-term deals comes with an associated risk: if you can’t keep your storage provider online for the duration of the deals, you will get penalized. Short-term deals that require extension have the downside of higher operational costs to extend (which requires that the data be re-sealed.).

• FIL Collateral pledged:

  A substantial amount of FIL is needed to start accepting deals in the Filecoin network. Verified deals require more pledged collateral than CC-deals. Although the collateral is not lost if you run your storage provider business well, it does mean an upfront investment (or lending).

• Hardware Investment:

  Sealing, storing, and proving the data does require a significant hardware investment as a storage provider. Although relying on services like sealing-as-a-service can lower these requirements for you, it is still an investment in high-end hardware. Take the time to understand your requirements and your future plans so that you can invest in hardware that will support your business.

• Operational Costs:

  Last but not least there’s the ongoing monthly cost of operating the storage provider business. Both the costs for technical operations as well as business operations need to be taken into consideration.

Was this page helpful?

The diagram below shows the major components of Lotus:

The following components are the most important to understand:

• Lotus daemon

• Lotus miner

• Lotus worker

• Boost

Click here for a compatibility matrix of the different components and the required Golang version.

Lotus daemon

The daemon is a key Lotus component that does the following:

• Syncs the chain

• Holds the wallets of the storage provider

The machine running the Lotus daemon must be connected to the public internet for the storage provider to function. See the Lotus documentation for more in-depth information on connectivity requirements.

Syncing the chain

Syncing the chain is a key role of the daemon. It communicates with the other nodes on the network by sending messages, which are, in turn, collected into blocks. These blocks are then collected into tipsets. Your Lotus daemon receives the messages on-chain, enabling you to maintain consensus about the state of the Filecoin network with all the other participants.

Due to the growth in the size of the chain since its genesis, it is not advised for storage providers to sync the entire history of the network. Instead, providers should use the available lightweight snapshots to import the most recent messages. One exception in which a provider would need to sync the entire chain would be to run a blockchain explorer.

Synced chain data should be stored on an SSD; however, faster NVMe drives are strongly recommended. A slow chain sync can lead to delays in critical messages being sent on-chain from your Lotus miner, resulting in the faulting of sectors and the slashing of collateral.

Another important consideration is the size of the file system and available free space. Because the Filecoin chain grows as much as 50GB a day, any available space will eventually fill up. It is up to storage providers to manage the size of the chain on disk and prune it as needed. Solutions like SplitStore (enabled by default) and compacting reduce the storage space used by the chain. Compacting involves replacing the built-up chain data with a recent lightweight snapshot.

Holding wallets

Another key role of the Lotus daemon is to host the Filecoin wallets that are required to run a storage provider (SP). As an SP, you will need a minimum of 2 wallets: an owner wallet and a worker wallet. A third wallet called the control wallet) is required to scale your operations in a production environment.

To keep wallets safe, providers should consider physical access, network access, software security, and secure backups. As with any cryptocurrency wallet, access to the private key means access to your funds. Lotus supports Ledger hardware wallets, the use of which is recommended, or remote wallets with lotus-wallet on a remote machine (see remote lotus wallet for instructions). The worker and control wallets can not be kept on a hardware device because Lotus requires frequent access to those types of wallets. For instance, Lotus may require access to a worker or control wallet to send WindowPoSt proofs on-chain.

Control wallets

Control wallets are required to scale your operations in a production environment. In production, only using the general worker wallet increases the risk of message congestion, which can result in delayed message delivery on-chain and potential sector faulting, slashing, or lost block rewards. It is recommended that providers create wallets for each subprocess. There are five different types of control wallets a storage provider can create:

• PoSt wallet

• PreCommit wallet

• Commit wallet

• Publish storage deals wallet

• Terminate wallet

The lotus-miner also gets an address to which funds can/should be sent. This address can be used to pay any fees and collateral. Withdrawal from this address is only possible with the owner wallet private key.

Lotus miner

The Lotus miner, often referred to using the daemon naming syntax lotus-miner, is the process that coordinates most of the storage provider activities. It has 3 main responsibilities:

• Storing sectors and data

• Scheduling jobs

• Proving the stored data

Storing sectors and data

Storage Providers on the Filecoin network store sectors. There are two types of sectors that a provider may store:

• Sealed sectors: these sectors may or may not actually contain data, but they provide capacity to the network, for which the provider is rewarded.

• Unsealed sectors: used when storing data deals, as retrievals happen from unsealed sectors.

Originally, lotus-miner was the component with storage access. This resulted in lotus-miner hardware using internal disks, directly attached storage shelves like JBODs, Network-Attached-Storage (NAS), or a storage cluster. However, this design introduced a bottleneck on the Lotus miner.

More recently, Lotus has added a more scalable storage access solution in which workers can also be assigned storage access. This removes the bottleneck from the Lotus miner. Low-latency storage access is critical because of the impact on storage-proving processes.

Keeping a backup of your sealed sectors, the cache directory, and any unsealed sectors is crucial. Additionally, you should keep a backup of the sectorstore.json file that lives under your storage path. The sectorestore.json file is required to restore your system in the event of a failure. You can read more about the sectorstore.json file in the lotus docs.

It is also imperative to have at least a daily backup of your lotus-miner state. Backups can be made with:

lotus-miner backup 

The sectorstore.json file, which lives under your storage path, is also required for restoration in the event of a failure. You can read more about the file in the Lotus docs.

Scheduling jobs

Another key responsibility of the Lotus Miner is the scheduling of jobs for the sealing pipeline and storage proving.

Storage proving

One of the most important roles of lotus-miner is the Storage proving. Both WindowPoSt and WinningPoSt processes are usually handled by the lotus-miner process. For scalability and reliability purposes it is now also possible to run these proving processes on dedicated servers (proving workers) instead of using the Lotus miner.

The proving processes require low-latency access to sealed sectors. The proving challenge requires a GPU to run on. The resulting zkProof will be sent to the chain in a message. Messages must arrive within 30 minutes for WindowPoSt, and 30 seconds for WinningPoSt. It is extremely important that providers properly size and configure the proving workers, whether they are using just the Lotus miner or separate workers. Additionally, dedicated wallets, described in Control wallets, should be set up for these processes.

Always check if there are upcoming proving deadlines before halting any services for maintenance. For detailed instructions, refer to the Lotus maintenance guide.

Lotus worker

The Lotus worker is another important component of the Lotus architecture. There can be - and most likely will be - multiple workers in a single storage provider setup. Assigning specific roles to each worker enables higher throughput, sealing rate, and improved redundancy.

As mentioned above, proving tasks can be assigned to dedicated workers, and workers can also get storage access. The remaining worker tasks encompass running a sealing pipeline, which is discussed in the next section.

Boost

Boost is the market component for storage providers to interact with clients. Boost is made of several components (such as boostd, boostd-data, yugabytedb, booster-http etc.). It works as a deal-taking engine (from deals made by clients or other tools), and serves data retrievals to clients who request a copy of the data over graphsync, bitswap or http.

Boost has become a critical component in the software stack of a storage provider and it is therefore necessary to read the Boost documentation carefully.

Boost requires YugabyteDB as of version 2.0. Plan your deployment so that you understand the concepts of Yugabyte well enough. See the Boost documentation for more details.

Helpful commands

The following commands can help storage providers with their setup.

Backup Lotus miner state

It is imperative to have at least one daily backup of your Lotus miner state. Backups can be made using the following command:

lotus-miner backup

View wallets and funds

You can use the following command to view wallets and their funds:

lotus wallet list

Check storage configuration

Run the following command to check the storage configuration for your Lotus miner instance:

lotus-miner storage list

This command return information on your sealed space and your scratch space, otherwise known as a cache. These spaces are only available if you have properly configured your Lotus miner by following the steps described in the Lotus documentation.

In some cases it might be useful to check if the system has access to the storage paths to a certain sector. To check the storage paths to sector 1 for instance, use:

lotus-miner storage find 1

View scheduled jobs

To view the scheduled sealing jobs, run the following:

lotus-miner sealing jobs

View available workers

To see the workers on which the miner can schedule jobs, run:

lotus-miner sealing workers

View proving deadlines

To check if there are upcoming proving deadlines, run the following:

lotus-miner proving deadlines

Was this page helpful?

A network indexer, also referred to as an indexer node or indexer, is a node that maps content identifiers (CIDs) to records of who has the data and how to retrieve that data. These records are called provider data records. Indexers are built to scale in environments with massive amounts of data, like the Filecoin network, and are also used by the IPFS network to locate data. Because the Filecoin network stores so much data, clients can’t perform efficient retrieval without proper indexing. Indexer nodes work like a specialized key-value store for efficient retrieval of content-addressed data.

There are two groups of users within the network indexer process:

• Storage providers advertise their available content by storing data in the indexer. This process is handled by the indexer’s ingest logic.

• Retrieval clients query the indexer to determine which storage providers have the content and what protocol to use, such as Graphsync, Bitswap, etc. This process is handled by the indexer’s find logic.

How the indexer works

This diagram summarizes the different actors in the indexer ecosystem and how they interact with each other. In this context, these actors are not the same as smart-contract actors.

IPNI and storage providers

Storage providers publish data to indexers so that clients can find that data using the CID or multihash of the content. When a client queries the indexer using a CID or multihash, the indexer then responds to the client with the provider data record, which tells the client where and how the content can be retrieved.

As a storage provider, you will need to run an indexer in your setup so that your clients know where and how to retrieve data. For more information on how to create an index provider, see the IPNI documentation.

Was this page helpful?

Storage providers with hardware cost or availability constraints can use Sealing-as-a-service, in which another provider performs sector sealing on the storage providers behalf. This page describes how sealing-as-a-service works, and the benefits to storage providers.

Overview

In a traditional setup, a storage provider needs high-end hardware to build out a sealing pipeline. Storage providers with hardware cost or availability constraints can use Sealing-as-a-Service providers, where another provider performs sector sealing on the storage provider’s behalf. In this model, the following occurs:

1. The storage provider provides the data to the sealer

2. The sealer seals the data into sectors.

3. The sealer returns the sealed sectors in exchange for a service cost.

Benefits

Sealing-as-a-service provides multiple benefits for storage providers:

• Available storage can be filled faster, thereby maximizing block rewards, without investing in a complex, expensive sealing pipeline.

• Bigger deals can be onboarded, as Sealing-as-a-Service essentially offers a burst capability in your sealing capacity. Thus, storage providers can take on larger deals without worrying about sealing time and not meeting client expectations.

• Storage capacity on the Filecoin network can be expanded without investing in a larger sealing pipeline.

Other solutions are possible where the sealing partner seals committed capacity (CC) sectors for you, which you in turn snap up to data sectors.

See the following video from Aligned about their offering of Sealing-as-a-Service:

Was this page helpful?

Each step in the sealing process has different performance considerations, and fine-tuning is required to align the different steps optimally. For example, storage providers that don’t understand the process expected throughput may end up overloading the sealing pipeline by trying to seal too many sectors at once or taking on a dataset that is too large for available infrastructure. This can lead to a slower sealing rate, which is discussed in greater detail in Sealing Rate.

Overview

The sealing pipeline can be broken into the following steps:

AddPiece

The sealing pipeline begins with AddPiece (AP), where the pipeline takes a Piece and prepares it into the sealing scratch space for the PreCommit 1 task (PC1) to take over. In Filecoin, a Piece is data in CAR-file format produced by an IPLD DAG with a corresponding PayloadCID and PieceCID. The maximum Piece size is equal to the sector size, which is either 32 GiB or 64 GiB. If the content is larger than the sector size, it must be split into more than one PieceCID during data preparation.

The AddPiece process is only uses some CPU cores; it doesn’t require the use of a GPU. It does write a lot of data on the sealing volume though. Therefore it is recommended to limit the concurrent AP processes to 1 or 2 via the environment variable AP_32G_MAX_CONCURRENT=1.

It is typically co-located on a server with other worker processes from the sealing pipeline. As PC1 is the next process in the sealing pipeline, running AddPiece on the same server as the PC1 process is a logical architecture configuration.

Consider limiting the AP process to a few cores by using the taskset command, where <xx-xx> is the range on which cores the process needs to run on:

taskset -c <xx-xx> lotus-worker run ...

PreCommit 1

PreCommit 1 (PC1) is the most CPU intensive process of the entire sealing pipeline. PC1 is the step in which a sector, regardless of whether it contains data or not, is cryptographically secured. The worker process loads cryptographic parameters from a cache location, which should be stored on enterprise NVMe for latency reduction. These parameters are then used to run Proof-of-Replication (PoRep) SDR encoding against the sector that was put into the sealing scratch space. This task is single-threaded and very CPU intensive, so it requires a CPU with SHA256 extensions. Typical CPUs that meet this requirement include the AMD Epyc Milan/Rome or an Intel Xeon Ice Lake with 32 cores or more.

Using the scratch space, the PC1 task will create 11 layers of the sector. Storage providers must host scratch space for this on enterprise NVMe. This means that:

• Every sector consumes memory equal to 1+11 times its size on the scratch volume.

• For a 32 GiB sector, PC1 requires 384 GiB on the scratch volume

• For a 64 GiB sector, PC1 requires 768 GiB.

In order to seal at a decent rate and to make use of all the sealing capacity in a PC1 server, you will maximize the concurrent PC1 jobs on a system. Set the PC1_32G_MAX_CONCURRENT= environment variable for the PC1 worker. You can learn more about this in the chapter on Sealing Rate. Sealing several sectors multiplies the requirements on CPU cores, RAM, and scratch space by the number of sectors being sealed in parallel.

The process of sealing a single 32 GiB sector takes roughly 3 hours but that time depends largely on your hardware and what other jobs are running on that hardware.

PreCommit 2

When PC1 has completed on a given sector, the entire scratch space for that sector is moved over to the PreCommit 2 (PC2) task. This task is typically executed on a different server than the PC1 server because it behaves differently. In short, PC2 validates PC1 using the Poseidon hashing algorithm over the Merkle Tree DAG that was created in PC1. As mentioned in the previous section, the entire scratch space is either 384 GiB or 768 GiB, depending on the sector size.

Where PC1 is CPU-intensive, PC2 is executed on GPU. This task is also notably shorter in duration than PC1, typically 10 to 20 minutes on a capable GPU. This requires a GPU with at least 10 GiB of memory and 3500+ CUDA cores or shading units, in the case of Nvidia. Storage providers can use slower GPUs, but this may create a bottleneck in the sealing pipeline.

For best performance, compile Lotus with CUDA support instead of OpenCL. For further information, see the Lotus CUDA Setup.

In the case of a Snap Deal, an existing committed capacity sector is filled with data. When this happens, the entire PC1 task does not run again; however, the snapping process employs PC1’s replica-update and prove-replica-update to add the data to the sector. This can run on the PC2 worker or on a separate worker depending on your sealing pipeline capacity.

When PC2 has completed for a sector, a precommit message is posted on-chain. If batching is configured, Lotus will batch these messages to avoid sending messages to the chain for every single sector. In addition, there is a configurable timeout interval, after which the message will be sent on-chain. This timeout is set to 24 hours by default. These configuration parameters are found in the .lotusminer/config.toml file.

If you want to force the pre-commit message on-chain for testing purposes, run:

lotus-miner sectors batching precommit --publish-now

The sealed sector and its 11 layers are kept on the scratch volume until Commit 2 (C2) is complete.

WaitSeed

WaitSeed is not an actual task that is executed, but it is a step in the pipeline in which the blockchain forces the pipeline to wait for 150 epochs as a built-in security mechanism. With Filecoin’s 30 second epochs, this means 75 minutes must elapse between PC2 and the next task, Commit 1 (C1).

Commit 1

The Commit 1 (C1) phase is an intermediate phase that performs the preparation necessary to generate a proof. It is CPU-bound and typically completes in seconds. It is recommended that storage providers run this process on the server where C2 is running.

Commit 2

The last and final step in the sealing pipeline is Commit 2 (C2). This step involves the creation of zk-SNARK proof. Like PC2, this task is GPU-bound and is, therefore, best co-located with the PC2 task.

Finally, the proof is committed on-chain in a message. As with the pre-commit messages, the commit messages are batched and held for 24 hours by default before committing on-chain to avoid sending messages for each and every sector. You can again avoid batching by running:

lotus-miner sectors batching commit --publish-now

Finally, the sealed sector is stored in the miner’s long-term storage space, along with unsealed sectors, which are required for retrievals if configured to do so.

Was this page helpful?

RAID configurations

Storage systems use RAID for protection against data corruption and data loss. Since cost is an important aspect for storage providers, and you are dealing with cold storage mostly, you will be opting for SATA disks is RAID configurations that favor capacity (and read performance). This leads to RAID5, RAID6, RAIDZ and RAIDZ2. Double parity configurations like RAID6 and RAIDZ2 are preferred.

The width of a volume is defined by how many spindles (SATA disks) there are in a RAID group. Typical configurations range between 10+2 and 13+2 disks in a group (in a VDEV in the case of ZFS).

RAID implications

Although RAIDZ2 provides high fault tolerance, configuring wide VDEVs also has an impact on performance and availability. ZFS performs an automatic healing task called scrubbing which performs a checksum validation over the data and recovers from data corruption. This task is I/O intensive and might get in the way of other tasks that should get priority, like storage proving of sealed sectors.

Another implication of large RAID sets that gets aggravated with very large capacity per disk is the time it takes to rebuild. Rebuilding is the I/O intensive process that takes place when a disk in a RAID group is replaced (typically after a disk failed). If you choose to configure very wide VDEVs while using very large spindles (20TB+) you might experience very long rebuild times which again get in the way of high priority tasks like storage proving.

I/O Behavior

The unsealed copies are used for fast retrieval of the data towards the customer. Large datasets in chunks of 32 GiB (or 64 GiB depending on the configured sector size) are read.

In order to avoid different tasks competing for read I/O on disk it is recommended to create separate disk pools with their own VDEVs (when using ZFS) for sealed and unsealed copies.

Write performance

Write access towards the storage also requires your attention. Depending how your storage array is connected (SAS or Ethernet) you will have different transfer speeds towards the sealed storage path. At a sealing capacity of 6 TiB/day you will effectively be writing 12 TiB/day towards the storage (6 TiB sealed, 6 TiB unsealed copies). Both your storage layout and your network need to be able to handle this.

If this 12 TiB were equally spread across the 24 hrs of a day, this would already require 1.14 Gbps.

12 TiB 1024 / 24 hr / 3600 sec 8 = 1.14 Gbps

The sealing pipeline produces 32 GiB sectors (64 GiB depending on your configured sector size) which are written to the storage. If you configured batching of the commit messages (to reduce total gas fees) then you will write multiple sectors towards disk at once.

A minimum network bandwidth of 10 Gbps is recommended and write cache at the storage layer will be beneficial too.

Read performance

Read performance is optimal when choosing for RAIDZ2 VDEVs of 10 to 15 disks. RAID-sets using parity like RAIDZ and RAIDZ2 will employ all spindles for read operations. This means read throughput is a lot better compared to reading from a single or a few spindles.

There are 2 types of read operations that are important in the context of Filecoin:

• random read I/O:

  When storage proving happens, a small portion of a sector is read for proving.

• sequential read I/O:

  When retrievals happens, entire sectors are read from disk and streamed towards the customer via Boost.

Internet bandwidth

LAN bandwidth

The bandwidth between different components of a network is also important, especially when transferring data between servers. The internal connectivity between servers should be at least 10 Gbps to ensure that planned sealing capacity is not limited by network performance. It is important to ensure that the servers and switches are capable of delivering the required throughput, and that firewalls are not the bottleneck for this throughput.

VLANs

Virtual Local Area Networks (VLANs) are commonly used to separate network traffic and enhance security. However, if firewall rules are implemented between VLANs, the firewall can become the bottleneck. To prevent this, it is recommended to keep all sealing workers, Lotus miners, and storage systems in the same VLAN. This allows for data access and transfer without involving routing and firewalls, thus improving network performance.

Redundancy

Network redundancy is crucial to prevent downtime and ensure uninterrupted operations. By implementing redundancy, individual networking components can fail without disrupting the entire network. Common industry standards for network redundancy include NIC (network interface card) bonding, LACP (Link Aggregation Control Protocol), or MCLAG (Multi-Chassis Link Aggregation Group).

Common topologies

Depending on the size of the network, different network topologies may be used to optimize performance and scalability. For larger networks, a spine-leaf architecture may be used, while smaller networks may use a simple two-tier architecture.

Spine-leaf architectures provide predictable latency and linear scalability by having multiple L2 leaf switches that connect to the spine switches. On the other hand, smaller networks can be set up with redundant L3 switches or a collapsed spine/leaf design that connect to redundant routers/firewalls.

It is important to determine the appropriate topology based on the specific needs of the organization.

Being a Filecoin storage provider involves more than just storing customer data. You are also responsible for managing Filecoin wallets and running systems that require 24/7 uptime to avoid losing collateral. This means that if your network or systems are compromised due to a security intrusion, you risk experiencing downtime or even losing access to your systems and storage. Therefore, maintaining proper security is of utmost importance.

As a storage provider, you must have the necessary skills and expertise to identify and mitigate potential security threats. This includes understanding common attack vectors such as phishing, malware, and social engineering. On top of that, you must be proficient at implementing appropriate security controls such as firewalls, intrusion detection and prevention systems, and access controls.

Additionally, you must also be able to keep up with the latest security trends and technologies to ensure that your systems remain secure over time. This can involve ongoing training and education, as well as staying informed about new threats and vulnerabilities.

In summary, as a Filecoin storage provider, you have a responsibility to ensure the security of your customer’s data, your own systems, and the Filecoin network as a whole. This requires a thorough understanding of security best practices, ongoing training and education, and a commitment to staying informed about the latest security trends and technologies.

Network security

When it comes to network security, it is important to have a solid first line of defense in place. One effective strategy is to implement a redundant firewall setup that can filter incoming traffic as well as traffic between your VLANs.

A next-generation firewall (NGFW) can provide even more robust security by incorporating an intrusion prevention system (IPS) at the network perimeter. This can help to detect and prevent potential threats before they can do any harm.

However, it is important to note that implementing a NGFW with IPS enabled can also have an impact on your internet bandwidth. This is because the IPS will inspect all incoming and outgoing traffic, which can slow down your network performance. As such, it is important to carefully consider your bandwidth requirements and plan accordingly.

System security

A second layer of defense is system security. There are multiple concepts that contribute to good system security:

• Host-based firewall (UFW)

  Implement a host-based firewall on your systems (also called UFW on Ubuntu), which is iptables based.

• SELinux

  Linux comes with an additional security implementation called SELinux (Security Enhanced Linux). Most system administrators will not implement this by default because it takes additional consideration and administration. Once activated though it offers the highest grade of process and user isolation possible on Linux and contributes greatly to better security.

• Not running as root

  It is a common mistake to run processes or containers as root. This is a serious security risk because any attacker who compromises a service running as root automatically obtains root privileges on that system.

  Lotus software does not require root privileges and therefore should run under a normal account (such as a service account, for instance called lotus) on the system.

• Privilege escalation

  Since it is not required that Lotus runs as root, it is also not required for the service account to have privilege escalation. This means you should not allow the lotus account to use sudo.

It is crucial to have a backup of any production system. It is even more crucial to be able to restore from that backup. These concepts are vital to a Filecoin storage provider because not only are you storing customer data for which you have (on-chain) contracts, you have also pledged a large amount of collateral for that data.