# Lotus Miner: seal workers

The Lotus seal worker is a separate application that can be used to offload part of all phases in the sealing process to a separate machine or process. This guide explains how to setup one or several Lotus seal workers.

While the Lotus Miner can run every of the sealing phases itself (and is configured to do so by default), Lotus workers allow to create a sealing pipeline that can improve resource utilization and free the main miner from CPU-intensive tasks so that it can focus on performing and submitting WindowPoSTs and WinningPoSTs to the chain.

# Resource allocation in workers

Each Lotus Worker will run at most two tasks -each slot is called a window-. The final number is determined by the number of available cores and the requirements of the sealing phase allocated to it. That means that a single worker on a 8-core CPU with a single GPU will run at most:

  • 2 PreCommit1 tasks (using 1 core each)
  • 1 PreCommit2 task (using all available cores)
  • 1 Commit task (using all available cores or using the GPU)
  • 2 Unseal tasks (using 1 core each)

If the worker process has only 1 available CPU-core (because of CPU-affinity configuration or GOMAXPROCS=1), then it will only be able to run 1 PreCommit1 task as there are no resources available to utilize its second window.

Note that, at all moments, the miner controls the total number of sectors that can be sealing at the same time via MaxSealingSectors and/or MaxSealingSectorsForDeals in the Miner's config.toml.

All of the above serves to setup custom sealing pipelines where available hardware can be utilized in the most performant fashion, for example, by having co-located workers on a single machine to perform PreCommit1 only, and then having a dedicated GPU worker for the PreCommit2 and Commit phases on a different hardware, leaving the miner to do the rest of the operations. The final setup will depend on the available hardware and its specifications.

Remember during sealing, significant amounts of data are moved/copied across workers, so good network connectivity among them is a must.

# Installation

The lotus-worker application should have been built and installed along with the others when following the installation guide. For simplicity, we recommend following the same procedure in the machines that will run the Lotus Workers (even if the Lotus miner and the Lotus daemon are not used there).

# Setting up the Miner

The Lotus miner needs to be ready to accept API connections from workers.

# Allow external connections to the miner API

Set ListenAddress and RemoteListenAddress to the IP of a non-local interface (or as documented here. Check connectivity to the RemoteListenAddress from the worker machine.

# Obtain an authentication token

lotus-miner auth api-info --perm admin

The Lotus Workers will need this token to connect to the miner. For more info check the API docs. Write down the output so that you can use it in the next step.

# Configuring the miner sealing capabilities

The Lotus Miner is itself a worker and will contribute to sealing operations like any other worker. Depending on what phases of the sealing process you would like your workers to perform, you may choose to configure which ones the miner will directly perform. This is done in the Storage section of the miner's config.toml:

  AllowAddPiece = true
  AllowPreCommit1 = true
  AllowPreCommit2 = true
  AllowCommit = true
  AllowUnseal = true

If you want to fully delegate any of these operations to workers, set them to false.

# Launching Lotus workers

# Environment variables

Ensure that workers have access to the following environment variables when they run. These are similar to those used by the Miner daemon (explained in the setup guide:

# MINER_API_INFO as obtained before
export TMPDIR=/fast/disk/folder3                    # Used when sealing.
export MINER_API_INFO:<TOKEN>:/ip4/<miner_api_address>/tcp/<port>/http`
export FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1 # when GPU is available
export FIL_PROOFS_USE_GPU_TREE_BUILDER=1   # when GPU is available
export FIL_PROOFS_PARAMETER_CACHE=/fast/disk/folder # > 100GiB!
export FIL_PROOFS_PARENT_CACHE=/fast/disk/folder2   # > 50GiB!

# The following increases speed of PreCommit1 at the cost of using a full
# CPU core-complex rather than a single core. Should be used with CPU affinities set!
# See https://github.com/filecoin-project/rust-fil-proofs/ and the
# "Worker co-location" section below.

# Run the worker

lotus-worker run <flags>

The above command will start the worker. Depending on the operations that you want the worker to perform and the hardware that it is running on, you will want to specify for which sealing phases the worker will make itself available:

   --addpiece                    enable addpiece (default: true)
   --precommit1                  enable precommit1 (32G sectors: 1 core, 128GiB Memory) (default: true)
   --unseal                      enable unsealing (32G sectors: 1 core, 128GiB Memory) (default: true)
   --precommit2                  enable precommit2 (32G sectors: all cores, 96GiB Memory) (default: true)
   --commit                      enable commit (32G sectors: all cores or GPUs, 128GiB Memory + 64GiB swap) (default: true)


If you are running multiple workers on the same host, you will need to specify the --listen flag and ensure each worker is on a different port.

Once the worker is running, it should connect to the Lotus miner. You can verify this with:

$ lotus-miner sealing workers
Worker 0, host computer
        CPU:  [                                                                ] 0 core(s) in use
        RAM:  [||||||||||||||||||                                              ] 28% 18.1 GiB/62.7 GiB
        VMEM: [||||||||||||||||||                                              ] 28% 18.1 GiB/62.7 GiB
        GPU: GeForce RTX 2080, not used

Worker 1, host othercomputer
        CPU:  [                                                                ] 0 core(s) in use
        RAM:  [||||||||||||||                                                  ] 23% 14 GiB/62.7 GiB
        VMEM: [||||||||||||||                                                  ] 23% 14 GiB/62.7 GiB
        GPU: GeForce RTX 2080, not used

# Miner and worker co-location

You can run the Lotus Worker on the same machine as the Lotus Miner. This can be helpful to manage priorities between processes or better allocate available CPUs for each task. To avoid conflicts, we recommend that the enabled sealing phases in the worker(s) and the miner do not overlap.

Additionally, be mindful of the local resources used by the sealing process (particularly CPU). WindowPoSTs are CPU intensive and need to be submitted by the miner regularly. If a miner is performing other CPU-bound sealing operations in parallel, it may fail to submit the WindowPoSTs in time, thus losing collateral in the process. For this reason, we recommend careful allocation of CPU cores available and sealing phases to miner and workers.

Note that if you co-locate miner and worker(s), you do not need to open up the miner API and it can stay listening on the local interface.

# Worker co-location for PreCommit1

Since some operations are bound to a single CPU-core or core-complex (namely PreCommit1), it is possible to run multiple workers on a single machine in order to maximize CPU utilization.

For example, to run a PreCommit1 worker in a way that it does not conflict with others, do:

# Use a unique storage location for each worker
export LOTUS_WORKER_PATH=/path/to/worker/storage/N
# Replace X with a different port for each worker.
lotus-worker run --listen --addpiece=false --precommit1=true --unseal=true --precommit2=false --commit=false

By default, the PreCommit1 base will use a single CPU core. This means that several worker processes can use remaining cores and perform the work in parallel for this phase, as long as there is enough memory available for all of them to run.

Alternatively, it is also possible to speed up PreCommit1 by setting FIL_PROOFS_USE_MULTICORE_SDR=1. This enables optimizations to memory access (opens new window), but each worker will need access to a full CPU core-complex (usually 4 adjacent cores). This reduces the number of workers that could run in parallel, but allows them to be faster and saves the memory footprint of running additional processes.

In both cases, whether using the single-core or the multicore PreCommit1 worker, the process should be restricted to the use of a single CPU or a single, full core-complex respectivly. This can be achieved with taskset:

# Restrict to single core number 0
taskset -c 0 <worker_pid | command>
# Restrict to a single core complex (example)
# Check your CPU model documentation to verify how many
# core complexes it has and how many cores in each:
taskset -c 0,1,2,3 <worker_pid | command>

It is also possible to set CPU affinity with systemd (opens new window):

# workerN.service
CPUAffinity=C1,C2... # Specify the core number that this worker will use.

Finally, remember that MaxSealingSectors and/or MaxSealingSectorsForDeals in the Miner's config.toml affect how many sectors can be sealing at the same time, so they should be set as needed and accordingly to the worker configuration.