Validator Security: The first step to leveraging PoS
Decentralization is the heart of PoS blockchains. Decentralization of security, consensus, ledger storage, and transaction validation is what ensures network and blockchain security.
A blockchain is only as secure as its block creation mechanism.
Proof of Stake is a mechanism in which token holders in the chain stake their holdings to earn the capability to make decisions on the blockchain. Validation is the name for this process of making decisions on the blockchain. A validator essentially verifies transactions and in some cases votes on the blockchain governance decisions as well. Hence, a PoS blockchain relies on its validators for the validation and creation of new blocks as well as operational security to maintain the blockchain.
A validator is a node that has staked enough holdings to earn the validation rights, the assets may be owned solely by the node or lent from some other nodes who do not directly validate. A compromised validator is a risk for its own assets, delegated assets, and the network as well. Therefore, ensuring the security of the validator is of prime importance.
Appropriate Blockchain Selection
Staking indeed is lucrative. It is a prerequisite to understanding what aligns with the validator’s strengths to leverage the best from the opportunity. The evaluation of a suitable blockchain involves a multifaceted analysis.
Leverage your strengths
Search for blockchains that support your strengths. A validator good at infrastructure or op-sec should look for blockchain where that is sought after. For someone good with smart contracts, a blockchain that provides programmable staking should be the choice. For game theory and economics experts, corresponding blockchains will be suited. The validator should build (in isolation) tools to reinforce its strengths and put them to best use.
Explore Reward Earning Options
Staking is the source of primary income in a PoS blockchain. It enables the validator to be able to verify transactions and earn incentives for doing so. However, most blockchains do have other modes to earn incentives as well. While selecting a blockchain consider the incentivization paradigms, it may include voting for chain governance decisions, exposing a compromised or a double signing validator, revealing the network vulnerabilities, or enhancing the smart contract.
The list may vary according to the blockchain and its process of verification.
As a validator make it a point to understand these and leverage them to increase incentives.
Nothing comes for free, and the validation opportunity for PoS blockchain is no exception. To set up a validator, take into account the costs of hardware, manpower, and the currency of the blockchain.
A validator needs memory to download the ledgers, hence, cloud storage solutions to manage the data center and software solutions to manage and run the validator.
Next in line, is the manpower, these are the validators and the team members who handle any unprecedented incidents and remain on alert 24x7x365. this is generally a rotational team.
Finally, is the hardware, without which nothing can be accomplished. a validator needs to check the local hardware costs and the costs incurred in separating the network access layer and key management (discussed below) layers from the validator as well.
Besides the above costs, a team of experts or service providers to handle regulatory and legal issues as and when they pop up is a must. They deal with taxation and commission disbursement costs as well.
Most importantly, the validator needs assets to stake and any asset on a blockchain can be owned either by being a long-term contributor from the smart contract or white paper stages or by using fiat to buy the stakable cryptocurrency/token to stake.
An understanding of the costs is essential to evaluate the profitability of participating in a blockchain’s consensus mechanism.
Every blockchain network sets in place some mechanisms to protect itself from any malicious behavior. PoS blockchains use slashing, which is confiscating the staked assets or even banning from consensus participation, for any behavior with the potential to hamper the blockchain’s security.
Classified majorly into two categories: uptime slashing and slashing for equivocation. Below are some of the most common slashing criteria:
- A validator producing two blocks of the same height
- A validator submits an invalid consensus vote i.e. cryptographically verifies a transaction that may be illegitimate or be against the interests of the blockchain.
- A validator fails to be live on the network for the required epoch.
- A case of double-spending by the validator is discovered
- Other validators on the network present evidence of a double signature or improper security of the validator. This one is like placing a bounty over validator security.
- An unfortunate event of the validator falling prey to mass slashing attacks by hackers, who may cast conflicting votes or simply initiate unstaking rendering it off the validator rights.
There are blockchains that function on ‘pure trust beliefs’ and do not employ slashing to maintain validator integrity though there might be other penalties or banning from the blockchain for unwanted behavior.
The validator node needs to download the blockchain and remain active on the network for extended periods. A top-level requirements assessment reveals that a validator would need backups for power cuts, storage, and network, and seamless connectivity solutions as staying connected to the network are primal in the blockchain world. General advice for not using a basic developer machine and rather using cloud-based tools to run the validator is pretty obvious based on the fact that an individual can never be sure of their machine not being compromised.
Below is an illustration of a prototype for a validator architecture based on what validation service providers have evolved into over the years. It depicts the layering of the validator based on the functional units that should have independent machines and access capabilities.
The idea here is to isolate the validator from the network to prevent any network-level threats and safeguard the validator from denial of service or mass slashing attacks. The second layer is the key management layer or remote signature generation mechanism, separating the key management from the participating validator and introducing another layer to apply security checks before signing any transaction. The blog discusses key management checks further.
SLOs and SLAs
Service level Objectives and Service Level Agreements should complement each other for a smooth validator function.
Service Level Agreements:
These are the requirements a validator agrees to fulfill once, elected for verifying transactions. Apart from verifying transactions, a validator may even propose blocks and vote for the blocks proposed by other validators.
Service Level Objectives:
A validator must be clear on the objectives of participating in the network maintenance process and ensure that they have enough resources, available stakes, and setup. A validator with the sole aim to earn more to stake will follow a different mode of operation than someone who aims to stay on the network to ensure security. While a sole validator would only take care of his/her personal asset goals a service provider needs to ensure that the delegator’s goals are also taken care of.
Beyond recognizing the SLOs and SLAs and the fact that the validator has the capability to manage both, it is important to make sure they form a functional union in favor of enabling the validator to stay active and earn rewards on the network. As an example, if a validator cannot stay online at predefined fixed time durations, a blockchain with a slashing for failing to be live on the network is going to be a suicide.
Triple Edged Security
The only rule of security is to deter breaches by making the attacking costs/effort too big in comparison to the benefits received. The security of the validator based on this principle has three aspects which are discussed below:
Key Security :
The keys on the blockchain network are what keep it secure. A key is a cryptographically generated code that a validator must use to perform its activities. Keeping the key secure is the goal of all security activities on the validator level. The cryptographic keys on a blockchain network are the account key and consensus key. It depends on the development rules of the blockchain that both are the same or different.
The Account key is a cold key, used very rarely when a validator node wants to change the details, reward system, commission schedule, or even rotate consensus among the team members. This key is advised to be kept off-network in something known as a cold wallet. This keeps it secure and in turn the validator. These hardware wallets must be airgapped and have multisig access.
The consensus key is the hotkey. It is used to vote for transactions, propose or sign the blocks and even vote for the blockchain governance decisions. This key will be used regularly for each and every transaction that goes through the validator. The consensus key is managed and maintained by the key security mechanisms.
Third-party key management solutions:
- Custom apps to ledger the keys – these have the drawback that they are not highly available and require physical presence for every single access. These kinds of solutions may be useful for cold keys but hotkeys need more fluid management options.
- Enclave solutions to manage keys– These are hardware solutions designed by the companies specializing in key securities. These can be used keeping in mind that the security is managed and handled by a third party thus, the validator’s trust in the solution must be unwavering and the validator should be able to utilize the provided facilities to the fullest.
- The most secure is a multi-part key management solution, where dependency is removed from a single cloud service provider and the key is partly stored on multiple wallets. Using the key requires access to each of the wallets. Distributing these wallets among team members adds another level of human-driven security at this layer.
Multiple key managers combined with enclave technologies further strengthen the security of key management systems. Using key management tools ensures a higher level of security from specializing organizations.
The node security is a two-part process. First is the physical security of the machine. Even though the validator uses cloud services to participate in staking processes, to access those some kind of laptop/computer system will be used. Ensuring the security of the same is the responsibility of the individual who accesses it. The second is the security from any attackers on the network or a targeted denial of service attack by malicious users. For this, the added network security layer is there to have sentries relay the transaction on the network preventing the validator from directly coming in contact with the network. Sentries possess the capability to evade network layer attacks while the application layer attacks need to be dealt with on the validator machine only.
For this reason, the signing capabilities are isolated from the validator node and a remote signer is added in between. The remote signer accesses the key after performing some more checks. Even though the remote signer recognizes the validator, it is essential to test the payload of the request to ensure that the keys are being used for legitimate purposes only as it’s best to assume that validator security might be compromised. The checks to be performed on the remote signer include verifying that the transaction is a validation transaction only. Ensure that the transaction load is generally uniform.
The remote signer should filter the possibilities of double signing by leveraging a high watermark or a monotonically increasing counter. Also, in case of disaster or failure recovery/rollbacks double signing detection proves to be vital. As an added level of precaution, list and understand all dependencies and their dependencies as well.
The next step is to automate the validation process to the maximum extent thus preserving it from any kind of human error. Basically restricting the production access to a bare minimum. For this, it is advised to utilize cloud storage or a git repository controlled by at least two team members to ensure that any change goes through a dual set of eyes before being committed to the system. Having cryptographically secured signatures strengthens the security on the repository level.
Apart from versioning the code and keeping track of all kinds of payouts, and commission disbursements it is essential to manage and censor the changes happening in the git code.
Track the SLOs and SLAs through a tracking system and update them from time to time to maintain high availability and function effectively on the network. A mistake here generally proves fatal for the validator node.
Test your Set-up
Emulate common failures:
Begin by causing basic failures like power or network failures, Denial of Service attacks, and test out remote signer vulnerabilities. Along with this direct peering with other trustable high-security validators, ensures better network security. Even if the network itself is having some loopholes a system of secure validators peering together may override any network-level attacks.
Be your own evil Twin:
Push your limits to test out the vulnerabilities if someone manages to get access to all the security knowledge that you have. Be your own evil twin for the validator. If any person can single-handedly manage to breach the system, you need to check your security again.
Learn from the Past:
Learn how validators have been compromised previously test those scenarios, create more scenarios around them and test validator resilience.
Always! Always I repeat you must start with testnets, before lodging your validator on the main-net. This gives an idea of how the actual functioning will proceed on a daily basis and reveal if any concerns remain unattended.
Monitoring: the only task inevitable on a regular basis is monitoring all the transactions, inflow, and outflow of stakes, accesses to cold/hotkeys should be diligently monitored. Keep records of all kinds of access.
Set up alerts for any kind of deviation from the natural operation. Make sure to classify alerts on their urgency and attend to them appropriately. Have a team to handle critical issues 24×7. Prepare on-call rotation schedules with escalation policies. Humans tend to turn lethargic or forget disaster management protocols, and regularly conduct mock trials of failure scenarios. This helps them stay calm and prepared to handle those situations under pressure as well.
Protocols for handling and Reporting accidents:
In case of any untoward incidents, ensure that proper mitigation of risks is done as per the protocols and that everything is documented and reported for analysis and further security enhancement purposes. What, how, when, and where of the incident should be clearly stated and test the validator against them over and over again. Use these incidents to improve practices for the future.
It is inevitable to spot vulnerabilities in a system even after deploying it. What’s worth noting is once up and running it becomes all the more essential to keep tabs on the monitoring data and what is going on in the space. A validator must ensure that all aspects of due diligence are performed with utmost care before starting out as a design created with security in mind is surely a good indication for a system to brave the tough network problems. Post that remaining alert to be able to handle any untoward situations is key to handling any kind of attack.
PrimaFelicitas is a blockchain-centric organization with expertise in building highly secure multisig wallets, security key management solutions, and validation service platforms and builds end-to-end security solutions for blockchain nodes.
Looking for help here?