Upgrading Decred's Consensus

BY JAKE YOCOM-PIATT ON NOVEMBER 16, 2016

Over the next few releases, Decred will enable Proof-of-Stake (“PoS”) miners to vote on proposed consensus changes. This voting will allow Decred users to exercise sovereignty over whether or not to accept hard fork changes, in addition to making other important decisions. Enabling this decision making requires versioning several subsystems: block headers, vote transactions and transaction scripts. Versioning these subsystems allows for reliable and relatively smooth hard forking on an ongoing basis, meaning Decred can continuously upgrade its consensus code provided the PoS miners agree with each of the changes. Further, we have added a series of full-block consensus tests that are automatically run on every pull request to the consensus daemon, dcrd. Before this process can commence, we must implement a soft fork to add and enforce vote transaction versioning, and enforcement will occur as part of the next release, 0.7.0. Decentralizing the larger-scale decision making is the first step of several towards making Decred into a robust and fully operational decentralized autonomous organization (“DAO”).

Versioning consensus

The consensus code in any cryptocurrency is the most important and most delicate code in its codebase. Correspondingly, this consensus code is the real “magic” that makes a blockchain possible: without this shared set of rules about what is and is not considered an acceptable block or transaction, a decentralized ledger cannot have a reliable distributed timestamp. As such, the consensus code can comfortably be considered the most interesting part of a cryptocurrency, however, most projects are extremely conservative about modifying consensus code because it is so delicate and typically necessitates a hard fork. Some projects, e.g. Bitcoin, go so far as to avoid hard forks under all circumstances, and instead opt for routine use of soft forks. Based on what we’re doing with Decred, we obviously don’t agree with soft forks as a means to upgrade consensus, mainly because it breaks the social contract that comes with a blockchain, but that is a whole separate blog entry.

Instead of treating consensus code as a “delicate snowflake” that is only to be touched if absolutely necessary, we have instead chosen to figure out how to reinforce the snowflake and make it more resilient. Part of this reinforcement process is a system whereby all pull requests to dcrd, the consensus daemon, are subjected to automated full-block consensus tests and will not be merged to master until those tests pass. The full-block tests use a separately-coded test harness that allows for crafting custom blocks on demand to test branch conditions in the consensus code. In some ways, this could be viewed as making a “cast” of the snowflake and ensuring each change to dcrd’s code does not break that cast. By having this battery of tests run on consensus code, we can be confident that both existing and new consensus code is stable and will not create unplanned forks.

Going from one set of consensus rules to another requires signaling via versions in several subsystems. Some of the changes required are pretty obvious, like having a version in the block header, but the others are less obvious, e.g. adding versions to vote transactions and transaction scripts. The block header and vote transaction versions signal that Proof-of-Work (“PoW”) and PoS miners are on the latest version, which means PoW and PoS miners will both have the latest set of consensus rules and votebits enabled. The votebits, part of every vote transaction, encode what issues are currently being voted on by stakeholders. Transaction script versioning is required because you must know the script rules for old pay-to-script-hash (“P2SH”) transactions, which may not be spent until some time in the future. While transaction script versioning is important, the important parts for consensus versioning are the block header and vote transaction versions.

Soft fork: vote versioning and version enforcement

Stepwise versioning consistent with the block header for network consensus rules has been added to votes. We will be soft forking to add the vote transaction versioning and begin enforcing a particular version once 95% of both the block headers and votes show the new version in a rolling 144 block window. Blocks and votes with a version less than the latest version will be accepted until 95% of the blocks and votes both (separately) have the latest version, at which point nodes will begin to reject blocks and votes with any version less than the latest one. The vote versioning has been added in the 0.6.0 release and enforcement of the block header and vote versions will occur starting with version 0.7.0. With the soft fork complete, the process of upgrading consensus can begin.

Upgrading to a new consensus rule set

Once the versioning soft fork is complete and activated, it means that a supermajority of the users in the Decred network who validate blocks are upgraded to the latest version. By signaling the latest version of consensus, these PoW and PoS miners confirm that they have the latest definition of the votebits and the corresponding conditional logic to implement any consensus changes, assuming a given consensus change is voted in. After a new consensus version is activated, the voting process begins and runs for 8064 blocks, which is approximately 28 days. Over this period, up to 40320 votes are cast on the various issues encoded in the votebits for the current version. Issues receiving 75% or greater ratios of “yea” to “nay” votes over the voting period are considered approved and will go into effect after a lock-in period of 8064 blocks after the end of that voting period. Once the end of the lock-in period is reached, the issues that were approved in the prior voting cycle go into effect, meaning that consensus changes take effect at that point. This lock-in period is intended to allow users of Decred some additional time to upgrade their full nodes before the blockchain hard forks. The choice of parameters for the voting period, lock-in period and approval ratios was intentional and deserves some explanation.

The voting period was chosen to allow for nearly a full ticket pool worth of votes, 40320 tickets, to vote, but not be so long that it would drag for several months. By having the voting period be 8064 blocks, we allow 40320 votes to be cast, which is roughly a full cycling of the ticket pool. Further, 28 days is an acceptable length of time over which to allow voting: it is neither too short nor too long a period. Similarly, the lock-in period was chosen to be 8064 blocks since it is also neither too long nor too short and gives users a reasonable amount of time to upgrade their full nodes. This means that consensus changes take 2-3 months to be encoded in votebits, voted on and activated.

Choosing a 75% threshold for approval of an issue being voted on is based on considerations for the costs of creating a long-lived fork of Decred. In order to have a properly functioning blockchain, there must be a single chain that has only relatively short-lived forks, or else users cannot determine which transactions and blocks are valid and which are not. Decred’s hybrid PoW/PoS consensus requires that blocks must be mined by PoW miners and then voted on by PoS miners, which makes voting on hard forks not only possible but rather stable. If we assume there are 2 forks of Decred, call them F and F', where F' uses a newly approved set of consensus rules and F uses the previous set of consensus rules, we can use the PoS approval threshold, call it P, to derive an expression for the orphan rate of blocks on F, the “old” chain, as a function of P. This function, graphed below, shows that if we require an approval threshold, P, equal to 75%, it means that the orphan rate for blocks on F is approximately 90%.

This orphan rate effectively drives up the cost of PoW mining on fork F by a factor of 10, and this is assuming 100% of the hash power were to stay on fork F. In practice, this means that attempts to create a long-lived fork of Decred by a minority of dissenting stakeholders will have a drastically (10x or greater) increased cost of maintaining the chain. For other PoW-based cryptocurrencies that have a long-lived fork, such as Ethereum, the only financial barrier to a fork is a short term reduction in the speed with which blocks are mined. By having a high financial barrier to maintaining a long-lived fork, Decred ensures that spirited dissent does not lead to divisive forking of its blockchain.

While many cryptocurrency projects are content to decentralize only select components of their system, Decred will continue pursuing decentralization as a process to build a more robust and distributed ecosystem. By formally including stakeholders in the decision making process on an ongoing basis, Decred is creating a truly multipolar system where users are governed only with their explicit cryptographic consent. We will be announcing a longer term roadmap for this process in the near future since making decisions on what constitutes consensus is only one step in the process. A major milestone in this process will be a proper DAO, built without the pitfalls associated with attempting to encode a DAO into a single monolithic smart contract.

If you’d like to leave a comment, please do so on the forum thread for this entry.