EOSIO Network back live, despite early hiccup

Less than a week after its Mainnet launch, the EOSIO blockchain unexpectedly experienced an issue where transactions became frozen on the network.  According to a status update in the network Telegram group, EOS Block Producers and Standby Nodes held a conference call this morning to identify and fix the bug.  The result of the call was a decision by Standby Block Producers to stop their network nodes and back-up any information that could be helpful in diagnosing the problem.  Several hours later, the root cause of the issue was identified, which was due to how deferred transactions were handled.

EOS blockchain creator Block.one immediately released a patch which was verified by the Block Producers and Standby Nodes and subsequently rolled out to the publicly available network endpoints.

A full statement was provided from the Block Producers recapping the timeline and issue identified.  Please see the official details below.

This message is from the Top 21 Block Producers and Standby Nodes of the EOS Mainnet Blockchain.  AT 9:56 UTC the EOS Mainnet halted. At 10:01 UTC Block Producers and many Standby Nodes joined together on an international conference call to identify and fix the issue.  At 10:57 UTC The choice was made that all Standby Block Producers would stop their nodes and backup any information that could helpful in diagnosing the problem.  At 11:02 UTC, a method to unpause the chain was formulated and is currently underway. Normal EOS chain functions should be available within 3 to 6 hours from the time of the publication. Incoming connections to the network are not being accepted while the work is taking place. Please wait for an update from the Top 21 Block Producers before attempting EOS mainnet transactions. 

13:03 UTC — Update #1: We identified the root cause and are working on a fix. A full release outlining the root cause will be issued once we’ve completed the work.

13:09 UTC — Update #2: BlockOne is issuing a patch in tag 1.0.5 which will be rolling out in the next few hours. The following steps will be taken:

A nodeos fix will be released.

The block producer nodes will be upgraded.

Block producers which had locally produced blocks will need to resync up until block 1027926, with production paused, and enable-stale-production disabled.

All nodes will upgrade to 1.0.5

Block producers will sync to each other to resume production

Chain will be opened up for public access.

13:28 UTC — Update #3: The root cause was due to an issue with how deferred transactions were handled.

13:36 UTC — Update #4: Pull Request 4158: https://github.com/EOSIO/eos/pull/4158
The EOSIO software is designed in such a way that when an error of this type occurs the chain will pause to prevent a hard fork. All block producers were halted when they detected a break in consensus. A fix was written, and is being tested and will be rolled out by the block producers community.

14:13 UTC – Update #5: Tag 1.0.5 was issued by Block.One

https://github.com/EOS-Mainnet/eos/tree/mainnet-1.0.5

14:48 UTC – The network is live. The Block Producers and Standby Nodes responsible for responding to this fix were able to work with Block.One, identify the issue, and merge a fix within 5 hours time.

English: https://t.me/eosmainnet

Chinese: https://t.me/eosmainnetchina

Korean: https://t.me/eosmainnetkr

Español: https://t.me/eosmainnetespanol

The EOS Block Producers