Between November 4-6, Runbox experienced powerful DDoS attacks by a group calling themselves “Armada Collective”. Other security oriented email services such as ProtonMail, Hushmail, and Neomailbox were also attacked, as recently reported by Forbes.

The initial threats and attacks that attempted to extort money were withdrawn by the attackers on Saturday morning, when they offered an apology.

During the attacks we were focused on coordinating with our partners, putting in place countermeasures, and providing our customers with necessary information. Since the situation was unclear and evolved quickly, we decided to not publish any details that could inform the attackers in any way.

The situation is now under control and we are publishing this summary of the events as it may help shed light on what transpired against both Runbox and the other services that were attacked.

Timeline

Wed. 04-Nov-2015

• Shortly after 16:00 CET we received a message demanding payment in bitcoins before 07-Nov-2015 otherwise there would be a sustained Distributed Denial of Service (DDoS) attack on our servers. The message said they would also begin an attack on our servers to demonstrate that this was not a hoax.
• Within a few minutes the attack begun, and we took action to mitigate this. This was a DNS amplification attack which peaked at 34 Gbps, and our email services were down for around 10 minutes before we restored service. The total duration of the attack at various levels was for around 1 hour and there was a resulting mail queue that we cleared shortly after that.
• We immediately began discussing what measures we would take should this happen again, and along with our partners developed what we believe to be an appropriate and responsible plan of action. Paying the demand to avoid future attacks was not an option that was on the table at any point. We informed our partners we would not pay, and they confirmed without reservation that this was the right course of action.
• Once we had gathered sufficient details regarding the attack, including all IP addresses used, we reported the attack to NorCERT (The Norwegian Computer Emergency Response Team) and Økokrim (The Norwegian National Authority for Investigation and Prosecution of Economic and Environmental Crime).

Thu. 05-Nov-2015

• Throughout the day we continued to monitor the situation, and detected a short low level attack of 2-3 Gbps on our servers that didn’t cause us any problems and was dealt with by the countermeasures that we put in place following the first attack.
• Later in the afternoon we recorded another DNS amplification attack which peaked around 18:25 CET at 42 Gbps, which we dealt with effectively with no downtime.
• We also continued to monitor the problems being experienced by other secure email providers that were affected, and made contact with some to share information that might be of use to the Internet community.
• Talks continued through the day on various strategies we could adopt in different future scenarios depending on the type and duration of any further attack.

Fri. 06-Nov-2015

• Now that we had a plan of action in place we informed our customers via a further blog post and information on social media. In addition we sent an email to all customers.
• The immediate response we got via email, our blog and on social media was overwhelmingly positive and supported our stance not to pay the money demanded. In fact, 100% of the messages we received about this supported our stance on the payment, and we were offered support and help from around the world. We are extremely grateful for the huge amount of support from our customers and others as it enabled us to confidently make difficult decisions in what is often an ever changing situation.
• During the morning we had a further demand from attackers for payment. The message quoted an extract directly from the blog post made by ProtonMail where they explain that to protect themselves they would need a solution that would cost them $100,000 per year. The attackers said that if we didn’t want to end up like ProtonMail, then we should pay.
• At 17:42 CET we suffered a further attack. This attack was more sophisticated and sustained than the first ones. The attackers employed both DNS amplification, NTP amplification, and TCP SYN flooding, and exceeded 50 Gbps. We adapted a more radical strategy in order to protect ourselves and our partners. Due to the countermeasures we put in place many of our services ran normally during the attack, and depending on location some customers were able to access them. We continued to respond to support requests during the attack.
• At 22:30 CET we decided that the level of the attack had dropped enough that we could adjust our countermeasures in order to bring up more services. Incoming mail from servers that were not able to deliver to us during the attack started arriving, and we monitored the situation closely to ensure that the incoming servers were not overwhelmed by the large amount of mail that was arriving for our customers. Mail delivery was back to normal at around 03:00 CET.
• Among the messages that arrived following the attack, was a message apparently from the attackers. They described their message as an apology, and said that attacking Runbox was a mistake and that we should not pay them. They were keen to point out that they were not behind the sustained attack on ProtonMail. They insisted that while they were responsible for the initial 15 minute attack on ProtonMail, the ongoing attack was not of their doing. The message was very much at odds with the previous messages, and while we do believe it was genuinely from the attackers, we have no way to verify this for sure.

Sat. 07-Nov-2015

• As already mentioned, mail delivery was back to normal at around 03:00 CET, and as of the time of writing we have no reason to believe we will be attacked again. We are pleased to say we have some excellent people working with us who are monitoring the situation closely. We remain on alert.

Commentary

The ability to keep Runbox running and accessible for all our customers is essential to us. The support we had both publicly and privately from customers and our partners Copyleft Solutions AS and Blix Solutions AS was extremely important in allowing us to make difficult decisions in dealing with these attacks.

Moreover, the response from our customers after we informed them that we would not pay any ransom was overwhelmingly positive, and we are sincerely grateful for the support we received via email and other comments.

Regardless of the apparent apology we finally received from the attackers, DDoS attacks are a criminal act. We built our company on the core values of ethics, security, and privacy, and we will continue to work with our partners, customers, and others to defend those values.