Working, in the light of Christ

Update on DDos attack

Dear Members and visitors,

This is an update to the degraded service event experienced by us over the last few weeks.

We previously communicated via twitter on 23rd September that our site was back online and there has been no recurrence of the event since 01:00 Friday 25th September.  Then on 5th October at 22.57 our IP address was once again the subject of another attack; due to the measures we had put in place the attack did not adversely affect our service.  During the attack periods when posting new jobs or news articles to twitter, images were sometimes not being embedded.  After the attack on Monday, our current service provider was concerned that if the attack was a prolonged one it could put their network in jeopardy and have a knock-on effect for other clients.   The attack on Monday lasted for two hours and for the following four nights at roughly the same time and for an average duration of two hours, during this period our site was live and was slow and some visitors received a 503 Error message.

Whilst we want to be as open and transparent as possible we also want it not to be “too techie” – but this is difficult due to the nature of the attack. If you would like to clarify anything further then please contact us.

If you would like to skip the timeline and explanation as to what happened, please scroll to the bottom of this mail to the heading of “The Future”.


THE ATTACK

As previously communicated, the degraded service was due to a Distributed Denial of Service (DDoS) attack. We deal with DDoS attacks on a routine basis but this attack was both exceptionally large at circa 80-100 Gbps against multiple targets and for multiple hours. In this instance, the attack was specifically targeted our IP address and the DNS name servers of our hosting provider.

There were four distinct phases of the attack:

  1. Circa 15:30 GMT 22/09/20 to 00:15 23/09/20, against our IP address.
  2. Circa 12:00 GMT 23/09/20 to 00:00 24/09/20, against the same name servers of our hosting provider.
  3. Circa 12:20 GMT 24/09/20 to 01:00 25/09/20, against all of the above.
  4. Circa 22:57 GMT 05/10/2020 to 01.12 06/10/20 against our new IP address. This attacked was repeated each night at the same time until 12.25 09/10/2020


THE IMPACT

The most immediate impact was against the attacks of the 22/09/20 and the 23/09/20:

  1. We quickly identified the problem and moved to transfer catholicrecruitment.co.uk from our hosting providers primary network and onto a secondary service, ensuring the service had traffic-scrubbing DDoS mitigation. This was complete by the early evening, and the Internet “glue” records which define the IP’s of the name servers were changed at the registrars. Due to the way these records work, propagation of this change then took some time and it was not expected to fully resolve across the Internet the issue before the next morning.
  2. The attack stopped later in the evening, and DNS services appeared to be resolving correctly by 01:00 23/09/20. Due to the DDoS attack ceasing, and the time taken for the glue records to propagate around the Internet, we were unable to validate that the new infrastructure for catholicrecruitment.co.uk was able to serve requests with an attack in progress, but we had no reason to doubt it would.
  3. The attack recommenced on 23/09/20 circa 13:00 GMT and it became apparent that the new provider seemingly could not scrub the amount of DDoS traffic involved and maintain clean DNS requests to the servers. In fact, during the attack of 23/09/20, DNS requests against our new provider were taking between 5 – 15 seconds to resolve. Effectively, this meant that the requests timed out and the name servers were functionally compromised against this volume of traffic. We then chose and moved the secondary name servers to another supplier, and again changed the glue records. This time, the chosen supplier could cope with the volume of the attack and maintain clean DNS requests to the secondary name servers.
  4. Whilst propagation of the glue record change around the Internet again took several hours, we believe that the vast majority of DNS requests were being answered and our website largely returned to normal by the evening of 23/09/20. This was based off observed traffic, which had returned to almost nominal levels.
  5. During the attack of the 23/09/20, the DNS servers of our new hosting provider became under attack, and the nameservers were moved. DNS glue records were changed for these nameservers and these changes again took some time to propagate around the Internet

Further to this, our website was attacked on the evening of 05/10/20 for two hours and this attack continued for the next fours night for a two hour duration.

  1. Whilst our website individually exists on different infrastructure, the attackers again attacked multiple IP endpoints simultaneously. Rather than follow a similar approach to moving the secondary name servers off network, we elected to move the websites behind CloudFlare and leave the web servers on our network. This had been scheduled internally for some time, but not completed due to some outstanding issues pending resolution. However, in the circumstances, we elected to reverse proxy the websites behind CloudFlare and deal with any issues arising as a consequence of the move.
  2. Again, this required DNS changes to propagate around the Internet and took some hours for service to be restored.
  3. We restored primary customer access to catholicrecruitment.co.uk by the early evening of 24/09/20. The DNS record propagation to CloudFlare controlled space was however underway and could be changed quickly the following day.
  4. On the 08/10/20 our new provider made it clear to us that they were unhappy with those attacks on our website and their infrastructure. We had planned to move to a more secure data centre during October half term. With the weekend approaching this plan was implemented on Friday morning with the website going offline Friday evening to go live on our new data centre and our own server from 10.00am on 10/10/2020. In fact we were 12 minutes late going live due to our SSL certificate being issued late!


THE FUTURE

In terms of observations, lessons learned and future planning:

  1. The move of the secondary name servers to being off-network and with different suppliers was already planned earlier in the year, but not completed – for a variety of reasons, COVID-19 induced-delays included. Changing a secondary name server locations usually requires significant lead times and we did not want to enforce this change during times where suppliers might be on furlough or otherwise unavailable.
  2. By the 24/09/20, the move of the secondary name servers behind a DDoS scrubbing service had largely nullified the attack, barring changing custom or improper name server records. With the above process already completed, the same attack vector should not be able to succeed in the future.
  3. The move of catholicrecruitment.co.uk behind CloudFlare was similarly planned, and tests had already been undertaken. This lead to the exposure of a critical fault with twitter automatic postings, which we were working through and testing various fixes, without impacting customer operations. Again, the nature of the attack changed the need to restore catholicrecruitment.co.uk and pushing on with the move behind CloudFlare, and then dealing with any matters arising later.
  4. By 25/09/20, the move of catholicrecruitment.co.uk behind CloudFlare and location obfuscation largely resolved availability. Whilst attacks against websites are continuously evolving, the attacks used on 23/09/20 and 24/09/20 against our websites should no longer be successful.
  5. Whilst we have already treated the infrastructure and services impacted between 22/09/20 and 25/09/20, and have our own DDoS network mitigation for our cloud servers, we are also now reviewing the use of an external DDoS scrubbing service for our inbound BGP traffic as another line of defence. We are already conducting testing against a small proportion of services for any problems or regressions and would expect that testing to continue during October with hopefully a rollout in November.
  6. After further attacks this week (05/10/2020) it became clear we needed to speed up our plan of having our own cloud servers in a secure UK data centre, we were aware this was going to be costly. This was always our plan once we reached a certain number of Subscribers, with our new and our fifth Provider expressing their concern regarding the attacks, we felt we needed to move quickly to providing and supplying our own Cloud servers and use a third party DDoS scrubbing service. 
  7. On the 09/10/2020 we put into place the plans we had for October half term, our Servers and equipment were already in place and being tested at our data centre, so it was a case of configuring the Servers to go live and backup and retrieve email from mailboxes, purchase bandwidth for public access to our Servers.  This was connected to our data centre at 6.00am on 10/10/2020.  All that remained was to stress test the connection then upload our website and configure the Servers for live public access.  We had planned to go online at 10.00am unfortunately our SSL certificate was issued late and was not activated until 10.10am so after twelve minutes delay we went live at 10.12am!
  8. With immediate effect and for improved security we have closed all catholicrecruitment.co.uk mailboxes, you will be unable to send an email to any @catholicrecruitment.co.uk email address.  We have separated our website traffic from our email traffic.  All inbound emails will now go to a @catholicrecruitment.uk mailbox all emails will be sent from @catholicrecruitment.uk.  Both domains are located on different servers at different locations.  So hopefully in the event of catholicrecruitment.co.uk being attacked and being forced offline, we should be able to divert website visitors to catholicrecruitment.uk

There will be further changes and improvements, which we will inform you of in due course.

We hope you have found this informative in what happened and how we dealt with it, and also reassuring in terms of communicating how we have (and will be) mitigating it from happening in the future.

If you have any further questions then please don’t hesitate to ask us, and once again we thank you for your understanding and patience.

God bless,

The Catholic Recruitment Team