Disaster Recovery Journal Spring 2023

REGISTER TODAY! www.drj.com/spring2023

Spring 2023 u Volume 36, Number 1

INSIDE ... Southwest Airlines Holiday Meltdown The Human Side of Conducting BIAs Career Spotlight: Chelsea Selbig Emergency Notification Directory

Don’t Miss An Issue u Subscribe Today! u www.drj.com/#sign-up

Small-Medium Business Integrated Toolkit BIA Surveys, Dashbaords BC/DR Plan Templates 350+ Reports Low Cost

Global Enterprises Unlimited User Access Asset Geo-tagging Flexible, Customazible Integrated Workflow Voice, SMS Notification

www.eZPlan

(888) 480-3277

Platform for Building a Resilient Enterprise

• Cloud hosted • Secure Solution • Scalable • Product support • FREE upgrades

BIA, Plan Templates Gap Analysis Reports Exercise Management Role-Based Access Management Dashboards Management Consultants

powered by

nner.net

Info@eZPlanner.net

Your North Star for Resilience

Move From a Reactive to Proactive Approach

Fusion provides easy, visual, and interactive ways to analyze every aspect of your business so you can identify single points of failure, key risks, and the exact actions you need to take next to mitigate impact.

Get started on your resilience journey today! Visit fusionrm.com

“Fusion Framework System offers a fantastic tool to consolidate key elements to manage Continuity and Crisis Management programs from infancy to maturity.” - Manager, Technical Services Continuity

Disaster Recovery Journal 1862 Old Lemay Ferry, Arnold, MO 63010 (636) 282-5800; Fax: (636) 282-5802

Internet: www.drj.com E-mail: drj@drj.com EXECUTIVE PUBLISHER Bob Arnold bob@drj.com EDITOR IN CHIEF Jon Seals jon@drj.com ASSOCIATE EDITOR Pam Clifton PRESIDENT Bob Arnold bob@drj.com DIRECTOR OF EVENTS Lesley Vinyard lesley@drj.com REGISTRATION MANAGER Rose Chotrow rose@drj.com SENIOR WEB DESIGNER Amy Faulkner amy@drj.com PROGRAMS MANAGER Emily Rice emily@drj.com EVENT MARKETING Sonal Patel sonal@drj.com

T A B L E O F C O N T E N T S

COVER The State of

Business Continuity Preparedness 2023 By AMY DeMARTINE 8

22 Southwest Airlines Holiday Meltdown By MATT DOERNHOEFER

40 Using the FLOOD Acronym to Safeguard People, Property, and Other Assets By SHANNON COPELAND

EXECUTIVE COUNCIL Dan Bailey, Jeff Dato, Peter Laz, Frank Perlmutter, Ann Pickren, Steve Piggott, Tracey Rice, Randy Till, Damian Walch, Belinda Wilson EDITORIAL ADVISORY BOARD Selma Coutinho, James Green, David Halford, John Hill, Kim Hirsch, Ray Holloman, Colleen Huber, Vaishali Jain, Cary Jasgur, Lisa Jones, Joan Landry, Joe Layman, Melanie Lucht, Katherine Whitaker, Matt Ziska + (51) 1 436 6456 fijo Perú + 1 (786) 600 1864 USA ruth.rocha@drjenespanol.com www.drjenespanol.com ASIA Business Continuity Planning Asia Pte Ltd (BCP Asia) Henry Ee 1 Commonwealth Lane #08-27 One Commonwealth Singapore 149544 Phone: (65) 6325 2080 Fax: (65) 6223 5363 General: enquiry@bcpasia.com Events: conference@bcpasia.com Direct: henry@bcpasia.com www.bcpasia.com UNITED ARAB EMIRATES Continuity and Resilience ADivisionofCOREMANAGEMENTCONSULTING Dhiraj Lal , Executive Director P.O. Box127557, AbuDhabi, UnitedArabEmirates ( +971 2 8152831 | 7 +971 2 8152888 dhiraj@continuityandresilience.com www.continuityandresilience.com SOUTH AMERICA DRJ en Espanol Ruth Rocha , Directora Comercial

26 The Human Side of Conducting BIAs By MICHAEL HERRERA

42 Career Spotlight: Chelsea Selbig By DAVID HALFORD

31 Unlocking the Benefits of Business Continuity and Information Security Management By MICHAEL C. REDMOND, Ph.D. 34 Five Best Practices for Selecting the Right Target-based Deduplication Appliance for Hybrid Cloud Deployments By JEROME WENDT 37 A Roadmap from Business Continuity to Operational Resilience By HAKAN KANTAS

44 Top Tools for Network Optimization: Security, Reliability, and Efficiency in

2023 and Beyond By JAMES CAPUANO

50 DEI as a Business Continuity Topic By BOB ARNOLD on behalf of DRJ Staff

51 Emergency Notification Directory

DISASTER RECOVERY JOURNAL (ISSN 1079-736X; USPS 013-076; Publication Agreement No. 40679000) is published quarterly by Systems Support, Inc., 1862 Old Lemay Ferry, Arnold, MO 63010. Subscriptions are free to all qualified personnel in the U.S. and Canada involved in managing, preparing, or supervising business continuity planning. Rate for all others in the U.S. is $10, Canada and Mexico $24, all other countries $47. For renewals or change of address, please include current mailing label. Periodical Postage Paid at Arnold, MO and additional offices at St. Louis, MO. POSTMASTER: Send address changes to DISASTER RECOVERY JOURNAL, 1862 Old Lemay Ferry, Arnold, MO 63010. Canada Post Publication Agreement No. 40686534. Return undeliverable Canadian addresses to: DISASTER RECOVERY JOURNAL, PO Box 456, Niagra Falls, ON L2E 6V2. DISASTER RECOVERY JOURNAL is copyrighted 1987-2023, by Systems Support, Inc., all rights reserved. DISASTER RECOVERY JOURNAL is a registered trademark of Systems Support, Inc. Reproduction in whole or part is prohibited without expressed written permission. Articles submitted by readers do not represent the views or opinions of DISASTER RECOVERY JOURNAL and are published for their informational content only.

DISASTER RECOVERY JOURNAL | SPRING 2023 5

FROM THE PRESIDENT’S DESK

Learning from Southwest Airlines A s DRJ publishes its first issue of 2023, our first order of business has to be thanking Southwest Airlines. In my view, everyone who is committed to making industry more resilient owes Southwest a debt of

BOB ARNOLD, MBCI Hon.

disaster recovery methodologies and operational resilience, among other areas. The issue also includes our cover story by Amy DeMartine of Forrester Research (on Page 8) summing up the findings of the latest DRJ/Forrester survey on business continuity. The 2023 survey notes, among other trends, an encouraging rise post-COVID in the number of companies performing BIAs compared to previ ous years. Unfortunately, this is coupled with a persistent tendency for those BIAs to be super ficial and imprecise, according to the survey results. Another item in the issue—I’ve saved it for last not because it is the least important, but because it just might be the most important—is an explanation of DRJ’s commitment to foster ing a culture of diversity, equity, and inclusion on Page 50. Accompanying this explanation is a request to our readers for help in turning this commitment into action. Specifically, we would like to hear from people who are interested in serving on a board that will help us in developing measures we can take to promote diversity and inclusion across our enterprises, including our conference, magazine, podcast, advisory board, website, and social media efforts. Given DRJ’s central role in the industry, board members might even have a chance to be a catalyst for positive change in the profession over all. There’s a lot more in the issue that is well worth knowing. For those of you who are interested in something worth doing, look no further than DRJ’s Spring 2023 conference March 12–15, 2023, at the Orlando World Center Marriott in Florida. Our theme this spring is “Solutions for a Resilient Tomorrow.” As usual we’ll be bringing together the industry’s best minds to tackle its hottest topics. Nothing beats the immersive learning environ ment, networking opportunities, and overall ben efits of the live, in-person conference experience. I hope to see you there.

gratitude. The airline’s holiday meltdown at the end of last year—during which it had to cancel more than 16,000 flights affecting some two million passengers—provided an object lesson in the criticality of operational resilience. That lesson might have had a greater educational impact than all the articles and conference presentations ever given on the subject. Southwest’s difficulties were the inspiration for a funny “Saturday Night Live” commercial. But there was nothing funny about the estimated $1 billion the meltdown is expected to cost the airline in revenue lost to canceled flights and passenger refunds. Operational resilience—the ability of an orga nization to maintain a certain minimal level of functionality no matter what disruptions occur— has become a must-have for serious companies in every industry. Among the key drivers for this need are the fact that threats are multiplying, impacts are increasing, and the 24/7 expectations for busi ness are growing. All of these facts were at top-of-mind as we were putting together the current issue of Disaster Recovery Journal. In it, you will find an article on Page 22 examining the Southwest fiasco from a resilience perspective by Matt Doernhoeffer, a technical resilience professional with nearly 20 years of experience with multiple federal agencies, as well as Deloitte, USAA, and United Airlines. Also in the issue (Page 37) is a piece outlining how organizations can transition from a business continuity approach to an operational resilience approach, written by Hakan Kantas, an IT direc tor whose 30-plus-year career has focused on IT

PRESIDENT bob@drj.com

6 DISASTER RECOVERY JOURNAL | SPRING 2023 S IN 202

The State of Business Continuity Preparedness 2023 By AMY DeMARTINE

8 DISASTER RECOVERY JOURNAL | SPRING 2023

F orrester Research and the Disaster Recovery Journal have partnered to annually field market studies on business continuity (BC) and disaster recovery (DR) trends to gather data for company comparison and benchmarking, guide research, and for the publication of best practices and recommendations. This study, which focuses on BC maturity and preparedness, was first fielded in 2008 and again in 2011, 2014, 2018, and 2021. That first study provided a baseline for BC preparedness we can now compare to all the subse quent studies to see how BC maturity and preparedness are trending across time. These trends are more impor tant than ever as 2022 was a year to settle into the “new normal” of heightened risk events post-COVID-19. Specifically, we designed this study to determine the following:

n To what extent have companies formalized ongoing BC management programs with executive-level sponsorship? To which executive does the head of BCM report? n How frequently, if at all, do companies conduct a business impact analysis (BIA) and risk assessment? n To what extent are business owners involved in the BC management lifecycle? n How well do companies document, keep up-to-date, and test their BC plans? What types of tests do companies run, and how frequently do they run these tests? What tools do companies use to manage plans? n What is the scope of BC plans? What threat scenarios do they address? Do they include components for workforce continuity? Do they include components for crisis and emergency communication? n How many times have companies invoked their BC plans in the past five years? What was the cause? How successful was the invocation? Organizations who hoped to return to “normal” post-pandemic were met instead with a “new normal” of continuing supply chain issues, work at home embraced by an average of 23% rather than 5% pre-pandemic, a war in Ukraine, and financial uncertainty which has kept business continuity top of mind. In this survey, we found: n Executive sponsorship remained high . In our 2023 survey, executive sponsorship stayed high at 96% after the leap in 2021 to 94% from a consistent 88% in both 2018 and 2014 (see Figure 1). Proving the “new normal” keeps business continuity front and center, of the 96% who report executive support 42% report it’s “significant,” up from 38% in 2021 and 30% in 2018. n BCM programs’ biggest mover is a reporting line into risk . BCM programs are slightly more likely to report to IT than the business. According to our study, 32% of BCM programs report into IT such as the CIO or CISO – a decrease from 2021 (26%) Overall BC Executive Sponsorship Remains High, Risk Jumps as A Reporting Structure

but an increase from 2018 (30%) and 2014 (26%) (see Figure 2). Programs that report directly into business line executives (CEO, COO, CFO, HR, board, etc.) are holding steady at 28% from 2021 (29%). But the biggest move was those programs reporting into the chief risk officer (CRO) or head of enterprise risk. In 2021, only about 9% of BCM

DISASTER RECOVERY JOURNAL | SPRING 2023 9

programs reported into a CRO (a decrease from 14% in 2018) but respondents from this year’s survey reported 23% now report directly into the CRO. As many businesses rely on technology, the CIO and CISO wield power and influence which BC programs can benefit from, but the constant barrage of headlining risk events puts more pressure on risk to own business continuity. n The variety of reporting structures continues to fall . The “other” response into which department or office ahead of business continuity reports fell again this year to the sixth most common response from fourth in 2021 and second in 2018 and 2014. When we prompted respondents to specify other, we discovered “legal” was the most common response. Legal lingers as the home for risk programs overall with 4% of all respondents saying risk management reports into general counsel/legal when the organization does not have a chief risk officer. BCM Program Funding Remains the Same for A Majority of Firms BC budgets jumped in 2021 unsurprisingly, but now the major ity of firms expect their BC funding in the next 12 months to hold steady along with the number of full-time equivalents dedicated to BCM. More specifically, Forrester found: n Many respondents expect increased funding, but not quite the majority . According to our study, 47% of respondents expect funding for their BCM program to increase in the next 12 months. This is a decrease over the 52% who expected more funding in 2021, but an increase over 2018 numbers (36%) which was largely unchanged from 2014. Now, 52% expect their funding to stay the same versus 42% in 2021 and 50% from 2018. The good news is respondents who expected their funding to decrease has fallen to 2% which is below the 2021 level of 5% and the 2018 level of 11% (see Figure 3-1). When asked what prompted the increased funding, respondents unsurprisingly cited – by far – to mitigate increasing or evolving risk to the organization (9 out of 22 respondents). n Staffing varies by company size, but the median is three full-time staff equivalents . According to our study, the median number of full-time equivalents (FTEs) supporting the BCM program is three, which is the same as 2021 and 2018. The mean is 9 FTEs but the number from very large enterprises drags up the mean. Staffing always varies by size. In Forrester’s experience, companies with fewer than 1,000 employees typically have only one or two FTE(s) supporting BC (4.46 mean, 2 median from this study), while enterprises with 1,000 to 4,999 employees have between two and three FTEs (10.27 mean, 3 median from this study). Enterprises with 5,000 to 20,000 have three to five FTEs

(6.07 mean, 2 median from this study), and those with more than 20,000 employees often have distributed BCM programs with five to eight FTEs (21.13 mean, 8.5 median from this study) establishing standards and oversight at corporate headquarters and dozens of local BCM leads in region or by business unit responsible for local planning and execution. n Staffing continues to represent the largest portion of the BCM budget . So much of BC maturity and preparedness depends on planning, so it’s no surprise staffing represents 34% of the BCM budget – only slightly larger than the 30% from 2021 (see Figure 3-2).

10 DISASTER RECOVERY JOURNAL | SPRING 2023

Technology services for IT recovery fell drastically from 2021 numbers (19%) but still come in second at 10%. Other areas for investment including IT support for workforce recovery, IT support for crisis and emergency communication, software for BCM program and planning, and software for crisis emergency command remain steady from 2021 numbers. Even Though BIAs and Risk Assessments Are Popular, Practice Has Room to Grow Our study found the vast majority of companies conduct a BIA and risk assessment in advance of BCP strategy develop ment and plan documentation. More specifically, Forrester’s survey found: n An even larger majority of companies conduct a BIA . Eighty one percent of respondents reported having conducted a BIA; higher than 2021 (71%), 2018 (74%), and 2014 (75%) (see Figure 4-1). As the new normal of heightened risk events settles in, the BIA is seen as even more crucial as a method of identifying critical business functions which support the mission of the business, dependencies, and recovery objectives. Although inspiring, there is a difference between performing a BIA and collecting detailed information. For example, many companies Forrester engages with do not have a detailed mapping from critical business functions to the services, applications, IT components, and critical employees who support those functions. Additionally, cost of downtime is a rough estimate rather than a true quantification of cost. n Conducting a risk assessment leapt from an already high percentage . 2018 saw a huge jump in those companies conducting a risk assessment as 72% of respondents reported conducting a risk assessment – a 15-point increase. That remained statistically remained unchanged in 2021 (71%). However, in this study, 83% of respondents reported performing a risk assessment. Once again, while inspiring, in Forrester’s experience there is still room for improvement. For example, risk events are sometimes not increased in likelihood until too late such as ransomware (a type of cyberattack) is given high impact but low probability until another company in the same industry has a similar event happen even though in general ransomware attacks have increased dramatically since 2021. n Risk is increasing, and cyberattacks drive the increase . Sixty five percent of respondents believe the level of BC or operational

risk is increasing as compared to 61% in 2021 (see Figure 4-2). When asked what was driving the increase, respondents once again cited cyberattacks as the top driver (13 out of 22 respondents). A Mix of Scenario- and Impact-based BCPs Are the Norm, BCM Tools Bounce Back As of the 2014 study, the percentage of organizations with documented BCPs jumped to 93% and held steady since (respondents this year reported at 94%). Resilience during a crisis doesn’t come with luck but starts with planning and a BC program without BCPs is in dire straits. Forrester found the following in this survey:

12 DISASTER RECOVERY JOURNAL | SPRING 2023

RISK UNDER ONE ROOF

•Information System •Claims Administration

•Third-party Risk Management •Enterprise Risk Management •Internal Audit •Policy Management •Compliance •Project Risk Management •Business Continuity & Resilience •Environmental, Social & Governance •Health & Safety

IS NOW...

Together, we’ll give you integrated risk management with end-to-end visibility and true enterprise resilience.

SALES@RISKONNECT.COM | WWW.RISKONNECT.COM

n Preference for scenario-based plans leapt again . How a company responds to an IT outage is different than a weather event. Generic plans by impact (e.g., loss of IT services) are helpful because they help to prepare for any unforeseen event, but they can lack the specifics necessary to respond appropriately to some events. Dealing with the unstable normal of today, companies now much prefer a mix of scenario-based and impact based plans. Sixty-nine percent now report a mix (versus 49% in 2021) while only scenario-based fell from 2021 numbers at 20% to only 6% this year (see Figure 5-1). Scenario-specific BCPs are important because it shows an organization understands the detailed differences between how a business must respond such as the differences between an IT failure versus a ransomware attack. n The majority still update BCPs only once per year . Fifty-one percent of respondents report updating their BCPs once per year, down from 54% in 2021. The goal should be to continuously update BCPs as business functions and their underlying services change constantly. Unfortunately, those who report they update their BCPs continuously have fallen to 11% from 13% in both 2021 and 2018 and 15% in 2014 (see Figure 5-2). n Organizations are turning back to commercial software to manage BCPs . In 2021, the backslide away from commercial software to manage BCPs was concerning and prompted Forrester to predict because staffing was flat, executive support increased, and COVID-19 raised the visibility of BC to business operations and strategy levels. BCM tools would be recognized as essential for mid- to large-sized firms. Happily, in this year’s study only 38% of respondents reported using internal tools (i.e., documents, spreadsheets, etc.) versus a whopping 64% in 2021 and even 51% in 2018 (see Figure 5-3). An additional 20% of respondents plan to use commercial BCM software in the next 12 months to the already 42% who already use it. BCPs Are Still Not Tested Frequently, Partner Involvement Remains Static Even though we say it every year we do this study, it’s worth repeating – if you aren’t testing your BCPs, you are not prepared. Only through testing do any of the people expected to respond to an incident practice their actions and interactions. Despite years of urging from industry experts and consultants (including us), testing remains a major area for improvement across organiza tions of all sizes and industries. More specifically Forrester found the following: n Most organizations only test their BCPs once per year with simple tests . Unfortunately, the situation is largely unchanged from 2008. For all test types (walk-through, tabletop exercises, plan simulations), the majority of organizations only test once per year. As tests become more extensive, test frequency declines to the point where 56% (up from 47% in 2021) of respondents never

perform a full simulation (see Figure 6-1). Simulations test not only the incident actions, roles, responsibilities, and interactions between teams but also allows for timing of the various plan steps. Timing gives a sense

14 DISASTER RECOVERY JOURNAL | SPRING 2023

Simple, Flexible Business Continuity Solutions.

With an end-to-end solution, such as Agility Recovery, business can recover 4 times faster than with no BCM solution.

The only integrated business continuity solution in the market that helps you plan , train , test , alert , and recover — all in one.

866-364-9696 contactus@agilityrecovery.com www.agilityrecovery.com

Copyright 2021 - Agility Recovery All Rights Reserved

of whether recovery targets are realistic and where to pinpoint improvements to the plan. n Managing third-party risk remains a critical issue . In Forrester’s 2022 Business Risk Survey, after financial instability at 35%, 34% of respondents said the increased reliance on third parties is a primary driver of risk. Fifty percent of respondents report they have a formal program for assessing the BC readiness of critical third parties (up from 48% in 2021 but down from 56% in 2018) (see Figure 6-2). Due to the increased risk from third parties, we reformulated the responses in our survey about what steps are taken to assess and validate the BCP readiness of critical third parties. On the positive side, the highest number of respondents (19 out of 24) said they negotiate SLAs for specific uptime/availability as well as recovery time, recovery point capabilities, and associated penalties for SLA violations. Unfortunately, only 12 out of 24 respondents reported they use the detailed audit/assessment of a third party’s program and readiness as a decision-making tool to determine whether to begin/continue the partnership. The Business Needs to Take a More Active Role in the BCM Lifecycle After The BIA For a BCM program to truly be successful, not only do you need executive-level support, but you need line of business owners and employees involved in the entire BCM lifecycle as they are the ones who understand the inner workings and pri orities of the business. Unfortunately, again this year, we found participation from these business owners is too limited. Business owners are more likely to be involved in the BIA: 54% of respon dents report business owners are very involved – a significant increase from 2021 (38%) (see Figure 7). However, other areas such as awareness and training, risk assessment, and plan devel opment need much more business involvement. Strategies For Workforce Continuity and Communication Rely on Remote Workers Fundamentally, workforce continuity strategies changed during and after COVID-19. Employees dispersed from main sites and many embraced “work anywhere” opportunities when an organization offered it. On the surface, not much has changed as remote access continues its popularity as a workforce con tingency plan, but how plans are invoked and which employees should be notified now needs to include geographic regions, not just sites. In this survey, Forrester found the following: n Remote access remains the dominant strategy for workforce continuity. Remote access was the most common strategy even in 2008 (86%), hit a peak in 2018 (88%), and now sits at 82% (see Figure 8-1). The use of another internal site as an alternate site decreased notably in popularity from 2018 (75%) to 62% this year. Remote access procedures became popular to support employees who wanted to work from home or who travel frequently but became a necessity during the pandemic. They are effective when power and internet services are still available or when employees can travel outside of an affected area. However, when wide

16 DISASTER RECOVERY JOURNAL | SPRING 2023

A great day at work, whatever happens

Give your company the space to recover and make sure you and your team have a great day at work, whatever happens. Through our global network of thousands of locations, IWG can support your business in every major city, town, or transport hub in the world.

FIND OUT MORE

iwgplc.com/workspace-recovery

swaths of a geographic region (rather than sites) suffer a loss of power or loss of internet services, BC pros will need to monitor how this affects the ability to deliver a service and possibly invoke a plan depending on the concentration of workforce in that region and the services they support. n Email surpasses both text messaging and phones for communication. Now, companies can assume their employees not only have mobile phone access but also computer access. As a result, email is the most popular communication mode (87%), text messaging falls to second place (82%), and phone is third (78%) (see Figure 8-2). We also found using an automated software is much more the norm with 64% reporting they have already adopted this software (versus 45% in 2021) and other 7% planning to adopt in the next 12 months. Invocations Are More Frequent; Communication Is Key to Successful Invocations In previous reports, we highlighted plans are invoked more fre quently than organizations would expect as in each of the years we have fielded this study, more than half of respondents had invoked a BCP during the previous five years: 2008 (50%), 2011 (61%), 2014 (53%), 2018 (75%), and 2021 (69%). But to see proof orga nizations are feeling the barrage of risk events, look no further than to the 81% of respondents who said they have invoked a BCP during the previous past five years – the highest reported number we have ever seen (see Figure 9-1). Consider that: n After pandemics, natural disasters/extreme weather and IT failure top the list again . Seventy-six percent of organizations invoked a plan due to a pandemic/epidemic which we can easily attribute to COVID-19. However, after pandemics/epidemics, the next are the same common causes of extreme weather and natural disasters and followed closely by IT failures and power outages as in 2008, 2011, 2014, and 2021 (see Figure 9-2). In our last report, we noted the importance of organizations not making the mistake of focusing solely on catastrophic disasters because in reality, extreme but not catastrophic weather such as winter storms, can be the culprit behind the frequency of power outages. Shortly after the completion of the 2021 report, the February 2021 Texas Electric Grid Blackouts caused a loss of power for more than 4.5 million homes and served as a great reminder of this.

18 DISASTER RECOVERY JOURNAL | SPRING 2023

RESILIENCY eLearning

Business Continuity

Disaster Recovery

Crisis Management

Physical Security

Life Safety

IT Security

For Employees & Recovery Teams

TRAINING THATWORKS FOR YOU Customized content: reflect your program, methodology, culture, and brand. Track your learners: courses are compatible with your Learning Management System. Subject matter experts: we create the content and you validate the outcome. 3 - 6 week development time: a quick launch builds momentum for your message.

POPULAR COURSES

Active Shooter

Cyber Security Awareness

Awareness Campaign

DR: All Employee Intro

BC: All Employee Intro

Home Prep (no charge)

Business Impact Analysis

End-User Software Training

CM: Introduction

Physical Security

CM: Roles & Responsibilities

Table-Top Exercise

w w w . r i p c o r d s o l u t i o n s . c o m

n Communication and collaboration beat long-term duration as top lesson learned . Many organizations were caught off guard when the COVID-19 pandemic required a BC plan which accounted for not only a long-duration but needed to change over time based on local infection rates and hospital capacity. This year, the first top lesson learned was plans did not adequately address organization-wide communication and collaboration (see Figure 9-3). With regional risk events such as political instability and extreme weather, employees expect their employers to tailor their communication to the specific event while also allowing for self-reporting as to their status. The other top five lessons learned focus on the need for plans to account for long-term duration of events, employee health and safety, are not out of date or untested (Update your plans! Test your plans!), and adequately address workforce recovery requirements. Study Methodology In the months of October, November, and December 2022, Forrester Research and the Disaster Recovery Journal (DRJ) conducted an online survey of 58 business continuity decision makers and influencers. In this survey: n All respondents indicated they were decision-makers or influencers concerning business continuity. n Respondents were from a range of company sizes: 27% had 1 to 999 employees; 27% had 1,000 to 4,999 employees; 29% had 5,000 to 19,999 employees; and 17% had 20,000 or more employees. n Respondents were from companies with a range of revenues: 33% of respondents were from companies with revenues of less than $500 million; 4% were from companies with revenues of $500 million to $999 million; 34% were from companies with revenues of $1 billion to $4.99 billion; 6% were from companies with revenues of $5 billion to $10 billion; and 19% were from companies with revenues of more than $10 billion. n Respondents were from a variety of industries. n Respondents worked in North America, Europe, and Asia Pacific: 77% of respondents worked in North America; 8% worked in EMEA; and 14% worked in Asia Pacific. This survey used a self-selected group of respondents (pre dominantly DRJ members and Forrester clients) and is therefore not random. These respondents are more sophisticated than the average. They read and participate in business continuity and disaster recovery publications, online discussions, etc. They have above-average knowledge of best practices and technology in BC/DR. While nonrandom, the survey is still a valuable tool in understanding where advanced users are today and where the industry is headed. v Amy DeMartine is a vice president, research director for security and risk at

Forrester Research. As part of her current responsibilities, DeMartin over sees the development of individual research plans, report outlines, research methodology, drafts and graph ics; edits each research report for quality and excellence; and oversees development and delivery of team and individual advisory and consulting offerings.

20 DISASTER RECOVERY JOURNAL | SPRING 2023

green IT

consolidation portfolio comput ing vmware itil security GRC forrester wave ser vice desk por tal outsourcing vtl business continuity opsware asset management host disaster recovery email in the cloud change management virtualization web 2.0 metrics storage risk IT service community cloud computing

Research – Resources – Solutions Forrester delivers independent action-oriented insight to solve your biggest challenges. Visit us at www.forrester.com/drjournal to learn how our research, consulting, and executive programs will help you succeed.

Making Leaders Successful Every Day

Key Point 1: The theory “resiliency is the enemy of efficiency” is dead While I hate to identify our profession as fine purveyors of FUD (fear/uncer tainty/doubt), there are real, demonstrable cases where resilience is not the enemy of efficiency. An Airbus A-320 has three sets of control systems for hydraulic lines. This triple redundancy takes up precious weight that could be used for fuel or cargo. Airbus could have stopped at two fully redundant and physically separated systems. Instead, they chose to add an additional system to ensure the airplane can function despite the worst levels of damage to the airframe. This extra set of hydraulics is not considered a waste among the pilots, maintenance engineers, and executives running airlines. The extra redundancy is accepted as the ability to continue to function despite the worst disruption, turning potential catastrophe into minor inconvenience. This is the mentality executives need to take. Resilience is not a drain on efficiency, it is a normal and accepted practice in operational costs. We as professionals can change the narrative around resilience from one of additional expense to part of the primary design. Southwest’s failure shows that resiliency is not an enemy of efficiency. It can enable efficiency as disruption recov ery is a major and costly part of opera tions. Over time, no one will remember the inciting incident, but everyone will remember how the response was handled. Key Point 2: Technical debt is corporate debt Technical debt is a concept well known to CIOs and their staff but isn’t familiar to finance, operations, or front line business people. At the end of a system lifespan, the application and its component hardware become “technical debt.” Just like regular debt, tech debt can come in many forms. For younger or smaller orgs, code-based tech debt can plague even the most online system, while legacy orgs could have entire com puter platform’s worth of debt. At the end of the day, systems that can’t keep up with the rest of the world

Southwest Airlines Holiday Meltdown ‘Remember the SWAlamo’ By MATT DOERNHOEFER W ell, that happened. We witnessed the complete failure of an airline’s processes at the whims of a series of unfortunate but predict able events. Disruption is expected in airline operations, as weather patterns will constantly challenge the best flight plans. However, in this instance, Southwest This is a defining event for Southwest Airlines. Incalculable brand damage, indeterminate regulatory fines, and reimbursement of expenses for thousands of passengers all potentially threaten the solvency of the airline. There were reports of individual heroism managing to get some airplanes out. But in the end, the event showed the vulnerability of manual processes and their inability to work at scale.

took the term “disruption” to a new level. Recriminations are still happening, but the general consensus is (at time of writ ing) the technical debt collectors came, and they were sent by Tony Soprano.

So, what does this mean for the resil ience professional? What can we learn from this? I suggest this be our “Alamo” moment going forward. Let’s break it down into two key points:

22 DISASTER RECOVERY JOURNAL | SPRING 2023

A Leader in Business Continuity for Now 25 Years

3 solutions

has been helping worldwide organizations manage the unpredictable for 25 years. Whether you need to implement your business continuity program, strengthen it or automate it, you can count on us. Work with a well-rounded BC partner, with a collaborative and holistic approach that supports your teams at every step of your BCM program. Premier Continuum

Automation software

Certified training

World-class consulting

LET'S BUILD SMART RESILIENCE

6 fields of expertise We've been in the business for 25 years. Talk about resilience.

BUSINESS CONTINUITY

OPERATIONAL RESILIENCE

IT/DR

CRISIS MANAGEMENT

RISK MANAGEMENT

EMERGENCY RESPONSE

will present a multitude of issues. It could be a lack of staff who remember how the systems work, inability to get vendor support in the event of failure, or worse, the impossibility of restoring the applica tion to service because the org lacks the hardware to do so. Making a conscious decision to ignore tech debt is to accrue interest on that debt. That interest comes from higher failure rates, increased maintenance costs in extended support contracts, and a lack of future feature sup port for business resulting in more kludge solutions to achieve business goals. Oh, and when I say “update,” I mean a “complete overhaul,” not simply insert ing a nice-looking front end on a decrepit back-end system. So how is this our Alamo moment? This opens a new opportunity for us to raise the alarms around technical debt. Southwest technology personnel knew the risks and challenges, raised them to management, but were overall unable to sway the executives. In a letter from the second vice president of SWAPA (union), Tom Nekouei said, “This meltdown was easily avoidable. It was predictable and it was predicted.” That single-mindedness led them to the scenario they’re in now. Technology, like any other investment, has a set lifespan; and when you exceed that, you invite disaster. For the rest of us, we can use this moment to educate and demon strate the true risks involved with taking on technical debt. Use this as a teaching moment followed by an analysis of the risks and likelihoods and that should raise the hackles of the hardest cost-conscious executives. Tech debt systems have a variety of weaknesses which can become apparent if you know where to look. These data points can help you make the case about rationalizing your application platforms:  Number of skilled staff in the system – Particularly egregious tech debt will see a proportion of the system support staff retire or leave the company. The number of retirees of a team can be a red flag the platform is no longer supportable.

 Availability of skilled workers on the job market – If you find it difficult to hire personnel for a particular system, it might be a good sign you’re behind the technology curve. Then, if you manage to find someone willing to work on a dated system, they are often long tenured and extremely experienced – which is to say expensive to hire.  Increasing number and severity incidents – Apps that have routine incidents, or an increasing number of incidents over time can show an application in distress. An increase in minor incidents can result in a large amount of time lost to maintaining a cantankerous application. In addition, incidents that increase in severity will require more extreme measures to respond.  Increasing time to restore – When mean time to restore or triage time increases over a period of years, it can show an application where staff is no longer skilled enough to support – or the system has become too complex to triage and manage effectively.  Extended support contracts – Systems in extended support are beyond end of life from the vendor. Costs for these contracts increase exponentially, and if

you haven’t thought about replacing this system, you need to immediately. It is unfortunate when vendors drop planned obsolescence on us, but the pain of not upgrading comes from the extended support costs, plus the loss of staff on the vendor side which supports the product.  Increases in vulnerabilities – Increased numbers of security vulnerability patches or other security weaknesses can be a sign that it’s time to kill an application. The amount of time dedicated to patching, testing the patches, and then rollout to production can quickly swallow entire product teams. Increased patch frequency can delay staff from doing other, more productive work. v where he worked with critical infrastructure protection, inci dent response and technical disaster recovery planning. He transitioned afterward to Deloitte where he served a variety of commercial clients in healthcare, fintech, heavy manufacturing, and shipping groups, consulting for all their resilience needs. Doernhoefer then moved on to work with United Airlines as their disaster recovery professional where he was cited as the BCI Professional of the Year for the Americas. He now serves as the lead infrastructure resilience manager at USAA. Matt Doernhoefer is an experienced tech nical resilience professional with nearly 20 years in a variety of industries. He started his career working with federal government agencies such as DHS, DOD, and DOJ

C

M

Y

CM

MY

CY

CMY

K

24 DISASTER RECOVERY JOURNAL | SPRING 2023

Introducing the new Virtual Corporation.

www.virtual-corp.com

The Human Side of Conducting BIAs By MICHAEL HERRERA I recently had an engagement at a Fortune 500 company where three of our consultants conducted about 100 BIAs. With so many BIAs to conduct, they naturally encountered a few bumps in the road. These included the following: n The session where the leader of the n A business unit which took two sessions to complete its BIA (rather than the usual one) because they brought many people more than the requested number, and every attendee weighed in on virtually every topic. n The episode where a business unit supplied us with data on its current processes and confirmed its accuracy, In each case, our team swiftly regrouped and found a way to obtain the informa tion we needed to successfully carry out our engagement. However, the stories reminded me of a very common misunder standing about BIAs: people tend to think doing one is all about the questionnaire. In fact, conducting a BIA is mostly about working effectively with the people pro viding the information for it. Understanding the human side of the busi ness impact analysis is critical to its success. If you are not sufficiently attentive to this aspect of the job, you run a high risk of having the following things happen: business unit said he already knew their unit was of critical importance to the company and therefore conducting a BIA was a waste of time. The group left the interview without providing any data. then stated – after we had loaded the information into the BIA tool – all of the resulting work was invalid because they had gone through a reorganization. Then they asked us why we hadn’t known about their reorg.

26 DISASTER RECOVERY JOURNAL | SPRING 2023

n Your sources are likely to become disengaged or resentful. n The information they give you will be inaccurate or incomplete. n The BIA based on that information will be divorced from reality. n The recovery plan based on the BIA will be suboptimal to say the least.

n Whose importance they doubt. n Which involves homework. n And which takes them away from the real work they need to get done. If you know this and accept it as natural – and work to help the people get past it – you are going to be on your way to getting better results in your BIAs. The Consequences of Doing It Wrong Nowhere is the expression “garbage in, garbage out” truer than with a BIA. If your participants talk to you simply in the spirit of getting you to go away as soon as possible – instead of digging deep and providing you with quality informa tion – your BIA and any plans based on it will be fatally weak. Moreover, the lackluster quality of the resulting BIA will reflect poorly on the person who conducted it. Many managers have the uncanny abil ity to review a BIA and realize at a glance something is off. Usually when that hap pens, the first person they will turn to in seeking an explanation is the person who conducted the BIA. Tips For Doing It Right So how do you successfully manage the human side of conducting BIAs? Here are some tips, divided among the four phases which make up the process. Phase 1: Preparing yourself for conducting BIAs If you aren’t ready, everything else you do to try to be successful might be in vain. Here are some things you should do to prepare yourself for conducting a BIA: n Know your BIA process in and out. n Define ahead of time how you will conduct the BIA, tailoring your approach to the culture of the organization (e.g., entrepreneurial vs. traditional). n Review each of the participating business units ahead of time; know what they do and who you are interviewing and their personalities. n Build a standard agenda for each BIA to ensure consistency in approach and timing.

A good starting point for obtaining a better outcome is to look at the BIA from the point of view of the participants. For the typical person, participating in the BIA preparation and interview usually means: n More work.

n On a subject they find boring. n Which they don’t understand.

28 DISASTER RECOVERY JOURNAL | SPRING 2023

The Global Leader in Organizational Resilience

BusineSs Continuity/Continuity of Operations information security Critical Environments

Incident Response Crisis Management & Communications

Legal, Audit, & Compliance Organizational Behavior Risk Management Supply Chain Resilience

Financial Health & Visibility Human Resources Management ICT Continuity

we educate. we credential. we lead.

Building Resilient Communities, One Organization at a Time

www.build-resilience.org | info@theICOR.org | 1-866-765-8321

you go so the BIA is essentially complete at the end of the session. n Try to finish in less than the scheduled amount of time. n Provide results as soon as they are available so the group can review them, and you can close out quickly. n Thank the participants for sharing their time and expertise. n If you have many BIAs to conduct, don’t do more than three on a given day. Because of how demanding the process is, it’s unlikely you’ll be at your best if you do more. n Review the data provided in the pre-work to be prepared to guide the discussion effectively and draw out any information not provided. A Collaborative Enterprise Conducting BIAs is a deeply collab orative enterprise. How you handle the human side of the endeavor can make or break your success. By following the tips in this article, you will increase the chances of getting your colleagues in other departments to give you their very best efforts, to the benefit of the BIAs you produce as well as your organization’s overall resiliency and pre paredness. Takeaways n Many people mistakenly assume conducting a BIA is mostly about the questionnaire. The key to success is working effectively with the people providing the information. n Understanding the human side of the BIA is critical to its success. n If your sources disengage, your data and results will suffer. n Try to understand what the BIA experience is like for those providing the information. v Michael Herrera is the CEO of MHA Consulting, a leading business continu ity planning and information technology consulting firm. Herrera is the founder of BCMMetrics, which specializes in business continuity software designed to aid organizations in devel oping and executing business continuity programs.

n Streamline the BIA process through pre-work, leveraging previous data, and focusing on the most critical processes. n Prepare mentally and physically to facilitate each interview. Phase 2: Creating the BIA packet The packet that goes out to the par ticipants announcing the BIA is your first chance to start building a good relation ship with them. Here’s how to make the most of it: n Write the cover letter in partnership with your sponsor in senior management; send it out over the sponsor’s signature. n In the letter, talk about the importance of business continuity and the BIA and what they mean to the company. n Also in the letter, ask the participants to do a modest amount of pre-work (to stimulate their thinking and speed up the eventual interview). n Specify the due date of the pre-work as well as the date, time, and location of the BIA meeting and who within the business unit should attend. Phase 3: Preparing the room The basics are critical when you are gathering together a bunch of people who would most likely rather be somewhere else, especially if you expect them to give you their best effort for two or three hours on a less-than-fun subject. In preparing the conference room, you should make sure of the following: n The temperature is comfortable. n You have enough space and chairs. n You have coffee and water for everyone. n There is a place up front where you can stand. n You have your BIA software up and running. n You have worked out any kinks in the projection of your computer screen onto the wall behind you. Phase 4: Facilitating the discussion This iswhere things get interesting. There are many types of people at a large com pany, and you have to be flexible enough to work productively with all of them. When leading a BIAmeeting, you should:

n Get a good night’s sleep. Facilitating a BIA meeting is very draining mentally. You will have to talk a great deal, and it will be challenging to get people to the right answers. n Dress the part. You are there to lead the meeting. Look like a leader. n Be enthusiastic and fun in how you present yourself; it’s not the most exciting subject. n Stay focused and on track. Your interviewees will appreciate a crisp and clean delivery of the subject matter. n Adapt your style to the personality of the interviewees. People often reflect the style of the field and company, as well as their role in the firm. Marketing and human resources people will probably come in smiling and joking; nurses are fun to interview, but doctors can be difficult; and engineers and actuaries respond best to a cool, businesslike approach. n Be energetic enough to keep a rein on the extroverts and patient enough to draw out the introverts. n Don’t assume the participants have read everything you sent them, or they completed the prework. n Don’t be shocked if you encounter people who have worked at the organization 10 or 20 years but do not know the basic information needed to complete the BIA. It happens. n If possible, have a scribe who can record what is said and decided so you can focus on leading the discussion. n Be prepared for the likelihood people will pay more attention to their phones and laptops than to you. n Many people tend to overstate the criticality of activities they are involved in. Make sure you push for facts which will illuminate the truth of the situation. n Some people will be impatient, and others will overanalyze every number. Keep things moving and stay focused on what is important. n The flow of the meeting should follow the screens of your BIA tool or the sections of your spreadsheet. Put in live data as

30 DISASTER RECOVERY JOURNAL | SPRING 2023