FuzzyLogicBits

Business Continuity Planning

In the age of hyper connected, digital online multi-factor authenticated world along with the increased need to have adequate safety both physical and virtual, there is a tendency to overlook a much needed practice of Business Continuity Planning. I am going to try to touch on a few aspects of this, share my thoughts, and practices to serve as a reminder for myself and my business owner peers.

It is my observation that every business large or small has a key person or persons, that are responsible for making the business work like a well-oiled machine. If this person or persons are suddenly taken out of the picture, then there is a very high chance that business can come to a sudden grinding halt.

The amount of pain and effort involved in getting the business back on track working at full speed will greatly depend on how well the Business Continuity Planning was done.

Let’s break down the planning and execution of this process.

The first part of this planning process always starts out with identifying who the Key Person(s) / the Key Roles and the Key Systems which make the business operate smoothly every day.

Most businesses have three or four Areas of Operation.

1. Customer Service.
   • Technical
   • Non-Technical
2. Accounting –
   • Back Office Functions
   • Human Resources
   • Business Management
3. Technical / Infrastructure –
   • Tier 1
   • Tier 2
4. Any Other Area of Importance For The Business –

Key Systems

1. Password Management
2. Billing & Collections
3. Accounting / Payables
4. Customer and Vendor Records Management
5. Network Infrastructure Documentation
6. Contracts Repository
7. HR Document Repository
8. Phone System / Digital World Presence

Let’s discuss the relevant details of each of the previously listed items –

Customer Service –

This is the lifeblood of any business.  Having a centralized system where customer records are kept, updated, and available to all the customer service staff with expedient ease is a must!  

 Should this be an on-premise computer system? Should it be a Cloud Hosted system? What is the Disaster Recovery (DR) plan to revive such a system in case of a primary system failure? What is the expected downtime for system recovery? What is the sustainable down time before quantifiable implications to business revenues? Who has the ‘keys’ for the system? Who is responsible for making sure it is always working? Who is responsible for back-up?  Who is the secondary person responsible? Are the key recovery processes documented? Do you have a list of Contacts/Vendors/System Providers handy?

All the above are valid questions that need to be asked. There is no one answer that would fit all. Every business should ponder and evaluate what is the best solution for their needs, which would provide them with the required balance.

I can only answer what we do for our business, and it is important to do periodic evaluations to determine if the process / system / documented procedures need to be updated.

We currently use Freeside for our Customer Service / Billing package, which is hosted on a virtual machine (VM), running on a Proxmox Cluster in one of the Data Centers that is part of our network infrastructure. We back-up the virtual machine daily, to an external storage system, and we keep 5 days of back-ups. In case of problems, our Senior Technicians can restore the VM. The expected time to restore is about 1 hour, which can be done remotely. We find that we can deal with the pain of approximately ½ to 1 day of system down time before the financial impact becomes measurable. This is primarily because 80% of technical information can be re-constructed / looked up from other systems. (Technical Documentation and Monitoring System.)

Accounting –

The accounting system acts as the lungs of a business, such that it provides required oxygen while cycling out the carbon dioxide. The system makes sure that monies are flowing in a timely consistent manner by way of both the incoming monies (i.e. receivables) as well as the outgoing monies (i.e. payables). All of the questions that were raised previously about a Customer Service System are applicable,  as well as several more specific ones related to Human Resources and Business Management.

Should this be an on-premise computer system? Should it be a Cloud Hosted system? What is the DR plan to revive such a system in case of a primary system failure?  What is the expected downtime for system recovery?  What is the sustainable downtime before quantifiable implications to business revenues?  Who has the ‘keys’ for the system? Who is responsible for making sure it is always working?  Who is responsible for back-up?  Who is the secondary person   responsible?  Are the key recovery processes documented? Do you have a list of Contacts/Vendors/System Providers handy?

 Who are the people who have access to the system and the information contained within it? Who has access to the bank accounts? Who has the authority to maintain them? Are there any multi-factor authentication mechanisms required? If so, what are they tied to (e-mail, cell phone number, company telephone number, etc.)? Are important account numbers, and contact numbers documented in an alternate safe place?

 Are Human Resource documents, employee information, employee benefits, contracts, and any other related resource kept in a secure repository?  Who has access to them? Who has the authority over access to those systems?

Business management in a small business is typically done by the business owner(s), which may include family members. Larger businesses may have a team approach where there may be multiple partners or division managers.  Most of the business management functions tend to be geared around business development, planning, and project management. These are highly subjective activities where it may not be easy to replace a person. It can also be very expensive to find a suitable replacement for such key functions.  Documentation and keeping records of plans, meetings, projects, and progress are a crucial requirement for business continuity.  What would be left is the need to manage the associated business risk with such an event. This is often done with an ppropriate type of insurance policy. For example, life insurance policy on partners for the purpose of a buy-out of their shares from their families, disability insurance, and key man replacement polices should be given serious considerations.

For back office accounting, we are using on-premises QuickBooks. The data file back-up is done on a regular basis automatically to both in-house storage as well as cloud storage.  I am certain this is something that would be changed over to cloud based services on the next evaluation.  We carry life insurance policy on partners for buy-out purposes and disability insurance has been an ongoing discussion without a final decision. We do provide company health insurance.

Technical Infrastructure –

Technical infrastructure acts as the limbs of our business. For this document, I am only going to address the administrative aspect of business continuity planning as related to technical infrastructure. We will take a deeper dive into the pure technical systems continuity planning on another date.

 Over the years, as we have grown to providing more and more sophisticated services to our clients, the amount of information that must be documented and tracked has grown exponentially.  We started out using paper and pencil to keep track of this but outgrew it. We moved on to keeping this in Word documents and Excel spreadsheets and outgrew them. We tried to keep them in a database and found those systems also fall short very quickly.  This area has been a constant challenge to keep on top of.

We believe we have found a system that can keep track of all kinds of information, technical details, non-technical details, other attributes, documents, pictures, along with the defined relationships of these different items of information. The key feature of this system is the ability to retrieve information in an ad-hoc manner and be able to follow / view the other related items. These systems also allow us to manage and assign appropriate access level permissions.

I-DoIT is our infrastructure documentation system. It keeps track of all kinds of information, system, cables, ports, connections, IP addresses, Locations, Lock Key combinations, Circuit ID’s, X-Connect ID’s, any and all details that are necessary to keep track of related to network infrastructure. This is a critical system which does not have a huge impact on day-to-day business, but it is the core repository of our network information. Loss of such a stem would be equivalent to the business losing major limbs!

We use CheckMK as our primary infrastructure monitoring system.  We also use secondary systems, such as Syslog Server, Oxidized, Radius Server, Unifi, AirControl, Mimosa Cloud, Ignitenet Cloud for managing manufacture specific items.  Each of them is running on their own virtual machine on the Proxmox Cluster at the Data Center.  We find the loss of such a monitoring system to be very uncomfortable making the entire technical staff feel blind. We don’t like to have this system down for more than a couple of hours maximum. Losing this system would have a huge impact on productivity and the staff’s ability to resolve network issues expeditiously.

Additionally, there are about 20 to 30 external vendor portals and systems that are required for ordering, processing, and provisioning services for customers. The access and authorized personnel need to be documented and kept track of.  

Passwords, Passwords, Passwords! –

Starting from my teenage years, as I grew older and gained more responsibilities, I found myself gaining more and more keys to keep track of. I know carrying a bunch of keys is a PITA. However, I found what worked for me was to carry a large keychain and clip it to my jeans belt hook. It was only a matter time before folks could tell my presence from afar by the sound of my keys jingling. Periodically, I would have to remove keys from the keychain. Unsurprisingly there were keys that I could not remember which lock they belonged to, and I would also came across locks that I should have had the key to but could not find the key for them.

The challenge of managing many physical keys pale in comparison to the challenges of managing passwords for the many different online systems of today.  

Let’s face it, any method other than an automated cloud based double encrypted system for storing passwords simply DOES NOT WORK! Especially when one considers the need to handle multi-factor authentication, key codes, seed keys, backdoor keys, strong passwords, the need to share them with staff / employees, as well as the need to change them on the fly…. No static, manual system of keeping such information is adequate.

There are many systems of keeping track of work, personal, and customer credentials. After using and outgrowing several them, these days we are staring to use PassPortal. It is not an inexpensive solution, but it offers a combination of features and reporting capabilities, which in my opinion are a requirement for the present and future!

Data Repository –

Where is your key data files being kept?  Are they in a physical server on-premise? Are they in the Cloud? Are they on a workstation or laptop?  All of these have drawbacks, what if the machine holding the data is lost?  What if the Cloud data is not available or lost? … Just because it is on the Cloud it does not mean it is 100% safe.

We believe that there should be a primary system [location of data storage], secondary system (i.e. back-up), and a Tertiary system (i.e. back-up of a back-up).  Managing such can become a challenge.  We are staring to use MorroData CloudNAS , which is a smart on-premise NAS cache device, with auto-sync of data to AmazonS3 or Wasabi, in conjunction with Microsoft Office365 SharePoint.  All of this is automatic and seamless, which does not require a lot of babysitting.  Incidentally CloudNAS also has a very nice versioning feature which provides a hedge against Ransomware infection.

Phone System and Digital World Presence –

This is another set of vital systems for business continuity. As we are learning today, a Hosted PBX with the right set of features can provide the ultimate flexibility because it is not tied to any physical location. However, one should plan on how to handle potential outages even if they are very few. The ability to adapt to a changed working environment was a bonus feature in the past but nowadays is a must-have requirement.

Other items related to Digital World presence are, e-mail, website, DNS registrar account, and company social media accounts.  One should be attentive to who has access to these and more importantly who has the authority to make changes and recover lost passwords. It is very important that the access and authority to these systems be set-up to be a Company Role as opposed to an individual.  Over the years, I have seen too many situations where folks have lost access with little recourse due to the authority of record of these being an individual.

In summary, it is very important for every business owner to take some time and give serious thought and complete consideration to the information above as well as try to address them to the needs of their business. This is not a one-time process. It should be periodically reevaluated, written down for the sake of consistency, and should be actively discussed amongst the key stake holders of a business.  Systems and processes should be evaluated based on suitability and changed when they are inadequate. In this regard, there is nothing more harmful than contorting a system or process to make it fit when in reality it has outlived its usefulness.  Furthermore, it is very important to have a buy-in in adopting the new system / processes by all the parties involved, and often the biggest challenge is in getting the die-hards to change and embrace the changes. Please consider this article as a thought provoking starting point. However, by no means is it meant to be a comprehensive guide. I have shared what our business does, some of what is working for us, and some that we also should be reevaluating to change for improvement.

I would love to see feedback and suggestions on what you all are using and what is working for your business.