This post is part of a blog series recapping highlights from our webinar series, where industry experts share their knowledge and field lessons from utilities building their NERC CIP-015 internal network security monitoring (INSM) programs.
In the eighth episode, we turned to audit readiness and, specifically, running a mock audit. I was joined by Steve Parker and Leonard Chamberlin, managing partners with Archer Energy Solutions. Steve has implemented the NERC CIP at a large utility and was briefly a NERC CIP auditor in the WECC region. Leonard worked for a large utility before spending five years at the Federal Energy Regulatory Commission (FERC).
Designing the Mock NERC CIP-015 Audit: Independence Is Critical
Above all, the person designing or running a mock audit can’t be the person who designed the CIP program or is responsible for it. Independence is critical; otherwise, you ‘re essentially auditing yourself. In a larger organization, there’s typically an internal audit team separated enough from the work to look at it objectively. But if no one on that team has ever been an auditor for a regional entity, NERC or FERC, they’ll lack the mindset an auditor brings. That manifests not just in expectations for evidence, but in the way they ask questions, which are often designed to uncover gaps. A qualified third party can get you closer to an actual audit, and their outside perspective can overcome internal politics.
An auditor is going to be very interested in how you determined which points in your network needed to be monitored. That decision determines everything else in CIP-015.
Get Inside the Head of the NERC CIP Auditor
Audits are stressful, and more so if you haven’t prepared. Conducting self-assessments including reviewing evidence, activities and processes, can only get you so far. Conducting a mock audit is the next step in the evolution of assurance, but not all mock audits are the same. While an independent person or group conducting the mock audit is the most critical element, understanding how audits are run in your region also matters.
Auditors are people, and the best mock auditors think like a real auditor and understand where they’ll be coming from. Talking with the people who were involved in recent audits can be a great way to gain an understanding of how they operate in your region. FERC wants a consistent auditing approach across all of the regions, but in practice the experience varies drastically even within a region.
Auditors have auditing standards they must abide by, and one of them is competence. The audit team is supposed to have the expertise to evaluate what it is looking at. With newer technologies, that’s not always the case. As uncomfortable as that may be, you need to be prepared to call that out.
Documentation Review: Great Ready to Dig In
Documentation review is one of six techniques that auditors use to test your evidence, and you want your documentation to stand up under scrutiny. Treating it as an investigation is key to uncovering things before an auditor but it needs to cover more than just your CIP-015 documentation because that’s not how audits work.
Two standards to rely on for your mock NERC CIP-015 audit: CIP-002 BES Cyber System Categorization and CIP-005 Electronic Security Perimeter(s) (ESPs). CIP-002 determines what’s in scope from a device and system perspective, and CIP-005 is where you draw the protections around it. For CIP-015-1 the ESP is most critical, although that will change with CIP-015-2 when we expand to cover EACMS and PACS.
From the auditor’s perspective, CIP-002 and CIP-005 help them understand:
- What’s the scope?
- What am I looking at?
- What am I evaluating?
A mock auditor wants to confirm you’re aware of every network in scope and confident your INSM program covers it.
Your Risk-Based Rationale for Date Feed Placement: The First Domino
R1 Part 1.1 of the standard addresses where you take in data feeds. This can be tricky because there's a lot of variation here. Focus on the requirement language, because that’s what you’ll be held to; anything else is guidance and is not legally enforceable.
In brief, the requirement is to implement, using a risk-based rationale, network data feeds to monitor network activity for the ESPs. It’s not clear exactly what’s meant by “risk-based rationale.” If you have a sense of déjà vu, you have been around for a while and remember the issues we ran into pre version 5. Not everyone approached that the same way, and some looked for opportunities to minimize their compliance risk. So, an auditor is going to be very interested in how you determined which points in your network needed to be monitored. That decision determines everything else in CIP-015. The rest of the requirements fall into place like dominos.
Simulate a Real Audit with Tough Questions
Unlike other standards, CIP-015-1 wasn’t written with a highly prescriptive, objectively measurable requirement. It gives you an objective and a purpose, but with a lot of flexibility. During documentation review, the mock audit is a chance to practice explaining the decisions you made:
- What risk rationale did you use?
- Why did you use it?
- Why do you believe it’s appropriate?
- What is an anomaly?
- How are you detecting anomalies?
- How are you evaluating detected anomalies?
- What are you retaining and for how long?
- How do you know you’re not missing
That comes down to professional judgment, which leaves room for persuasion. You don’t want to be winging it for the first time when the real auditors show up. Use the mock audit as an opportunity to put subject matter experts (SMEs) in the hot seat and have the auditor poke holes in the rationale. Just as attorneys prepare witnesses, making your SMEs a bit uncomfortable helps them prepare for tough questions they may face in the real audit.
On a related note, if a term is not well defined by the NERC Glossary of Terms or in the standard, as the regulated entity it behooves you to document what the term means to your organization. If you don’t, the auditors will tell you their opinion, and if those definitions don’t align, it makes for a very interesting conversation. The running example in this series is “anomaly.” You don’t want an auditor to define anomaly for you. Making sure your SMEs align to the definitions defined in your program is important.
R1 Part 1.3: Anomaly Evaluation, Playbooks and Escalations Process
INSM generates massive amount of network data from network activity. To meet R1 Part 1, you must have a documented method for evaluating anomalies and taking further actions. At audit time, you may be asked to demonstrate an anomaly evaluation: what were the results, and how did you determine the further action? In reality, utilities will be evaluating anomalies in huge data sets. That opens the door for incorrect evaluation or for things to happen in bulk because of the batch.
This is where the expertise of the people doing the evaluations is important at audit. Somebody can look at a flagged anomaly without understanding the underlying system or protocols well enough to recognize that something is amiss and wave it off as not malicious when in fact, it’s a subtle indication of a sophisticated attacker. There needs to be a formal process, not an ad hoc hand-off to somebody who’s busy doing other things.
This is where we’re likely to see findings in audits that you want to expose during the mock audit. You might have an evaluation method in place, but maybe it’s not be very good at determining what those further actions should be. Because of the vague language in the requirement, the finding might not result in a Potential Noncompliance (PNC), or at least not one that would stick. But if you want to avoid those uncomfortable exchanges altogether, make sure the people evaluating anomalies day-to-day know what they’re looking at and how to defend the process during an audit.
R2 and R3: Store the Data, Protect the Data
The other two requirements in NERC CIP-015-01 are boring in comparison to R1. Store the data, protect the data. We have really good guidance on data protection of BES Cyber System Information (BCSI) data protection in CIP-004-7 and CIP-011-3, so align your program with that guidance as much as you can. Write what you are going to do, then have evidence showing you’re doing.
Notably, none of the requirement language references timeframes, even though FERC was clear that it expects this to be an expeditious process. If an anomaly takes three months to close out, will an auditor raise it? Absolutely. Can they enforce it? Those are two separate questions. The loophole is not on the detection side, which is automated and nearly instantaneous. It’s not a question of fast the rules fire, because they’re packet rules, and initial triage usually follows within seconds. It’s the evaluation piece in Part 1.3 where you may see auditors push back on excessive timeframes. If you find an instance during your mock audit, this is a great place to challenge the team.
Bottom line, look at what the objective wants you to accomplish. Anytime they're telling you to do something and those measurable components are missing, look at what's reasonable from a qualitative and effectiveness perspective. If you're looking at monitoring a network for anomalies with the objective of detecting attacks and looking into them, those timeframes have to be relatively short.
Sampling: Trust but Verify
Sampling is a technique that’s been used heavily on other standards. For CIP-015, the auditor may ask for a percentage of the alerts that fired in your environment in, say, the last three years and look at what you did with them to verify that you’re doing Part 1.2 (detection) and Part 1.3 (evaluation) correctly.
They’re also going to look at your network diagrams for data feed validation, and a technical auditor may want to look at actual network traffic. That will evolve as the standard is audited and auditors start to see what different entities have implemented, what they think works well or doesn’t.
Best Practices and Pitfalls
Guidance for a successful NERC CIP mock audit isn’t unique to this standard, but many utilities still need coaching to make them less painful. All of the recommendations in the table below are important, but a few are worth calling out.
Using an independent facilitator who has actual experience as a NERC CIP auditor is invaluable, for all the reasons already stated. Documenting your rationales is a close second. This allows you to take the high ground and force the auditors to challenge vs. the other way around.
As for pitfalls, don’t over rely on guidance. It’s useful but not legally binding, and auditors don’t have to abide by it. That includes this webinar and blog series, which should not be your primary source of information. We’ve been pulling in top industry experts with real-world experience and researching each topic to ensure the information we present is as accurate as possible, but don’t let this series be your primary source of information become an expert and join the community.
Finally, be proactive in your prep but conservative with submission. Prepare more evidence than you think you need but keep most of it in your back pocket and provide just enough to demonstrate compliance. Sharing too much upfront generates additional questions and takes you down rabbit holes. If the auditors want more, they will ask for it.
Final Thoughts on Acing NERC CIP Mock and Actual Audits
A mock audit is the surest way to find out whether your CIP-015 program can survive contact with an auditor. Use an independent mock auditor, treat your risk-based rationale as the domino that sets up everything else, and put your SMEs in the hot seat while the stakes are low.
This post only scratches the surface of what was covered in the webinar. Register for the webinar series and watch the Episode 8 replay for the full discussion. And if you’d like to talk through how any of this applies to your own environment, reach out. We’re here to support you.
