How to Integrate NERC’s Requirements in an Ongoing Automation and Integration Project Framework Jacques Benoit, Cooper Power Systems Inc., Energy Automations Solutions - Cybectec Robert O’Reilly, Cooper Power Systems Inc. Energy Automations Solutions - Cybectec Abstract This paper addresses the challenges faced by utilities and/or integration companies during deployment and engineering phases of automation and integration projects, with regards to complying with the new cyber-security requirements set out by NERC. This paper will focus on approaches to these new challenges to ensure the project stays within schedule and budget, from the point of view of substation requirements, management and of the different SCADA systems. Introduction This technical paper will discuss the challenges of minimizing the impact of adding NERC CIP compliance to an ongoing project consisting of updating a substation’s automation systems. Originally aimed at providing faster access to a higher amount of operational and non-operation data within a substation framework, the changeover is an opportunity to upgrade some of the protection and metering devices. But now, the project must also include compliance with cyber-security requirements. While at first glance NERC requirements may seem to be an insurmountable task, when one takes a closer look at the standards, it becomes obvious that proper planning and best practices are the key to accomplishing compliance. Moreover, proper planning will minimize the impact of NERC CIP compliance on the project’s budget and timeline. From a project implementation point of view, NERC CIP mainly describes what is required from utilities, but does not provide any technical information on how to implement a project to meet those requirements. This leaves a lot of room for interpretation and implementation. From a project viewpoint, one must decide quickly which requirements would normally be addressed outside of a project scope and hence would not impact adversely its timeline or budget. Since they should be the responsibility of other groups within the organization, we will not discuss the following CIP standards in this paper: CIP-001 CIP-008 CIP-009 Sabotage Reporting Incident reporting and Response Planning Recovery Plans for Critical Cyber Assets Instead, we will discuss how the following key CIP requirements have a direct impact on your ongoing project and should be addressed in any ongoing project: CIP Reference # CIP-002-R3 CIP-003-R4.1 CIP-003-R5.1 CIP-003-R6 CIP-004-R2 CIP-004-R3 CIP-004-R4 CIP-005-R2 CIP-006-R2 CIP-007-R1 CIP-007-R2 CIP-007-R3 CIP-007-R4 CIP-007-R5 General Description Critical cyber asset identification Critical cyber asset information to be protected (items defined by management team) Access control (personnel cleared to access protected information) Change control and configuration management Training of all personnel (operation, technical, contractors, etc.) Personnel risk assessment Personnel access to critical cyber assets Electronic access controls (ensure electronic access is only permitted to approved personnel) Physical access controls Test procedures (supplied by others) Ports and Services (ensure only the required ports and services are active, all others are turned off) Security patch management Malicious software prevention Account management For the readers’ convenience, we have summarized the different CIP requirements in the appendix. The Original Project The example chosen is that of modernizing an existing substation automation system. This type of project was selected because it is probably the worst case: not only is new equipment added, but legacy equipment is also kept in the substation. The implementation must be done while keeping legacy systems in operation. Moreover, the project must allow for compliance with all applicable CIP requirements, and be able to pass a compliance audit near the end of the project. We will use the example of a typical legacy substation which has been in operation for more than fifteen years and undergone normal additions required by increased client demand. In most cases, such substations would resemble the following diagram. Fig. 1 - Existing substation automation before project The first order of business when moving an ongoing project towards NERC compliance is to plan for the substation’s auditability As the project engineer, one must keep in mind that the plan must be approved by the company’s NERC committee, and that deadlines and budgets are not expected to be impacted. A Review of the Project Usually, projects are planned and budgeted with preliminary engineering performed more than 18 months before actual implementation. This delay can create an issue relating to equipment and software costs, as well as delivery lead times. The first project review for CIP compliance will require the retrieval of all information on the previously selected components for the project. Then all potential critical cyber-assets will need to be documented. The list will finally be reviewed to ensure that the security requirements can be met with the equipment that had been originally selected. The initial substation planned architecture is presented below: Fig. 2 - Automation overview diagram of planned project It is important to review any potential new features of the equipment that had originally been selected in the planning stages of the project. Quite often, the product contains new features/capabilities that will in the end save time in the detailed engineering and commissioning phases. Hence, although one may feel the operation is time-consuming at first, it will probably save time by the end of the project. What remains to be examined is the increased paperwork and preliminary audit added to the factory acceptance test (FAT). The risk assessment portion could be performed during the audit and FAT. The project should be executed with a best practice approach which should bring the risk within a manageable context. Establishing the Security Perimeter The connection between the substation and the corporate WAN had been planned using a router and firewall. This setup had been approved by the IT group. In view of the NERC CIP requirements for an electronic perimeter, this configuration can no longer be considered adequate. For instance, this device does not meet the access control and logging requirements. Most substations also contain older devices such as power meters and DFRs with limited communications capabilities. These devices require some form of protocol converter. Also, in addition to the main access points, the EMS group requires the use of a dial-up connection for remote access to the metering equipment. Dial-up access is flagged as a major potential security risk by NERC CIP standards. Now it is clear to the engineering team that using only a router will not comply with NERC CIP’s required electronic security perimeter. One might recommend a gateway device in addition to the router. Gateway devices usually provide secure communications capabilities using modem connections, serial, and TCP/IP. They create a single point of access to the substation making it easier to secure the electronic perimeter. Although they will vary form vendor to vendor, these gateway devices usually also provide an additional firewall and security features. Isolating the substation’s critical assets and physically installing them in strategic and secure locations within the substation also helps to meet the CIP physical perimeter security requirements. Equipment Inventory Once the electronic security perimeter has been defined, the inventory of equipment must be established and documented. Although this seems a difficult task on the onset, it is more easily prepared than one might think. All the information required is already available so that equipment data is brought back to the central systems (be it SCADA, EMS, Asset Management, or others) via the intelligent gateway. Designing how information is to move from substation to control center will also help define what information is more important. During this phase it is recommended to have short and to the point brainstorming sessions with the different groups wishing to have access and to have them document their requirements. One might be surprised how demands are reduced when written versions are required. Once this information has been identified, the intelligent gateway can be used to limit access to this information. Access levels and user groups should be used to only allow specified systems and users read or write access. Any other system should not be allowed to retrieve/operate on the information. For information which is made available via the intelligent gateway; the unit’s security environment should be configured to let only the specified computer system(s) access the specified and approved information. This information should be documented for future auditing requirements. Access Control, Personnel Risk Assessment, Access to Cyber Assets and Account Management Before NERC CIP standards, these points were not normally part of a project. However, CIP standards make their assessment and documentation mandatory. Fortunately, help usually can be found in other groups within one’s organization. Human resources and senior management can define access levels and the personnel who will have them, as well as perform the personnel risk assessment. This should not impact the project’s budget. Only documentation of those accesses would remain to be produced. One can use a central security server or the intelligent gateway’s security features to manage accounts. Obviously, central account management is much more efficient in providing comprehensive authentication and simplifies meeting the NERC requirement of being able to remove access rights rapidly. Central user management may however require new servers and software, which would normally be expensed from the IT budget. Change Control Although change control and configuration management may seem new, most project managers who have been through a number of projects understand this as the mandatory documentation process to control risk during an automation upgrade project. Hence it is usually planned in the original weekly review list. At this point already seven items of your CIP requirements list have been addressed or planned for: CIP-002-R3 CIP-003-R4.1 CIP-003-R5.1 CIP-003-R6 CIP-004-R3 CIP-004-R4 CIP-005-R2 CIP-007-R5 Critical cyber asset identification Critical cyber asset information to be protected Access control Change control and configuration management Personnel risk assessment Personnel access to critical cyber assets Electronic access controls Account management So far, there was very little impact on budget or timelines, except for delays regarding reviews of personnel risk and their security clearance. However, this requirement is usually the responsibility of human resources for personnel and of the purchasing group for the contractors. Security Patch Management and Malicious Software Prevention The manufacturer of the gateway device will usually provide the tools to properly handle any patch management and prevent malicious software. Many techniques exist and it is not in the scope of this paper to decide which approach is better for this facet of the CIP requirements. Suffice it to mention that today, tools and equipment are available for this purpose. However, it is still up to the project team to validate that these tools will perform as required by the project and corporation. Test Procedures and Port Blocking During the final engineering phase, one should prepare a framework of the testing methodologies that could be required to validate the new automation system and its integration into the current substations operations. This is usually done with the help of the vendor or the integrator. When dealing with a change/addition to an existing substation, careful planning must performed to ensure that the system will interface and react properly and promptly to the substation operation requirements. This detailed testing phase is the most appropriate time to check that all of the ports and services not required by applications are turned off. This can be done remotely, since it is easy to forget this type of work during commissioning of the systems at the substation. The IT group can provide the tools required for these tests. However, vulnerability testing should not be performed on a live system as it may render it inoperable. If possible, one should change the default ports. For example, DNP3 via TCP/IP uses port 20000 by default. With newer systems and applications, this can be changed, hence preventing anybody from accessing your system by trying to ping the standard ports. Fig. 3 – Overview drawing of the final concept for the new automation systems Personnel Training Training is usually the last item on a project’s list. In today’s complex operational environment it should not be neglected. Personnel training has always been a priority for most organizations and is planned and budgeted accordingly. NERC CIP standards simply require more detailed documentation regarding training sessions, attendees and the personnel’s ability to react appropriately in different situations. Training should be a requirement from all vendors providing the software/hardware for the project. The training should include detailed hands-on lessons with the applications, hardware and general software. Security Software Security software should be chosen together with the IT group and should provide a centralized approach, where it is easier to manage access rights and users, data logging, intrusion monitoring and system health monitoring. Local security should also be implemented in the substation, for onsite personnel. Local security must provide the capability of being integrated into the centralized approach to simplify overall user and application management but also to provide the capability for the security to be available at the local level when connection(s) to the centralized systems is not available. Conclusion Proper planning is the key to minimizing the impact of NERC CIP standards on a project’s timeline and budget. Individual steps towards NERC CIP compliance are not complex: they simply require a little more effort on the documentation and planning sides. When one has experience with retrofit projects, proper documentation and training become a life-saver at project’s end. Appendix: CIP Standard Solutions Breakdown Requirement Description CIP-002-R3 Critical cyber asset identification CIP-003-R4.1 Critical cyber asset information to be protected CIP-003-R5.1 Access control CIP-003-R6 CIP-004-R2 CIP-004-R3 CIP-004-R4 CIP-005-R2 CIP-006-R2 CIP-007-R1 CIP-007-R2 CIP-007-R3 CIP-007-R4 CIP-007-R5 Solution Reuse project inventory Review with different groups requiring access to information Seek access models from upper management, use centralized authentication model Change control and Reuse project change configuration management management infrastructure Training of all personnel Improve documentation Personnel risk assessment Human Resources and Purchasing to conduct assessments Personnel access to critical Seek access models from upper cyber assets management, use centralized authentication model Electronic access controls Use centralized authentication model Physical access controls Install card reader or video camera Test procedures Use exhaustive FAT procedures Ports and Services Reassign ports when possible, use intelligent gateway to restrict access Security Patch Management Use intelligent gateway with security patch management feature built-in Malicious software Use intelligent gateway with prevention malicious software prevention feature built-in Account management Use centralized authentication model