By Gregory Hale
By far, the water industry is the most important sector in the world in part because it keeps people alive by providing access to clean drinking water, it provides access to water for cleaning and it keeps industries running.
But then why is it this vital global asset a disaster just waiting to happen when it comes to protections against intrusions?
As the world is observing right now, the water industry is suffering from droughts in places like the West Coast of the United States in areas like Lake Mead, and the Colorado River, and in Europe with the Danube and Rhine rivers to name a few. On top of that, most of these valuable water sources supply millions of residents with this vital resource.
- UK Water Utility Hit in Cyberattack
- Cisco Suffers Attack, Data Stolen
- Foxconn Recovering from Ransomware Attack, Again
- Chip Maker Hit in Cyber Attack, ‘Holding Production’
So, after the attack on South Staffordshire PLC,’s South Staffs Water and Cambridge Water in the UK by the Clop ransomware group and on the heels of the attack on the Oldsmar, Florida water department last year where an attacker remotely accessed a computer for the city’s water treatment system and briefly increased the amount of the potentially fatal sodium hydroxide by a factor of more than 100, the questions still remain: When it comes to cybersecurity, what can understaffed and underfunded water companies do the protect themselves? And what can they do or where could they go for help?
Least Funded for Cyber
“Water treatment facilities probably make up the most significant amount of critical infrastructure due to their sheer number,” said Dewan Chowdhury, chief executive and founder of security provider, malcrawler. “The local municipality manages most water treatment facilities, and America has them in the thousands. Even though they’re the most in numbers, they are probably the least funded for cybersecurity. They are understaffed and use a technology stack meant to last for 20 years, making it challenging to leverage modern cybersecurity functions to help secure them. It’s just the nature of critical infrastructure where the return on investment may be years.”
“I completely agree the water utility sector is weak when it comes to cybersecurity,” said Eric Byres, chief technology officer at software bill of material (SBoM) provider, aDolus. “Small utilities often struggle to deploy modern operations and safety management processes in multiple areas; being small, ratepayer funded operators means they are always scrambling for dollars. Any improvement or process that isn’t urgently needed to keep water flowing today tends to get deferred to a future budget.
“To make matters worse, small utilities tend to be chronically understaffed. I’d be surprised if many even have an in-house cybersecurity expert. So, this creates an interesting dilemma: There are excellent websites, training programs and conferences where water utility staff can learn how to create secure OT environments, but, if no one at the utility is focused on security, will they even know where to look for guidance? For example, the Water ISAC is a great but underutilized resource for the sector. Similarly, I could describe a simple water utility security program in ISSSource right now, but I’d bet most water utilities that need the guidance don’t have anyone following this website.”
It has been established securing the water industry is key, but where do water providers need a place to go to shore up their security
“The water sector is an industry in most need of government partnership,” said Mark Carrigan, senior vice president at Hexagon PPM. “I believe all can agree that safe, reliable water supply is in the national interest of virtually all nations. In many cases (such as the USA), water supply is dominated by small local entities that are managed by municipal governments, often contracting with a private company for operations. These entities operate on tight budgets that currently do not have the means to effectively secure their operating systems.
“Government, whether it be at the state or national level, needs to provide more resources and funding to help these entities secure their assets. These entities often do not have the ability to raise additional funds for security due to local regulations, operating contracts, or their inability to determine tax or user fee policies. The government must provide a mechanism for these entities to secure funding for security which could include issuing additional bonds, adding fees to consumers, or devoting tax resources. Without clear, strategic direction from government the water sector is likely to remain one of our must vulnerable critical industries,” Carrigan said.
In January, the Industrial Control Systems (ICS) Cybersecurity Initiative launched by the Biden administration to extend to the water sector with an action plan to take place over the next 100 days, the White House said.
The Water Sector Action plan outlines surge actions to take place over the next 100 days in an effort to improve the sector’s cybersecurity profile. The action plan was developed in partnership with the Environmental Protection Agency (EPA), the Cybersecurity and Infrastructure Security Agency (CISA), and the Water Sector Coordinating Council (WSCC).
The plan will assist owners and operators with deploying technology that will monitor their systems and provide near real-time situational awareness and warnings. The plan will also allow for rapidly sharing relevant cybersecurity information with the government and other stakeholders, which will improve the sector’s ability to detect malicious activity.
Non Cyber Answer
There are other avenues water companies could take to help ensure their product stays safe.
“Large water systems should have budget to do ‘proper’ cybersecurity,” said Andrew Ginter, vice president industrial security at Waterfall Security Solutions. “If you serve a half million homes with even as little as $50/month in water bills, we’re talking an annual budget of $300M, before even looking at industrial customers.
“Smaller water systems are more challenged. If you serve a thousand people and half that many homes, we’re talking an annual budget of $600K, most of which goes into people in trucks,” he said.
To that end, he is working with providers by putting together advice for small operators where it will “focus on deterministic, engineering solutions, rather than cyber solutions. For example, there should be physically no path of pipes or valves from any untreated water source to get through into the treated water distribution system. The final stage of filtration should include electro-mechanical (unhackable) safeguards to make sure filters are not torn or otherwise compromised. Additive (chlorine, fluoride, NOH – added for PH to keep ultra-pure water from dissolving lead piping) mechanisms should be physically sized to make dangerous amounts of additives impossible.”
As was attempted in the Oldsmar incident, an attack on the IT side of the house attempted to jump over to the OT side. The same could have been true for the South Staffs Water attack where attackers said, but was disputed by South Staffs Water, “It would be easy to change chemical composition for their water, but it is important to note we are not interested in causing harm to people.”
No OT Disruption
“Luckily, in the case of this attack, IT disruption did not impact the OT systems or compromise the safety and availability of its services,” said Peter Lund, vice president of Products – OT Security at OPSWAT. “South Staffs Water published in a statement, ‘This is thanks to the robust systems and controls over water supply and quality we have in place at all times, as well as the quick work of our teams to respond to this incident and implement the additional measures we have put in place on a precautionary basis.’
“It’s commendable (South Staffs Water) was able to keep their water system online and prevent the attackers from crossing into their OT environment. However, while not all water and wastewater systems have the staff, funding, and resources as larger critical industries do to implement such a robust security program, there are certain controls and processes that should be prioritized among others to secure their environment when an attacker strikes.
“The first is auditing what is currently in place, including what assets are connected to both IT and OT environments, and then planning and budgeting for future changes. Changes should comply with cyber security best practices as defined by the Cybersecurity & Infrastructure Security Agency (CISA).”
As recommended by CISA, Lund said funding should be allocated toward the segmentation of OT and IT networks by allowing one-way communications, which enables secure remote monitoring of OT assets while reducing operational cost as compared to a firewall.
“The first challenge is getting the security guidance to the small utilities that need it,” Byres said. “This is where the government needs to step in. They need to set the baseline requirements and provide the technical assistance to achieve these requirements. These programs need to be tailored to the capabilities of small utilities, and not be a quest for perfect security. There are some good models for this in the energy industry for rural electricity providers and the same is needed for water utilities.
“Assuming a utility has designated someone to be responsible for cybersecurity, the Water ISAC is an excellent resource they should investigate. The American Water Works Association (AWWA) also offers a Cybersecurity Guidance and Assessment Tool along with numerous seminars,” Byres said.
In the end, the cybersecurity sector needs to come together to help the water industry.
“We must empathize with the water industry and encourage cost-effective solutions to minimize their cyber threats,” Chowdhury said. “These are not organizations that have steep budgets to buy everything in anything.”
COMMON WATER INDUSTRY SECURITY ISSUES
Weak cybersecurity has been a sore point for the water industry for years. Through no fault of their own, they just don’t have the funding or staffing to handle the ongoing potential for attacks.
The following are what Dewan Chowdhury, chief executive at security provider, malcrawler sees as the most common cybersecurity issues known within the water industry are:
Weak Remote Access: You may find one engineer responsible for the OT and IT equipment, and they don’t have the workforce to manage everything on site. You find various remote access solutions to allow engineers to access equipment 24×7. You cannot strip the remote access away because of the reality they face. They should be encouraged to use multi-factor authentication with the remote access solution to help reduce the risk.
Flat Network: Another common problem is using flat networks that connect IT and their OT networks. Suppose they leverage network segmentation to prevent an attacker from pivoting from the IT network to the OT. This is very cheap to implement just leveraging network VLANs on existing switches is a simple fix.
Lack of OT Firewall or Data Diodes: Finding a firewall on the OT side of their industry is quite rare. If they were to implement simple firewalls to enforce ICS level protocols (e.g., DNP3/Modbus/etc.) They can drastically reduce the man-in-the-middle attack scenario of sending malicious control commands to the SCADA devices. The lack of data diodes in the industry is another problem. Data diodes restrict two-way communication and force a one-way communication path. This is commonly found in power utility networks to protect critical systems like EMS. The data diode solution is not very expensive.
— Gregory Hale