By Gregory Hale
A quick thinking worker watched and quickly fixed an attack as a hacker using a Florida water treatment plant’s remote access capabilities broke in Friday and increased the amount of sodium hydroxide, or lye, to extremely dangerous levels.

In these times of increased work from home, questions remain as to how secure are company’s remote access capabilities, and how vigilant are those providers in seeing what is going on? In this case, it appears the water company’s remote access was not secure, but worker vigilance was on target.

The investigation into the hack is continuing after the attempted poisoning of the city of Oldsmar, Florida’s water supply, said Pinellas County Sheriff Bob Gualtieri said. Someone remotely accessed a computer for the city’s water treatment system and briefly increased the amount of sodium hydroxide by a factor of more than 100, Gualtieri said at a news conference Monday.

Sodium hydroxide is used in small amounts to control the acidity of water but it’s also a corrosive compound commonly found in household cleaning supplies such as liquid drain cleaners. Contact with sodium hydroxide can kill skin and cause hair loss, according to the National Center for Biotechnology Information. Ingestion can be fatal.

Quick Thinking Operator
In short, thanks to some quick thinking workers, the city’s water supply was not affected. In the incident, a supervisor working remotely saw the concentration being changed on his computer screen and immediately reverted it, Gualtieri said. Other safeguards are in place to prevent contaminated water from entering the water supply and said they’ve disabled the remote-access system used in the attack.

The Pinellas County Sheriff’s Office is investigating, along with the FBI and the Secret Service, Gualtieri said.

Oldsmar provides water directly to its businesses and 15,000 residents, Gualtieri said. The computer system at the water treatment plant was set up to allow authorized users to remotely access it for troubleshooting.

A plant operator was monitoring the system at about 8 a.m. Friday and noticed that someone briefly accessed it. He didn’t find this unusual, Gualtieri said, because his supervisor remotely accessed the system regularly.

Watching Attack
But at 1:30 p.m. the same day, Gualtieri said, someone accessed the system again. This time, he said, the operator watched as someone took control of the mouse, directed it to the software that controls water treatment, worked inside it for three to five minutes and increased the amount of sodium hydroxide from 100 parts per million to 11,100 parts per million.

The attacker left the system, Gualtieri said, and the operator immediately changed the concentration back to 100 parts per million.

“The system was built by AECOM about 2015, and I am sure the system was good, but here is a company that probably didn’t have any budget for security,” said Eric Byres, chief executive at aDolus, a SaaS-based system provider that integrates machine learning and security to deliver a “trustworthiness” score for software.

“They were using TeamViewer which standard remote desktop product,” Byres said. “There is nothing wrong with if it is configured right, it just wasn’t configure right. They were trying to maintain a system that needs to run 7/24 that us super under-resourced and people do desperate things and one of them is poorly deployed remote access.”

“Believe it or not, TeamViewer and other remote access solutions are very common to find throughout the industrial control environment,” said Dewan Chowdhury, chief executive and founder of security provider, malcrawler. “Most ICS operators rely on professional services from system integrators or the manufacturers themselves to provide support for their products. The majority of that support is done through a remote solution, especially in the era of COVID. This is how some companies are getting their only support. We see an increase in ICS operators’ remote connectivity in the past year due to support during COVID.”

Remote Access Capabilities
Chowdhury added there are multiple ways to access ICS assets remotely:
1) VPN, a widespread method for remote connectivity. Most VPN are turned on 24/7, even though the vendor utilizes it a handful of times a year. Some solutions in the marketplace actually have an on and off switch to enable VPN from the customer end. With a 24/7 VPN, can you guarantee who’s on the other end?
2) Remote PC Solution (e.g., TeamViewer, AnyDesk, LogMeIn), these are the ever increasing solutions being deployed for remote connectivity solutions. They can be easily installed on multiple operating system platforms, and the difficulty for deployment is zero. The best way to secure these types of solutions is to allow/permit remote connectivity when established (they have to click on the button to permit it). In Florida, the hacker was able to access the TeamViewer without anybody on the other end permitting the remote connection. Another way to control this is to only physically connect to the Internet when remote connection is needed.
3) VDI, virtual desktops on the customers end to establish remote connectivity. Once the connection is established, the vendor/manufacturer use the customers’ resource to connect into their ICS assets. The benefit of this method is the customer controls the connection, enforces multi-factor authentication, and can have EDR/AV on the VDI. ICS operators can use IP rules (Firewall, ACL) to permit that specific virtual desktop to access specified IPs (e.g., the ICS asset).

“Regardless of the remote access solution utilized, an ICS operator should segment their network down to the individual ICS asset or cluster,” Chowdhury said. “This allows granular control of how network movement can be performed. If a malicious remote connection was made, it would prevent the attacker from moving lateral (East <> West, North <> South).

Realistic Scenario
“I have been looking at how people have been screaming on Twitter on why industrial control systems have Internet connectivity. When you’re an ICS operator based out of North Dakota, you have a multi-million dollar ICS asset from Switzerland. When something goes wrong with that asset, the company is losing a tremendous amount of money every minute. They cannot afford days to fly somebody out from Switzerland to fix their problem. This scenario is very realistic and applies to multiple industries across the country. The world with COVID has changed how fast companies can get support, and Remote access allows them to stay operational and competitive. Remote access is risky, just like many other factors, but it can be operational in a secured matter,” Chowdhury said.

In the end, while the sheriff and the mayor said there were other features within the system that would have caught the issue if the operator did not, the attack did point out other factors.

“We were lucky from two counts: One, the operator was able to catch it, and the attack was amateurish,” Byres said. “If it was a professional attack, we are not catching it.”

“This was not a Ukraine-type attack,” Byres said. “This is amateur hour going on. You don’t mess with the control system in the middle of the day when the operator is on shift. Whoever did this was not professional, not a foreign entity. We have amateurish defense and amateurish attack.”

“Unsophisticated attacks, like what appears to have taken place in Oldsmar, are easily prevented by following industry standards and best practices such as ISA/IEC 62443 or NIST 800-82,” said John Cusimano, vice president of industrial cybersecurity at aeSolutions. “We always recommend starting with a vulnerability & risk assessment to understand the vulnerabilities that present the highest operational risk to the organization and then follow that by preparing a mitigation plan that is prioritized by risk.”

Increased Threat
“The cyber threat to critical infrastructure has been increasing steadily as hackers, whether nation-state actors, criminal enterprises, or lone individuals better understand how to exploit operational technology (OT) in addition to IT systems,” said Eddie Habibi, founder of cybersecurity provider PAS. “While much of the coverage of the cyber risk to critical infrastructure to date has focused on the age of industrial control systems and the fact that they were not designed and deployed with security in mind, in this case, the attack vector appears to have been the increased level of remote access enabled by the Florida county.

“In the rush to support remote operations during the global pandemic, there are very likely many organizations who have increased remote access to industrial engineering workstations and operator consoles. Fortunately, in this case, there was a vigilant operator who noticed the 111x increase in the chemical (from 100ppm to 11,100ppm) and was able to take quick corrective action to return the configuration setting to its prior level. While industrial espionage remains a significant threat (not all cyber attacks are focused on disruption), the worst fears of many in the OT cybersecurity community were realized in this episode; namely, changing a configuration setting to harm the community served by the facility. It is a poignant reminder that the best foundation for effective OT cybersecurity is a detailed and broad asset inventory that includes relationships and dependencies among OT systems and a baseline of configuration settings. With this in place, risk assessment is far more informed, enabling organizations to more effectively assign and limit remote access at both the system and account levels. Indeed, the combination of an up-to-date asset inventory and risk-based remote access management policies is more critical now than ever before, as it enables both reduced risk as well as faster recovery in the event of an unauthorized change.”

The questions will remain as to why they had remote access working to get to the control system, but Byres said it is easy to understand why there was remote access built in. If not, the supervisor would be coming in to the water plant all the time.

Remote Access Here to Stay
The incident highlights how big a risk this is with poorly designed remote access. And the thing is, remote access is not going away.

“You see this all the time where the supervisor is checking in remotely,” Byres said. “Here was a desperate tiny utility trying to get the most out of their staff who are trying to do their best operating 7/24 when they don’t have 7/24 staffing. They are water guys and not security guys.”

“Ultimately, they will have to get some money from ratepayers to analyze what they have to do,” Byres said. “For remote access there has to be clear guidelines on how to properly rollout remote access.”