ChatGPT can write smart contracts; just don’t use it as a security auditor

Researchers from Salus Security tested GPT-4 and other artificial intelligence systems’ ability to detect seven common security vulnerabilities.

A pair of researchers from Salus Security, a blockchain security company with offices in North America, Europe and Asia, recently published research showcasing GPT-4’s talents when it comes to parsing and auditing smart contracts.

As it turns out, artificial intelligence (AI) is pretty good at generating and parsing code, but you wouldn’t want to use it as a security auditor.

Per the paper:

“GPT-4 can be a useful tool in assisting with smart contract auditing, especially in code parsing and providing vulnerability hints. However, given its limitations in vulnerability detection, it cannot fully replace professional auditing tools and experienced auditors at this time.”

The Salus researchers used a data set of 35 smart contracts (called the SolidiFI-benchmark vulnerability library), which contained a total of 732 vulnerabilities, to judge the AI’s ability to detect potential security weaknesses across seven common types of vulnerabilities.

Related: Crypto lost in BNB Chain heists down by 85% in 2023: Report

According to their findings, ChatGPT is good at detecting true positives — actual vulnerabilities that, outside of a testing environment, would be worth investigating. It reached greater than 80% precision in testing.

However, it has an apparent problem with generating false negatives. This is expressed through a statistic called “recall rate,” and in the Salus team’s experiments, GPT-4’s recall rate was as low as only 11% (higher is better).

This indicates, as the researchers concluded, “that GPT-4’s vulnerability detection capabilities are lacking, with the highest accuracy being only 33%.” As such, the researchers recommend using dedicated auditing tools and good old-fashioned human know-how for auditing smart contracts until AI systems such as GPT-4 can be brought up to speed.

“In summary, GPT-4 can be a useful tool in assisting with smart contract auditing, especially in code parsing and providing vulnerability hints. … When using GPT-4, it should be combined with other auditing methods and tools to enhance the overall accuracy and efficiency of the audit.”