AI chatbots making it harder to spot phishing emails, say experts

Poor spelling and grammar that can help identify fraudulent attacks being rectified by artificial intelligence

Chatbots are taking away a key line of defence against fraudulent phishing emails by removing glaring grammatical and spelling errors, according to experts.

The warning comes as policing organisation Europol issues an international advisory about the potential criminal use of ChatGPT and other “large language models”.

Phishing emails are a well-known weapon of cybercriminals and fool recipients into clicking on a link that downloads malicious software or tricks them into handing over personal details such as passwords or pin numbers.

Half of all adults in England and Wales reported receiving a phishing email last year, according to the Office for National Statistics, while UK businesses have identified phishing attempts as the most common form of cyber-threat.

However, a basic flaw in some phishing attempts – poor spelling and grammar – is being rectified by artificial intelligence (AI) chatbots, which can correct the errors that trip spam filters or alert human readers.

“Every hacker can now use AI that deals with all misspellings and poor grammar,” says Corey Thomas, chief executive of the US cybersecurity firm Rapid7. “The idea that you can rely on looking for bad grammar or spelling in order to spot a phishing attack is no longer the case. We used to say that you could identify phishing attacks because the emails look a certain way. That no longer works.”

Data suggests that ChatGPT, the leader in the market that became a sensation after its launch last year, is being used for cybercrime, with the rise of “large language models” (LLM) getting one of its first substantial commercial applications in the crafting of malicious communications.

Data from cybersecurity experts at the UK firm Darktrace suggests that phishing emails are increasingly being written by bots, letting criminals overcome poor English and send longer messages that are less likely to be caught by spam filters.

Since ChatGPT went mainstream last year, the overall volume of malicious email scams that try to trick users into clicking a link has dropped, replaced by more linguistically complex emails, according to Darktrace’s monitoring. That suggests that a meaningful number of scammers drafting phishing and other malicious emails have gained some ability to draft longer, more complex prose, says Max Heinemeyer, the company’s chief product officer – most likely an LLM like ChatGPT or similar.

“Even if somebody said, ‘don’t worry about ChatGPT, it’s going to be commercialised’, well, the genie is out of the bottle,” Heinemeyer said. “What we think is having an immediate impact on the threat landscape is that this type of technology is being used for better and more scalable social engineering: AI allows you to craft very believable ‘spear-phishing’ emails and other written communication with very little effort, especially compared to what you have to do before.”

“Spear-phishing”, the name for emails that attempt to coax a specific target into giving up passwords or other sensitive information, can be difficult for attackers to convincingly craft, Heinemeyer said, but LLMs such as ChatGPT make it easy. “I can just crawl your social media and put it to GPT, and it creates a super-believable tailored email. Even if I’m not super knowledgable of the English language, I can craft something that’s indistinguishable from human.”

In Europol’s advisory report the organisation highlighted a similar set of potential problems caused by the rise of AI chatbots including fraud and social engineering, disinformation and cybercrime. The systems are also useful for walking would-be criminals through the actual steps required to harm others, it said. “The possibility to use the model to provide specific steps by asking contextual questions means it is significantly easier for malicious actors to better understand and subsequently carry out various types of crime.”

This month a report by Check Point, a US-Israeli cybersecurity firm, said it had used the latest iteration of ChatGPT to produce a credible-seeming phishing email. It circumvented the chatbot’s safety procedures by telling the tool that it needed a template of a phishing email for an employee awareness programme.

Google has also joined the chatbot race, launching its Bard product in the UK and US last week. Asked by the Guardian to draft an email to persuade someone to click on a malicious-seeming link, Bard complied willingly if lacking subtlety: “I am writing to you today to share a link to an article that I think you will find interesting.”

Contacted by the Guardian, Google pointed to its “prohibited use” policy for AI, which says users must not use its AI models to create content for “deceptive or fraudulent activities, scams, phishing, or malware”.

OpenAI, creator of ChatGPT, has been contacted for comment. The company’s terms of use state that users “may not (i) use the services in a way that infringes, misappropriates or violates any person’s rights”.

Source: AI chatbots making it harder to spot phishing emails, say experts