The team found that the AI would produce the correct web address just 66 percent of the time. 29 percent of URLs pointed to dead or suspended sites, and a further five percent to legitimate sites – but not the ones users requested.
Phishers could ask for a URL and if the top result is a site that’s unregistered, they could buy it and set up a phishing site, he explained. “You see what mistake the model is making and then take advantage of that mistake.”
The problem is that the AI is looking for words and associations, not evaluating things like URLs or a site’s reputation.