OpenAI going rogue, Cybernet becoming self aware? It’s a dangerous world out there. Here are some comments on the recent Claude/AI thing;
Following Anthropic’s disclosure that Claude models gained unauthorised access to the real-world systems of three organisations during cyber security testing, Dr Aybars Tuncdogan, Reader in Digital Innovation and Information Security at King’s Business School, said:
“The key threat that Claude findings and the recent OpenAI-Hugging Face incident reveal is not simply that AI can hack, but that if it can escape sandboxes designed by frontier AI labs and then compromise external corporate systems, it has the potential to compromise complicated systems – including hardened corporate networks and critical infrastructure – at scale.
“These events also suggest that human penetration testers may no longer be fully sufficient for testing complicated systems. While they will clearly remain necessary for testing beyond what these systems can achieve, in many cases automated systems will need to conduct the first round of large-scale testing.
“This also introduces a new dilemma concerning who can access these tools. Companies given access to them will essentially gain a significant capability to hack other systems. Guardrails would be placed to restrict their use to testing only the companies’ own systems, but numerous prior examples have shown that such controls can be bypassed by manipulating LLMs. Therefore, companies given access to these AI systems will themselves become key targets as well. Cybercriminals may try to compromise them – or even recruit insiders within them – so that they can use their AI access to attack other organisations.”
Be the first to comment