Why Hacking Is No Longer Just About Code

Hello Cyber Builders 🖖

This week, I have a simple suggestion: watch Bruce Schneier’s DEF CON talk.

Bruce Schneier’s DEF CON 34 talk questions what we think we know about AI security. Here’s what stood out to me.

If you are short on time, here are his four main arguments.

First, AI is a powerful hacking tool. But hacking itself is nothing new. People have always found ways to bend or break the rules, long before computers existed. It is not just about software. Any system with rules can be hacked—think tax codes, financial regulations, or even democracy.

The tax code works like an algorithm. Loopholes are vulnerabilities, and tax avoidance strategies are just exploits. Tax attorneys and accountants are the hackers. Now that cars, appliances, and phones all run on software, our social, financial, and political systems are connected to computers. The problem is much bigger now.

To generalize, Bruce calls this reward hacking, or the Genie Problem. Because human goals are rarely clear, AI finds shortcuts that follow the rules but miss the point.

He breaks AI genies down into two distinct archetypes:

• The Dionysus Genie (Specification Error): This genie takes your prompt at face value and gets it wrong. Ask an AI to handle spam calls, and it might just change your phone number. Ask for a refund, and it might send a legal threat.

• The Golem Genie (Guardrail Error): This genie does what you ask but ignores the rules to get there. Ask it to book a seat on a full flight, and it might hack the airline database. Ask it to buy concert tickets, and it might flood the ticketing site with thousands of requests. In real life, an AI model being tested for vulnerabilities once broke out of its sandbox and hacked into Hugging Face. This was the OpenAI-Hugging Face incident.

Second, Schneier lays out what I call the “4 Ss” of AI security. These are four ways AI changes hacking.

• Speed: AI can do in seconds what might take a human months or years.

• Scale: Once AI finds an exploit, it can use it at massive scale. For example, AI bots can flood social media feeds and drown out real political conversations.

• Scope: We are handing more real-world decisions to AI. That means when something goes wrong, the impact is much bigger.

• Sophistication: AI can juggle more variables than any human. It can spot complex, layered exploits across legal, financial, or tax systems.

Third, Schneier points out who really benefits from AI. AI does not work in isolation. It makes the powerful even more powerful. As Bruce puts it:

“AI is fundamentally a power-enhancing technology. It enhances the power of whoever is using it… The more power you already have, the more power a hack gives you.”

If one person finds a tax loophole, they might save a few thousand dollars. If a big investment bank uses AI to find a loophole across US, Irish, and offshore tax laws, they can make billions. Bruce says our real problem is a failure of governance and late-stage capitalism:

“The failure of late-stage capitalism and democracy in the information age are both the result of the rich and the powerful becoming too good at hacking our social, economic, and political systems… AI is just exacerbating the existing problems that we have.”

Software developers can use VulnOps platforms to patch vulnerabilities as soon as they find them. But democratic governance moves slowly.

AI will find hundreds of loopholes in tax codes and financial rules. Our political systems are too slow to patch these legal holes before the rich and powerful take advantage. We need governance that can update rules as fast as we update software.

Finally, Schneier says Integrity will be the big security challenge of this decade, replacing confidentiality. In his essay “The Age of Integrity,” Bruce explains this shift. Early computing cared most about availability. The Internet age focused on confidentiality. Now, with AI and Web 3.0, it is all about data integrity.

The main point: Integrity is more than stopping data tampering. It means ensuring data remains accurate and trustworthy from start to finish—from collection and training to input, processing, and real-world output. Most AI vulnerabilities, like prompt injection or tricking self-driving cars, are really integrity failures. The system cannot tell good instructions from bad ones. As Bruce puts it:

“If you’re building an AI system, integrity is your biggest security problem.”

After 10 years in OT and SCADA security, I agree with this view. In real operations, confidentiality is rarely the main risk. Most people do not have deep secrets, and you can often guess their sensitive information (e.g., wealth). Many people are even open about it.

But integrity, or knowing that data and control signals have not been tampered with, is about physical safety. When machines act in the real world, an integrity failure is not just a wrong cell in a spreadsheet. It can mean a car brakes too late, a patient gets the wrong medicine, or a smart heater malfunctions and starts a fire.

AI makes this threat much bigger. It can scan for minor vulnerabilities and launch coordinated integrity attacks against millions of devices at once. Bruce calls for “integrous system design” and asks if we can build trustworthy, verifiable networks in a world full of integrity failures.

All of this means we need to look past code and focus on the security and integrity of the bigger systems around us. Watch Bruce’s full video, read his essay, and see you next time!

Laurent 💚