When innovation is built on insecure foundations
The case of McKinsey’s Lilli platform is not just a dramatic breach story. It is a warning about how many AI platforms are being deployed today across companies and institutions, with speed and adoption taking priority over security by design. According to CodeWall’s public disclosure, an autonomous offensive agent reportedly identified a chain of vulnerabilities that led to access to sensitive production data. An AI platform that is not built with strong security from the outset becomes a large scale risk multiplier.
The problem goes far beyond a software bug
Many organizations still treat AI platforms as if they were ordinary productivity tools with a more sophisticated interface. In reality, they concentrate search histories, uploaded files, internal documents, financial material, workflows, model settings, and the day to day footprint of how knowledge work is actually performed. That means a single weakness may expose far more than a database table or an API. It can reveal what employees are working on, what information they are searching for, which documents are being retrieved, and how organizational knowledge is being structured and reused.
In public sector settings, or in any environment handling citizen data, the stakes are even higher. An insecure AI platform can become an indirect channel for personal data leakage, a window into administrative processes, or even a surveillance layer over work itself. Once prompts, histories, embeddings, outputs, and internal files are processed inside the same system, the blast radius of a compromise is much larger than in a conventional application.
The prompt layer is now critical infrastructure
What makes modern AI platforms distinct is that defending the application layer is no longer enough. The prompt layer also has to be secured. System prompts, retrieval rules, filtering policies, and agent workflow settings shape how the platform behaves, what it reveals, what it refuses, and what users come to trust. If those controls are altered, the damage may not appear as a visible outage or a classic breach. It may appear as distorted advice, misleading analysis, hidden data exfiltration through outputs, or the quiet erosion of the system’s safeguards.
That is why the security question in the age of agents and RAG systems is not only about confidentiality and availability. It is also about the integrity of machine assisted judgment. If employees, consultants, or public officials rely on these systems to prepare recommendations, risk assessments, case handling notes, or responses to citizens, then compromising the AI platform becomes an institutional problem, not just a technical one.
The only credible response is security by design
The core lesson is simple. AI cannot be deployed first and secured later. It requires zero trust architecture, strong access controls, data minimization, encryption, logging, red teaming, DPIAs where relevant, and clear governance over prompts, models, and data flows. It also requires inspectable infrastructure and open standards, so institutions are not locked into opaque systems they cannot meaningfully audit.
If cybersecurity is not built into the architecture from the start, AI does not modernize institutions. It amplifies their fragility and exposes knowledge assets, labor processes, and the rights of the people whose data those systems touch.
Sources
CodeWall, How We Hacked McKinsey’s AI Platform, public disclosure of the reported vulnerability chain: https://codewall.ai/blog/how-we-hacked-mckinseys-ai-platform
McKinsey, Rewiring the way McKinsey works with Lilli, our generative AI platform, official overview of the Lilli platform: https://www.mckinsey.com/capabilities/tech-and-ai/how-we-help-clients/rewiring-the-way-mckinsey-works-with-lilli
OWASP, Top 10 for Large Language Model Applications, concise reference on major LLM application risks: https://owasp.org/www-project-top-10-for-large-language-model-applications/
NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0), foundational framework for trustworthy and secure AI: https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf
NCSC, Guidelines for secure AI system development, guidance for secure AI system development: https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development
EUR-Lex, Regulation (EU) 2016/679, GDPR, the EU legal framework for personal data protection: https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng
ENISA, Multilayer Framework for Good Cybersecurity Practices for AI, European good practices for AI cybersecurity: https://www.enisa.europa.eu/publications/multilayer-framework-for-good-cybersecurity-practices-for-ai