Lakera launches to defend large language models from evil prompts

The generative AI movement relies on large language models (LLMs) to interpret and create human-language texts from simple prompts, such as summarizing a document, writing a poem, or answering a question using data from multiple sources.

Bad actors can use “prompt injection” to trick an LLM-powered chatbot into giving unauthorized access to systems or bypassing strict security measures.

Lakera, a Swiss startup, launches today to protect enterprises from LLM security vulnerabilities like prompt injections and data leakage. Along with its launch, the company announced a $10 million funding round earlier this year.

Data wizardry

Lakera’s database includes insights from publicly available open source datasets, its own research, and, intriguingly, data from Gandalf, an interactive game it launched earlier this year.

Gandalf gives users linguistic tricks to “hack” the LLM and reveal a password. Gandalf becomes more adept at defending against this as the user advances to the next level.

Gandalf, powered by OpenAI’s GPT3.5 and LLMs from Cohere and Anthropic, appears to be a fun game to highlight LLM weaknesses. However, Gandalf’s insights will inform the startup’s flagship Lakera Guard product, which companies integrate via API.

“Gandalf is literally played all the way from like six-year-olds to my grandmother, and everyone in between,” Lakera CEO and co-founder David Haber told TechCrunch. But “a large chunk of the people playing this game is actually the cybersecurity community.”

Over the past six months, Haber said the company has recorded 30 million interactions from 1 million users, allowing it to develop a “prompt injection taxonomy” that categorizes attacks into 10 categories. Direct attacks, jailbreaks, sidestepping, multi-prompt attacks, role-playing, model duping, obfuscation (token smuggling), multi-language attacks, and accidental context leakage.

This lets Lakera customers scale their inputs against these structures.

“Ultimately, we are turning prompt injections into statistical structures,” Haber said.

Though prompt injections are one cyber risk vertical Lakera is focused on, it also protects companies from private or confidential data leaking into the public domain and modifies content to ensure LLMs don’t serve anything unsuitable for kids.

Haber said the most requested safety feature is toxic language detection. “So we are working with a big company that provides generative AI applications for children to protect them from harmful content.”

In addition, Lakera is addressing LLM-enabled misinformation and factual errors. Haber says Lakera can help with “hallucinations” when the LLM output contradicts the initial system instructions or when the model output is factually incorrect based on reference knowledge.

In either case, our customers give Lakera the context in which the model interacts, and we make sure it doesn’t act outside of that limits,” Haber said.

Lakera is a mixed bag for security, safety, and data privacy.

EU AI Act
Lakera is launching at an ideal time with the EU AI Act, the first major AI regulations. Article 28b of the EU AI Act requires LLM providers to identify risks and take precautions to protect generative AI models.

Haber and his two co-founders have advised the Act, laying technical foundations before its introduction in a year or two.

“There are some uncertainties around how to actually regulate generative AI models, distinct from the rest of AI,” Haber said. Technology is progressing faster than the regulatory landscape, which is difficult. We share developer-first perspectives in these conversations to complement policymaking by understanding what these regulatory requirements mean for the people in the trenches bringing these models into production.

Security blocker
In conclusion, ChatGPT and its peers have taken the world by storm in the past 9 months like few other technologies, but enterprises may be more hesitant to use generative AI in their applications due to security concerns.

“We speak to some of the coolest startups to some of the world’s leading enterprises — they either have these [generative AI apps] in production or are looking at the next three to six months,” Haber said. We are already working behind the scenes to ensure a smooth rollout. We help these companies bring their generative AI apps to production because security is a major obstacle.

Lakera, founded in Zurich in 2021, claims major paying customers it cannot name due to security concerns about revealing their protective tools. The company says LLM developer Cohere, which recently reached a $2 billion valuation, is a customer, along with a “leading enterprise cloud platform” and “one of the world’s largest cloud storage services.”

After going public, the company has $10 million to build its platform.

Haber said, “We want to be there as people integrate generative AI into their stacks, to make sure these are secure and mitigate risks.” “We will evolve the product based on the threat landscape.”

Swiss VC Redalpine led Lakera’s investment, with Fly Ventures, Inovia Capital, and angel investors contributing.