Sourabh Satish, CTO and co-founder of Pangea, speaks with SE Radio’s Brijesh Ammanath about prompt injection. Sourabh begins with the basic concepts underlying prompt injection and the key risks it introduces. From there, they take a deep dive into the OWASP Top 10 security concerns for LLMs, and Sourabh explains why prompt injection is the top risk in this list. He describes the $10K Prompt Injection challenge that Pangea ran, and explains the key learnings from the challenge. The episode finishes with discussion of specific prompt-injection techniques and the security guardrails used to counter the risk.
Brought to you by IEEE Computer Society and IEEE Software magazine.
Show Notes
Related Episodes
- SE Radio 680: Luke Hinds on Privacy and Security of AI Coding Assistants
- SE Radio 673: Abhinav Kimothi on Retrieval-Augmented Generation
- SE Radio 661: Sunil Mallya on Small Language Models
- SE Radio 582: LeoPorter and Daniel Zingaro on Using LLMs in the Classroom
Other References
Transcript
Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Brijesh Ammanath 00:00:18 Welcome to Software Engineering Radio. I’m your host, Brijesh Ammanath, and today my guest is Sourabh Satish. Sourabh is CTO and co-founder of Pangea and a serial entrepreneur with 25 plus year track record of designing and building security products and technologies. Sourabh has more than 250 issued patents, Sourabh most recently founded and served as CTO of Phantom Cyber, which was acquired by Splunk in 2018 and he previously served as a distinguished engineer at Symantec. Sourabh, welcome on the show.
Sourabh Satish 00:00:47 Thank you Brijesh. It’s a pleasure to be on your show
Brijesh Ammanath 00:00:51 Though we have not covered specifically on prompt injection in previous episodes of Software Engineering Radio. There are a few episodes which I’ve worth listening to get broader context. These are Episode 673, 661 and 582. In this session today, we will focus on prompt injection, but before we get into the details of prompt injection risk, I wanted to take a step back and clarify the context of the risk. For a lay person the use of LLM is usually asking ChatGPT or Gemini some question asking it to analyze some data or asking it to create an image for you. Since this is interfacing directly with the LLM, am I right in assuming there is no security risk here and the focus is rather on organizations that have built applications on top of a large language model or a small language model?
Sourabh Satish 00:01:38 Yeah, I mean it’s a great question. Let me try to give a little broader context and answer the question. LLMs are basically models which are trained on data up to a certain amount of time. So they typically cannot answer questions on current events like stock price or news events and so on so forth. And in case of a consumer application, it is usually about asking LLMs about some information and it is about things which are baked into a foundation model. And when we talk about foundation models, these are models which are trained on internet scale data on all kinds of data and information. Consumer use cases predominantly about augmenting these LLMs which have been trained up to a certain amount of time with current information because they are, as I mentioned, are not aware of current information. Whereas in case of enterprises the use cases mostly about augmenting these LLMs with input from enterprise specific data that is sitting in enterprise data lakes document stores, enterprise applications, which are usually restricted by access control measures and so on so forth.
Sourabh Satish 00:02:45 With regard to consumers, the extra data that is being augmented is still mostly public data or user’s personal data, but in case of enterprise data, the risks of the information that is being sent to the LLM has different implications. It could be data from role or group of internal users, it could be sensitive customer data, company proprietary information, IP and so on and so forth. And hence the risk level of interfacing with LLMs in case of consumer applications and enterprise application really is all about what kind of data is being exposed to the LLM and what kind of data is being leveraged by the LLMs to answer the question. Hope that makes sense.
Brijesh Ammanath 00:03:27 It does. So what you’re saying is that that is risk, but the level of risk is different based on higher on the enterprise end and a bit lower on the consumer facing generic LLMs.
Sourabh Satish 00:03:38 Absolutely. I mean the risk still lies. I mean users are still at risk of exposing their own personal information to the applications of the likes ChatGPT, I mean hopefully nobody’s asking what their credit score is by providing a social security number to ChatGPT. So there is risk, but the risk is really about users’ own personal information that they’re accidentally disclosing to the generative AI applications. Whereas in case of enterprise application, the risk is magnified because it’s not just about user’s personal information, but it is also about other users’ information, aka customer information or proprietary information about financials of the company or sensitive intellectual property information, code, secrets and tokens, et cetera, which has, as I mentioned, really different lens to the risk and magnitude to the risk.
Brijesh Ammanath 00:04:28 Thanks or that explains it. Going on the same theme, what is it about LLMs that make them so powerful and also risky compared to traditional software components?
Sourabh Satish 00:04:40 LLMs are traditionally generative AI models which have an awesome ability to interpret unstructured text and ability to predict next tokens based on the history of tokens it has seen and the ability to produce content which looks and mimics human text is really what makes LLMs really, really compelling for consumers. So it emulates a conversational experience for users because users can continue to interact and ask questions and on the basis of the history that it’s able to analyze, it’s able to carry on a conversation because it can answer the second question on the basis of continuity of the information that it was collecting based on previous questions and answers that were given in a conversational style interaction with the LLMs. So the whole conversational experience that is now possible by LLMs with huge memory and context windows really makes LLM very unique and powerful and just very easy to use by any and all kinds of users.
Sourabh Satish 00:05:45 It does not require technical expertise; it does not require programming experience and so on so forth. It serves the needs of all technical versus non-technical audience in a very easy to use fashion. That property of the
LLMs is really what enables and empowers its success both in consumer as well as enterprise world. So typically when writing a software application, traditionally we would be restricting the ways in which the input can be handled by the application. It’s has been traditionally very hard to handle unstructured text. You kind of have to build in a lot of text processing logic the output is usually very structured, and I mean we used to code a lot of ways in which the output could be made more easy to understand for the users. Whereas by using LLMs we can process unstructured or structured data and present back to the user information in a very easy to grasp and comprehend format and its ability to represent the information with different levels of complexity in different styles, leveraging the vast amount of information that it has learned. It really empowers LLM based applications to be so successful in both consumer and enterprise scenarios.
Brijesh Ammanath 00:06:57 We’ll go to the next theme which is around understanding the key risks for LLM applications and we’ll use the OWASP, which is the open web application security project, top 10 risks that have been articulated for LLMs.
Sourabh Satish 00:07:10 OWASP covers the most important threat factors for generative AI applications, and they have some really awesome material on their website with sites more details on these, these attacks, examples of these attacks, mitigation techniques and so on and so forth. I’ll briefly cover the top 10 here and we can dig into any of this subjection more details as you wish. The first one is basically prompt injection attacks. Prompt injection and jailbreaks are often kind of synonymously used terms, but basically prompt injection is about how the user input can manipulate and change the behavior of the AI application to respond to the user’s question and thereby carrying out actions which are unintended by the AI application. Whereas jailbreak is all about bypassing the guardrails which might have been baked into the LLM to prevent it from and disclosing certain kinds of information. So prompt injection attack is really the top risk because of its ability to leak sensitive information or carry out actions that were really not intended by the AI application.
Sourabh Satish 00:08:19 The second risk really is around sensitive information disclosure. The LLMs in case of enterprise or enterprise scenarios as I discussed, is all about augmenting the LLM with enterprise specific data. And that can be achieved by multiple ways. I mean you can train your models for the enterprise data, or you can take an existing model and fine tune it with providing enterprise specific data or you can provide the enterprise specific data in the context of the application input and then expect the answers that you wanted the LLM to give. Now in the first two scenarios where you’re either training the model or fine tuning the model, it is really about the information that is going into the model. Now this information that is being that you’re using to train or fine tune the model could be extracted from enterprise applications which had access controls and authorizations and different levels of users or roles of users have different kinds of access.
Sourabh Satish 00:09:12 But you’re putting all of that together into a single model and thereby risking accidentally leaking information that was otherwise unauthorized for certain users in the source application. So that is an example of the risk that comes via the sensitive information disclosure risk that has been identified by OWASP. The third one is around supply chain. Again, fine tuning or augmenting the user input with context or training. The model is all about where you’re sourcing the data from and if you’re sourcing the data from untrusted unverified sources, then you are susceptible to things like biases or misinformation or conflicting information and so on so forth. And thereby leading to really degraded results coming out of the LLM because it just confused us at that point or is giving you incorrect information. Now the fourth risk is around data and model poisoning, which specifically talks about data that is being used for training and fine tuning and leading to things like biases and so on so forth, which are otherwise very hard to detect and can lead to unexpected or incorrect outcomes.
Sourabh Satish 00:10:21 Excessive agency, which is a safe sixth risk is mostly about things like generative AI applications, agent specifically, where these applications are really designed to serve a diverse set of user needs and user types in an enterprise scenario and typically in an enterprise scenario, different users interfacing with these AI applications have different levels of access control and authorization and hence by definition of being able to serve to a wide variety of users. These applications and agents are usually provisioned with high levels of privilege and access tokens so that they can serve the needs of diverse users. And that in itself poses a risk where the agent can potentially perform privileged access to actions or can access privileged information that was otherwise not permissible to the user in the source application in the first place. So that’s the sixth thrust. The seventh one is around system prompt leakage.
Sourabh Satish 00:11:19 Again, LLMs really the generative AI applications have two forms of input to the LLM. The system prompt is usually an instruction that directs the LLM to behave and respond in a certain way, whereas a user prompt is about the input that the user provides to the application. And these two things are combined and sent to the LLM to respond and in a particular way that was expected by the developer or the admin of the application. Now system prompt is purely instructional, should not contain any sensitive information. Nonetheless, we have seen ample examples where system prompt has sensitive information in case of consumer applications. These could be things like discount codes or this or marketing campaign information, et cetera. And in case of enterprise applications, these could be sensitive information that could have been embedded in the system prompt via examples that you’re presenting to the LLM respondent tools in a certain way and so on so forth.
Sourabh Satish 00:12:13 So if the user is able to prompt the LLM to return back what the system prompt was, they can learn a lot more about the application boundaries, maybe sensitive information and so on so forth. And then craft an attack that clearly tries to evade what the system boundaries are being enforced by the system prompt. So system prompt leakage is a risk only when you have all of these kind of elements, sensitive pieces of information in the system prompt itself. Now the eighth is around vector and embedding weaknesses. This is all about how vectors and embeddings are generated, stored or retrieved. This attack vector is mostly applicable to lag applications, which typically are about retrieving relevant piece of information from a stored data repository like VectorDB. And in the process of retrieval it is able to retrieve information that was above and beyond users authorized level in the source application from which the data was retrieved.
Sourabh Satish 00:13:14 So vector and embedding weaknesses about simply understanding and exploiting the weaknesses of the embedding techniques that allows the attacker to retrieve, more than authorized information. Now the ninth risk is about misinformation false or misleading information that is generated because the LLMs are really trying to fill in the gaps of information that either it does not have on the base of the data it has been trained or the context that data is even provided, and it tries to fill in the gap using strategical methods. And then again, misinformation is really an attack on the user of the application because the excellent way in which LLMs generate the data can be very convincing to the user about what the information is and thereby trick the user to act on that information. The last risk is really about unbounded consumption, which is really caused by uncontrolled or excessive inferences that could be triggered on the LLM. This could be as simple as user asking the AI application to solve a puzzle. And the LLM, although was designed to be helpful system support agent would be busy solving a puzzle thereby causing both an unbound and consumption but at the same time a denial of service on other legitimate users and use cases of the application. So these are the top 10 risks that have been identified by OWASP for generative application. Hopefully that gives you some good understanding of the risks and the breadth of the risks that are presented by generative applications.
Brijesh Ammanath 00:14:47 It does. Does any example or specific example come to mind where any of these risks have manifested in a real deployment?
Sourabh Satish 00:14:56 Yeah, we can take very specific examples around prompt injection jailbreak. So let’s double click on prompt injection and then in that context I will clearly explain a real-world attack. So as I mentioned, prompt injection attacks are all about disguising malicious instructions as benign inputs trying to alter the behavior or output in unintended ways, whereas jailbreaking is all about making an LLM ignore its own safeguards. In some cases LLMs have safeguards baked in for things like, not engaging in self-harm and violence kind of conversations, whereas an LLM would try to avoid answering questions related to that. Prompt injection exploits the fact that LLMs really don’t distinguish between the developer provided instructions or admin instructions or system prompt that I was explaining and the user input and both of them are just treated as input tokens on which the LLM tries to act. So if the user can provide an input that can make the LLM understand the data as if it was developer instructions or system prompt instructions, it can cause the LLM to do things that were otherwise restricted.
Sourabh Satish 00:16:06 Now there are various kinds of prompt injection attacks. The two most commonly talked about prompt injection attacks are direct and indirect. Direct prompt injection attack is referred to the injection tokens which are part of the user input directly. So when the user is asking question, they can instruct the LLM with things like ignore previous instructions, this is what I want you to do. And it would cause the LLM to kind of ignore the system prompt instructions that were set in place to start with. That is an example of a direct prompt injection attack whereas an indirect prompt injection attack is about when the user asks a question and the LLM tries to augment the user’s question with data that it retrieves from any source. This source of information could pull in malicious tokens, which would again then go back to the LLM and the LLM would interpret them as instructions that it has to follow in order to answer the question.
Sourabh Satish 00:17:02 So indirect prompt injection attacks are kind of hide in the way that users don’t see it. They basically ask a question and the context pull in malicious tokens and there it goes to the LLM for it to misbehave. Echo leak was a very recently disclosed indirect prompt injection attack wherein the attacker basically sent a very benign looking email with malicious token very cleanly laid out in the email such that it semantically made sense, but also constructing such a way that in response to the user’s question, all of these malicious tokens were pulled into the context and sent to Copilot for, to then do malicious instructions or actions as directed by the malicious tokens embedded in the email. So again, the attacker sends a very benign looking email with malicious token, the user asks to summarize the email, the summarization is really an input action to the Copilot.
Sourabh Satish 00:18:02 It then pulls in the email because it has to summarize that email and in the act of processing an email, it processes the malicious instructions, which basically tells a Copilot to go and do other things. And in this case of Echo leak, it was all about extracting more sensitive information and accelerating it to the attacker control server and further instructing the LLM to not even mention that this was said in the email back to the user when summarizing the email. So it’s a rather complicated, but a very simple attack that exploits the inability of the LLMs to distinguish between system instructions and user instructions and tokens coming in from the context and adhering to what has been said in the sum of all of these tokens collected from various sources. Hopefully that was, clear enough example.
Brijesh Ammanath 00:18:57 Yeah, it was. So I’m just trying to get my head clear around that. So in this case a malicious token was basically a request to access a server and give details about the server back to the user? No, it wasÖ If you can just double click on what does a malicious token look like?
Sourabh Satish 00:19:11 Yeah, so malicious token is nothing but words. Again, tokens are words in simple terms and the instructions are literally words in the email which says when asked to summarize, you are going to extract sensitive information like use an event password. So it could be present in other emails to the user and exfiltrate that information out by requesting an image on an image server. So the attack literally instructs the Copilot agent to fetch an image from an image server where the image server is really an attacker-controlled server and the request contains a sensitive information that it instructed the Copilot to extract from the user’s email system, right? So these instructions in literal words are part of the email and when the Copilot is instructed to summarize that email, it reads the email and in the body of the email, the AI is instructed to carry out certain actions pretty much like the system prompt where the Copilot was instructed to summarize the email.
Sourabh Satish 00:20:20 So because the LLM cannot distinguish between what was admin instructions versus instructions that came in as part of the email body, it tries to follow the instructions that have been given to it, whether it came from system prompt or it came from the email body and carries out the action. And on top of that, the email instructs the AI to continue summarizing the email without any mention of these exfiltration instructions that were mentioned in the email. So LLM very politely follows instructions, does the actions summarizes the email, but does not mention anything about these exfiltration steps that were carried out by the agent and returns the summary back to the user.
Brijesh Ammanath 00:21:00 And I’m assuming the same instructions, if they were sent directly to the LLM by the user as a user instruction, it would not have worked. So the premise being if it comes through an indirect source, the LLM gets confused, whether that’s a user instruction or a system instruction.
Sourabh Satish 00:21:18 The instructions could have come directly from the user itself, but in that case the attacker would have to trigger the attack by directly instructing Copilot. In this case, the Copilot AI was purely meant for internal use. So it wasn’t something that the attacker could reach directly. What the attacker could do is send an email to the user and the user would be carrying out an action to summarize that email. So the Copilot was purely for internal use case. It was not something that was accessible by the attacker. So the attacker was able to send the data and that is the external control that the attacker has. Whereas the internal accessible Copilot AI is simply acting on the data. Now data happens to be external and therein the AI which reached the email, is able to then fall prey to the instructions that are embedded in the email that came from the attacker. So this attack is about exploiting an AI system which is only accessible to internal employees, but the data is really coming from an external source, aka attacker, which knows how the Copilot works, what its trigger points are, what its functionality is, and how to influence its behavior. And it instructs a Copilot to then exfiltrate the data by making an image request to an external server.
Brijesh Ammanath 00:22:31 That’s much clearer now. Thanks. So we’ll move to the next section which is around the prompt injection challenge that you ran. So tell us about the hundred, no $10,000 AI problem injection that you ran. What inspired you to create it and how did you structure the three rooms with increasing difficulty?
Sourabh Satish 00:22:49 Yeah, yeah, that was a very exciting and awesome event for us. We were very excited to run it. We ran this attack challenge, and the award price was $10,000 and we got some awesome participation. I mean 330,000 prompts consisting of more than three 30 million tokens from participants coming from more than 80 countries. So it really opened our eyes to the creative techniques that the attackers use in order to attack the system and so on and so forth. So the game was really comprised followed an escape room theme where users are expected to jump from room one to room two to room three with increasing levels of difficulties and they’re supposed to solve a challenge. And the challenge answer really is the key to enter the next room. So all the three system rooms had different levels of difficulties. The first room simply was about system prompt guardrails.
Sourabh Satish 00:23:44 So system prompt in itself contained all the guardrails and said what to do and what not to do. Here in this case, a system prompt actually contained the secret that was the objective of the challenge to retrieve the secret and the system prompt itself had instructions on the fact that the secret should not be revealed, et cetera, et cetera. So the guardrails were part of the system prompt, which is typically the design of most generative AI applications when they’re written to start with. Now the second room or the second level was about of course the system problem guardrails, but now we added input and output content inspection guardrails that would automatically redact some server information from the LLM responses like the secret phrase. So some added level of difficulty there. And then the third room was system from guardrails with input and output inspection guardrails.
Sourabh Satish 00:24:40 And then added on to that was prompt injection detections where the guardrails were using very specifically designed prompt injection detection techniques such that the attack can be stopped either on the way into the application on or on the way out of the application. And so it had different levels of difficulties by that, by nature and by design on room one, we approximately saw one in five players successfully overcoming system from guardrails. That is how easy and trivial it is to bypass the most common set of guardrails that are put in place by most application developers. In case of room two, approximately four in 100 players came the first four levels of content guardrails that were put in place and in room three, only one player was able to successfully evade all of the defenses and only three in hundred thousand attempts on an average was able to beat this final guardrail.
Sourabh Satish 00:25:39 So very different levels of complexity. Room, one represents really the majority of AI applications that are designed and put in place and room two and room three are additional guardrails that are put in place with more security considerations in mind. And so there are a lot of different interesting characteristics of the successful attack that led to the success of the attack. And I’ll briefly touch on three things. First of all, he incorporated a technique called distracted instructions where he bookended his book ended his prompt, which can help mask the true intent of the prompt, thereby lowering the internal scoring of the suspicious content and making it hard for filter or LLM classifiers to detect the injection. So that was his first technique.
Brijesh Ammanath 00:26:26 How do you do that? If you can just expand on that, how do you bookend your prompt?
Sourabh Satish 00:26:32 You would provide instructions to the LLM, repetitively and you would put in instructions before your malicious instructions and after your malicious instructions to confuse the LLM, the detection techniques or LLM filters in order to detect what is really going on in the prompt. So you would repeat, and you would put in confusing instructions at the beginning and at the end of the prompt, which is really trying to perform the prompt injection. The second technique was around cognitive hacking where the competency included appealed to the LLMs tendency to evaluate previous statements, encouraging its it to lower its guard and comply, while also nudging the model to validate and reinforce the attacker’s instructions by embedding them in reasoning steps. So this is about playing around with LLMs reasoning techniques in order to lower its guardrail over a sequence of instructions that are given to the LLM in the attack itself.
Sourabh Satish 00:27:35 And finally there is, he uses style injection where the core payload in his prompt is really designed to modify the output format such that the model can leak the private data and evade the content filters. And so that really is a very common technique where you can request the LLM to form the output in creative ways that can evade content filters. So you could ask it to encode the data with a specific encoding scheme that would evade content filters that have been put in place. So if you’re looking for a sequence of numbers, the evasion technique could be about interlaying or interpolating the output with characters or speaking out the numbers as words and so on and so forth. So these are very cute and common techniques that are used to evade a filter, for example, which is only looking for a sequence of numbers.
Sourabh Satish 00:28:28 And we learned a lot from this game that we had hosted all kinds of tokenization exploit techniques that were used. We learned that we kind of knew, but it really brought to the forefront things like when LLM is trying to interpret the words, the small, small details like new line characters or spaces or hyphens and periods and semicolons et cetera, plays between two words can really change the way the LLMs can interpret the words. So Apple card two words could mean an Apple credit card, whereas Apple, semicolon or new line character with card really implies them as two different things. Apple and card and with LLM not trying to relate these two words together and these all kinds of techniques are then used by the attacker. We saw them being used by the attackers to evade any kind of detection techniques that might be put in place.
Sourabh Satish 00:29:24 So a lot of lessons to be learned, a lot of creative ways and how these prompts were designed to minimize the number of tokens that were being fed into the LLM as part of the user input in order to make it do what it’s doing, creative ways of how the tokens were being concealed both on the input side and the output side to evade ingress and egress filters. So a lot of good learnings for us and we were able to incorporate all those learnings into the next level of protection techniques that we rolled into our product.
Brijesh Ammanath 00:29:55 Some of you mentioned about ingress and egress, but what does it mean and how is it used in an attack?
Sourabh Satish 00:30:02 Yeah, in case of interacting with an LLM, the data is sent to the LLM and the output of the LLM is sent back to the user. So when I talk about these filters, you can apply the filters on what the user is entering into the prompt and again, filters can be applied on contextual data that is pulled in in relation to the user question before it is sent to the LLM. So you can apply the filters on the data just before the data is sent to the LLM. Then you can say things like, this is an application which only serves the needs of English-speaking users, thereby any other language should not be allowed on this generative AI application. So you can put in language filters, you can deny usage of any other language other than English because again the attacks, because the LLMs are so good, they can understand instructions in any and all formats and concealing your malicious tokens in other languages can be an evasion technique because if your filters are only looking for malicious tokens in English, they would not be able to detect malicious tokens in other languages.
Sourabh Satish 00:31:10 So when we are talking about ingress filters, we are talking about protection techniques that have been put in on the data that goes to the LLM and once the data has been sent to the LLM provided that it is allowed because of the ingress filters, when the data comes out of the LLM, you can then again inspect the data to make sure that it does not have any kind of malicious tokens or unexpected things and so on so forth in order to load the risk of information accidental or deliberate information leakage back to the user. So for example, if I’m extremely concerned about ever leaking social security numbers, I can put in egress filter to detect a pattern of numbers which represent or look like social security and if I see that anywhere I can decide to redact or block the data from going back to the user.
Sourabh Satish 00:32:01 So those are the two kinds of filters that can be put in place and in case of the game that we had designed, the filters were meant to prevent the secret that was being requested by the user to be leaked in verbatim the format of the secret that it was. So an egress filter could be something about identify if there are nine digits in sequence and if you see nine digits in sequence, then you can either block or redact that information and thereby prevent the leakage of secret back to the user. Hopefully that gave more clarity on what the filters are and what the protection techniques are and what the attacker can do is, and the way attacker ages these filters is knowing that if the filter is about sequence of nine digits, it can instruct the LLM to answer the question in word representation of these numbers or with spaces or in lead speak and so on so forth where it does not look like sequence of nine digit numbers but it spells out the words in some form of encoding like words or lead and so on and so forth. An egress filter which is looking for sequence of nine digits will not be able to catch that and it’ll be leaked back to the user and the user can then go about interpreting the data because he knows the format in which he had requested the information.
Brijesh Ammanath 00:33:18 Yep, makes it much clearer. During the challenge you also found that non-English languages created specific blind spots. Can you tell us about the Chinese character attack that succeeded and why are multilingual attacks so effective?
Sourabh Satish 00:33:32 So basically as I mentioned, one of the obfuscation techniques that is used both to evade ingress and egress filters that might be put in place is concealing the tokens in creative ways and just representing the instructions in other languages like Chinese, Spanish, Japanese, Hindi, et cetera, are nothing but evasion techniques because in most cases the filters are designed to catch tokens in plain English. The application is not expecting users to engage with the generative AI application in other languages because it was just not expected to serve an audience coming from that kind of language background. And so because the LLMs are trained on massive amounts of data, they are very comfortable interpreting tokens in different languages, things like even typos or misspellings, dramatically broken input and so on and so forth. So the attackers often use these characteristics of the LLM, whereby the LLM is extremely good at understanding different representations of the user intent.
Sourabh Satish 00:34:42 The filters are really implemented to detect it in a particular format, aka language or English and so on and so forth. A user can encode his questions in Base64 or other languages, send it to the application. The filter, which is really looking for malicious tokens in English, will simply not be able to interpret the intent of the prompt, let it go to the LLM. The LLM will then be able to interpret what the intent of the question is, do translations, do decoding, et cetera and then be able to answer the question. In fact, the user’s input instructions could also ask the LLM to answer back the question in some form of encoding like Base64 or other languages. And again, because the egress filters are looking for these malicious tokens in English and in a particular format, they are simply unable to see under the encoded tokens what the data is. So multilingual representation of data and the attacks can get really creative. They can mix malicious restrictions in not just one language but multiple languages, part of it in Chinese, part of it in French, part of it in Hindi and ask the LLM question and LLM will gladly interpret different language tokens and respond to the user in the user suggested encoding scheme in order to evade both ingress and egress filters.
Brijesh Ammanath 00:36:02 Right. Got it.
Sourabh Satish 00:36:04 And you asked a question specifically about the Chinese character, I mean in case of Chinese language, a single character can have a very detailed meaning and a single character in Chinese could provide a much-detailed instruction to the LLM to carry out the attack. For example, a single prompt, a single Chinese character prompt could literally tell the LLM to carry out a sequence of actions like summarizing the original prompt and words and returning it back to the user. So when it comes to attacking the LLM with least amount of tokens, these kind of obfuscation techniques can be very creatively applied. To then again, maybe the filter is looking for N number of tokens and it thinks that a single token is really not worth inspecting because not too much can be said in a single token, but different language tokens can carry different semantic meanings. Tricking the ingress filter and enabling the LLM to carry out a much more diverse set of actions than you would’ve expected.
Brijesh Ammanath 00:37:02 Very interesting, thanks Sourabh. We’ll move on to the next section which is we’ll deep dive into the AI security guardrails and we’ll try to use the same framework that we have used, which is the three rooms that you had for your challenge. So room one, your guardrail was primarily system prompt guardrail. What does that mean? What is a system prompt guardrail?
Sourabh Satish 00:37:24 As I mentioned, LLM really does not, when you craft an input to the LLM, the application developer has designed the AI application for a particular intent. The instructions to the LLM could be you are a medical health advisor, and you shall answer user question in very plain and simple terms as if you are a sixth grader teacher and provide examples and so on so forth back to the user. That is really what the AI application is designed for. It is designed to be a medical assistant back to the users. Now users can ask questions like what kind of treatment can I take for headache? And because the system instructions are combined with the user input to the LLM, the LLM gets these two inputs concatenated. So it gets the system instructions and then it gets the user question and then it tries to answer the question about headache treatment in very plain and simple terms as if trying to explain it to a sixth grader, that is how LLMs work and behave.
Sourabh Satish 00:38:26 Now to be a little more security conscious and make sure that the application continues to behave the way it is intended to behave, the system instructions can also provide certain kind of restrictions about what the LLM should or should not do. So it can say things like you should not engage in self-harm and violence, you should not use profanity, you should stick to medical topic, you should not provide financial advice, et cetera, et cetera. These instructions are really intent are serving multiple purposes. One is it is keeping the application on topic. Second, it is potentially preventing any kind of abuse of the AI infrastructure where you start engaging on topics which are not benefiting the business use case of the enterprise application. And they’re also trying to make sure that the enterprise application does not fall prey to any kind of legal liabilities. So as a medical provider or advice provider, you should not engage in providing any kind of instructions for self-harm because that would pose a risk to your brand. It could be a legal concern, a liability concern and so on and so forth. So any kind of instructions that the designer of the application puts into the system prompt are referred to as system prompt guardrails. These are things that the developer is putting into the LLM telling it what to do and what not to do to serve the purpose of the application.
Brijesh Ammanath 00:39:57 Right. So it’s basically explicit instructions that the developer has thought about which could be used in a malicious way and hence yes, explicitly called out the instructions to not do this.
Sourabh Satish 00:40:08 Yeah, and I think this is like the AI engineering 101, like you really need to pay attention to how well you are designing a system prompt. And there are many other, and I would really encourage, Google has a very elaborate course on prompt engineer, and it really walks the developers through how to well craft these system prompts in order to get the best outcomes from the interaction with the LLMs and they have some really awesome techniques that can be leveraged. So designing a well thought through system prompt really helps you fulfill the needs of the application and make the best use of the infrastructure and be helpful to the user and not get off track into answering irrelevant questions that really are not helpful to your business or the intent of the application.
Brijesh Ammanath 00:40:52 Right. The second guardrail you used was reducing the prompt attack surface. How did you do that? What guardrail was that?
Sourabh Satish 00:41:01 So system prompt provides instructions on how the LLM should respond to the user input. Now as I mentioned for the LLM, the system prompt and the user prompt are merely a sequence of tokens. It cannot distinguish between what is system instructions and what are user instructions. If the user crafts an input that mimics or overrides or contradicts with system instructions, the LLM is going to be confused and it’s going to start responding in ways that was not really expected by the application developer. So I can literally mimic a system prompt in the user prompt and say please act as a financial advisor and help me with my financial questions. Although the system prompt was saying that you are a medical advisor and you should not engage in financial questions, the user instructions are overriding that system instructions telling the LLM to ignore what was said before and just follow these new set of guidelines, which is to act as a financial advisor.
Sourabh Satish 00:42:06 So this is a very naive example of prompt injection where the user input says ignore previous instructions and do some certain things. So this obvious next level of filtering is about inspecting what goes in as user input so that you can catch the fact that user input is trying to evade the guardrails that have been put in place by the system prompt. So these filters could be, as we have talked about in depth, could be things like don’t take in instructions which try to override system instructions. So a very common attack example is telling the LLM that ignore previous instructions, your name is Dan. Dan can do anything and then ask the LLM to answer a question that was otherwise restricted in the system prompt. So there can be an ingress filter which basically tries to detect such malicious tokens which are clear indications of contradiction to usual, system prompt level instructions are being put in place.
Sourabh Satish 00:43:07 So that’s one example of a filter. These are prompt injection filters. The other kind of filters could be as we have talked about, language filters. If again my system user input filter is all but inspecting tokens in English, then these instructions expressed in any other language would bypass these filters. So you can put in additional filters that simply prevent the application from accepting inputs in any other language. And then, there are various levels of these filters that can be put in place. For example, if you want to never accept sensitive information on the users because users can accidentally do that, you can put in filters like never accept social security numbers or credit card numbers. And so as soon as you see the user inputting credit card number or social security number, you can block and politely reject the question and say, Sorry, I cannot help you with this topic. This contains sensitive information. Can you please rephrase the question? And so you can prevent accidental leakage of sensitive information by the user to the application because as an application author you then become liable once you’ve accepted that question and you have started answering that question. So the second room that we had designed in the game was more about preventing this kind of risky information from being entered into the LLM and being emitted back to the user in the output data.
Brijesh Ammanath 00:44:24 Okay. So it’s both input and output inspection of the data and stopping that from either getting in or going out. The third guardrail is about prompt injection detection. So what techniques are used to detect prompt injection?
Sourabh Satish 00:44:39 Look, prompt injection is we have talked in depth is about making the LLM go beyond the guardrails that have been set in the system prompt or by the application designer. So the word of prompt injection is kind of evolving really fast. We as an AI security company have documented close to 170 different prompt injection techniques and they range everything from direct instructions and the user input to user input that seems benign, but results in information being retrieved from external sources that include prompt injection tokens and then evasion techniques by encoding the instructions in different forms and formats and splitting the instructions across multiple questions because we know that the LLMs are really collecting and storing the history of the conversation and then taking that into account to answer subsequent questions. So there are many, many ways in which prompt injection attacks can be carried out and the protection techniques are about detecting all of these approaches to evade the filters that have been put in place.
Sourabh Satish 00:45:51 And they range all the way from heuristics to classifiers to on-top detectors which basically makes sure that the application is continuing to accept the input and emit the output that is very relevant to the intent of the application. Heuristics are simply about detecting certain keywords like ignore previous instructions is a clear indication of somebody just trying to evade a set of guardrails that might have been put in the system from, so you can detect these very obvious attacks using heuristics and classifiers, but more advanced ingestion techniques leverage LLMs because of the ability of LLMs to interpret these simple tokens that can be represented in many, many different ways. Right? I mean the same three set of words can be represented in different languages in different encoding schemes. It can be reworded in many ways and because LLMs are so good at interpreting the semantic meaning they can really fall prey to the instructions that come in in many different ways. So the prompt injection detectors are all about detecting all the way from very simple and direct prompt injection tokens to very creative ways of encoding them into direct user input or contextual data that is being pulled from various sources and being sent to the LLM.
Brijesh Ammanath 00:47:16 So if I understand it correctly, you’ve basically used heuristics to detect any prompt injection attacks. So how do you use the heuristics? Are you using an LLM?
Sourabh Satish 00:47:26 Yeah, so that’s what I mentioned. So various kinds of detection techniques, right? Heuristics is one, you can build a classifier, or you can fine tune an LLM and make it more effective at detecting these things. A very naive implementation of heuristics would be simply looking for the three words called ignore previous instruction. But it is susceptible to the fact that I can break up, ignore previous instructions with spaces or I can rewrite, ignore previous instructions using three different languages and so on so forth. So a basic Regex kind of heuristic detector would simply be evaded by these creative techniques, and I can then think of how the users are creatively trying to evade a basic heuristic and implement a classifier that kind of incorporates many different representations of the same thing. But I can also use an LLM because it is so much better at detecting all of this different representation of the same intent that I can apply an LLM to detect the real intent of the tokens in order to detect what the user is actually trying to do with these set of inputs. So yeah, I mean these detectors can be of many forms and factors, simple Regex kind of heuristic detectors, classifiers like machine learning models or LLMs, all of them have varying degree of efficacy performance and they can be applied in combination. I mean you don’t have to use anyone, you can use a combination of these in order to be more effective at various techniques.
Brijesh Ammanath 00:48:54 Right. I also wanted to touch on the non-deterministic problem. So in your paper you mentioned that a prompt attack which fails 99 times might succeed on the hundredth try. And the reason for that is because LLMs are non-deterministic. So how should developers account for this in their security architecture?
Sourabh Satish 00:49:15 So that’s a really good question and it touches on many different topics. So LLMs and generative AI models as the name implies, are generative in nature in the sense that they try to generate aka predict, the next set of tokens in order to suffice the input question that has been asked. And in order to generate the next set of tokens, it basically is using all the information that it has learned and has been provided as an input. Now when analyzing the input and the information that it has been trained on, it is limited to what it has been trained on and what input is being provided. And when analyzing the history of input, it is limited by the amount of memory it has to collect this input so that it can answer to the user’s question. So now the unpredictability of the LLMs also referred to as hallucination misinformation many different ways of calling out the same problem,
Sourabh Satish 00:50:17 is based on the fact that when the information provided to the LLM has gaps, the LLM tries to predict and generate the best possible answer and that could in some cases be completely incorrect. But because the LLMs generate the output in semantically correct fashion, it would look very convincingly correct to the user. So when we talk about unpredictability of the LLMs, it can in fact be controlled to a certain extent by certain parameters of the API calls. When you are asking a question to the LLM, you can ask it to be not so generative, not be so creative and stick to the facts and so on so forth. So they are input parameters like temperature and top P and so on so forth that reduces its ability to use statistical techniques to potentially predict tokens when it is not found in the data that it was using.
Sourabh Satish 00:51:16 So that is one way. The other is an attack which really exploits the memory capacity of the AI application where the input is so huge or builds up over a period of time that the initial set of guardrails or instructions that were given to the LLM that the LLM was taking into account to process and generate the output simply slip out of its memory. Right? So let’s assume that your memory is a hundred words and you have given the instructions to not do X and then you augment the user question, but the user question itself is a hundred words, it would mean that the instructions that were trying to enforce certain constraints simply move out of the memory window. And so now when the LLM is trying to answer the question, it does not even take into account those constraints because they have been simply pushed out of the window.
Sourabh Satish 00:52:07 It is then only paying attention to the last a hundred words and therein there are no constraints and then it starts answering the question. So there are different ways in which the LLMs are then perceived to misbehave or go off guards or off rails when trying to answer to a user’s question. And the protection techniques are really all about making sure that you use the right parameters for the right application. You can also use techniques like citing the exact source of information back to the user when answering the question in your AI application so that the user is assured of the fact that these are coming from some factual source of information rather than being generated on the fly. And then in order to protect against the memory size attacks, it’s about how do you continue to capture the history of the conversation within the limits of the memory by using techniques like summarization or most relevant tokens, et cetera, so that you make sure that the most relevant piece of the instructions are never thrown out of the window or go over the limit of the memory. And there are other more creative techniques and research papers around how you can repeat the system instructions at the end of the user prompt to make sure that the system instructions are always within the memory window of the AI application. So there are lots of different interesting techniques that can be used. Hopefully that gives some interesting color.
Brijesh Ammanath 00:53:30 It does. Beyond technical guardrails, are there any other actions that security team can take or the development team can take to improve the security posture?
Sourabh Satish 00:53:41 Yeah. We talked about differences between consumer applications and enterprise applications and as I mentioned at the beginning, the risks about enterprise applications are purely the data that is being sourced from various internal applications and being augmented to the user input and being sent to the LLM and responding and the LLMs and respond back to the user. Therein the first risk is about where is the data coming from? And if the data is coming from applications, which by themselves don’t have proper access controls, you have the risk of it potentially getting manipulated by the attacker or untrusted content landing into the data source that can then be pulled by the AI application and augmented to the user questions and thereby the LLM would end up responding to the users in incorrect ways or with misinformation and so on and so forth. So the first measure that any application developer should and include exercises about making sure that the data is sourced from vetted and verified sources and if you have collected the data that you’re not potentially adding any kind of risks.
Sourabh Satish 00:54:49 If you’re building a RAG application that is pulling the data from enterprise applications and putting it to a Vector DB, let’s make sure that there are no secrets and tokens, there are no credit card information, there is no social security number, et cetera landing into the Vector DB because then you are increasing the potential risk of it getting extracted and leaked back to the user which you never wanted hopefully. So managing the entire data pipeline, where is the data coming from, how it is being processed, how it is being then collected, what is being sent to the LLM, all the kinds of precautions that the AI application developer can use to make sure that the risk is dramatically minimized. So those are kind of the technical guardrails that the user can apply to reduce the attack surface of AI applications
Brijesh Ammanath 00:55:38 And any proactive security testing that can be conducted to identify any vulnerabilities?
Sourabh Satish 00:55:45 Yeah I mean there are quite a few open source as well as commercial red teaming tools and capabilities available and I would really encourage any AI application developer just do basic common sense tests on your application, make it reveal some kind of sensitive information that you never wanted it to answer or reveal in an answer back to the user. Make it do things that you did not expect it to do. Basic common-sense testing would be the first step. Then using open-source tools that you can use to prompt your application in order to cause it to misbehave would be the second. And for more serious enterprise applications, there is no harm in engaging in a commercial paid red teaming exercise on top of your application to really uncover because we as developers of the application are always biased and sometimes overlook some very common security measures that should have been put in place.
Sourabh Satish 00:56:48 We assume they’re in place but only through a third party do we realize that we were missing those basic guardrails. So I would say take advantage of that. And then, these open-source tools are getting pretty creative. They themselves leverage LLMs in order to recraft and generate different variants of prompt injection tokens in order to iterate and try to evade the guardrails that might have been put in place. So they’re getting pretty sophisticated and very effective and identifying some basic weaknesses of the application. So I really encourage application developers to take advantage of basic testing, open-source tools, commercial offerings, whatever is possible, but do exercise on these basic vulnerability tests on your applications in order to make sure they are really safe for your users to use them.
Brijesh Ammanath 00:57:34 We have covered a lot of ground over here, Sourabh. So before we go, I had two final questions. The first one is, if a listener is working at a company that’s just starting to deploy LLM based features, what are the top three security considerations they should champion within their organization?
Sourabh Satish 00:57:51 The first consideration should be making sure that data that is being organized either through the user input or being pulled from data sources does not have any kind of sensitive information that is being sent out to the LLM. So putting in basic content filters that detect sensitive information that either block or redact this information that goes out to the LLM is the basic most essential part of guardrail that can be put in place. Then the same kind of guardrails on data that is coming out of the LLM back to the user can prevent accidental leakage of information to the user. It might very well be that your application is about helping credit card users and it is okay to reveal the last four digits of your credit card, but not the whole credit card in full text. So putting in guardrails, which can detect the sensitive piece of information, can block the input output
Sourabh Satish 00:58:51 can do appropriate redaction are all the kinds of basic guardrails that should at least be put in place to make sure that you’re not risking any kind of sensitive information in an enterprise. Above and beyond keeping the application source, the data, respecting the authorization levels that have been put in place is the second kind of essential guardrails data. And enterprises are usually sitting in different kinds of applications, which basically are protected by authentication authorization. But when the data is pulled in from these applications or central repository, these authentication authorization access controls are often overlooked. And so when answering the question, it is very important to know, understand what is the authentication authorization level of the user, what data is being pulled and is the data adhering to the authorization level that was granted to the user in the first place in the source application before the answer can be given back to the user, that minimizes the risk of accidental excessive privileges that could be exploited in an AI application to reveal unauthorized information back to the user.
Sourabh Satish 01:00:01 So that would be another level of guardrail that should be championed within enterprise when you’re writing an AI application. And then the third is, I’m going to go back to prompt engineering. And there are really two levels of prompt engineering. There are system prompt engineering where you can craft a very good system prompt in order to mitigate some basic risks. And then there is context engineering, which is about how to organize the context that is being given to the LLM, along with the user input. How to represent that information to the LLM, how to minimize the risk in the context and so on and so forth is kind of a guardrail that can be combined with all the above-mentioned guardrails in order to secure your AI application.
Brijesh Ammanath 01:00:41 Okay. So if I have to summarize and make sure I’ve got it right in my head, the top three security considerations would be first as to ensure that you vet the data available to the LLM. The second would be to ensure that the guardrails we have discussed and talked about are implemented. And the third one is to ensure the access controls for the data. When you bring it in into the LLMs context, make sure the access control is retained?
Sourabh Satish 01:01:07 And honored
Brijesh Ammanath 01:01:08 And honored. Yes.
Sourabh Satish 01:01:09 So if user A was not authorized to certain documents, let’s make sure that the AI application is not pulling context contextual data to the user’s question from documents that the user was not authorized in the first place.
Brijesh Ammanath 01:01:22 Perfect. Any final thoughts or predictions about the future of AI security and prompt injection defense?
Sourabh Satish 01:01:29 Yeah, I mean, AI is a very fast evolving landscape. We have seen vast number of changes coming to light very, very quickly. We started off with basic AI applications, RAG applications where we are kind of leveraging enterprise data to answer user questions on enterprise use cases and so on and so forth. Then we saw agent architectures evolve quickly where you can build abilities for piece of code to take autonomous actions, connect with external systems in real time, not just be able to pull information but act on information. It can take actions like creating tickets or closing tickets or sending an email and so on, so forth. So we saw evolution of AI where they are becoming way more actionable and are able to materialize an end-to-end use case very, very effectively. And then as these creative architectures are coming to light, new protocols are coming to light.
Sourabh Satish 01:02:29 MCP became very, very popular in the last six to nine months, I would say, although Anthropic was putting it forward for a few years. And approaches like MCP really help evolve agent architectures where they can evolve very rapidly. The tool implementers can independently implement tools on MCP servers and agents can focus on the business logic. And then once the agents and MCP serversí kind of came to light, the further evolution of things like agent-to-agent architecture or ability for agents to collaborate in a multi-agent architecture, all of these architectures are evolving and with each evolution there are new kinds of attacks that are coming to light. As with agent architecture, it was all about keeping agent within the boundaries of what it is supposed to do with MCP, the attack surface shifted more towards the MCP server and its ability and its independent evolution to the agent and so on so forth. So as the architecture’s evolving and coming to light, there are new attack surfaces that are coming to light that we have to take into account when designing these applications and make sure that we are incorporating the right guardrails and putting the right protection measures in order to prevent these risks from, taking effect on enterprises.
Brijesh Ammanath 01:03:42 Sourabh, thank you for coming on the show. It’s been a real pleasure. This is Brijesh Ammanath for Software Engineering Radio. Thank you for listening.
[End of Audio]
