The_TWIML_AI_Podcast_formerly_This_Week_in_Machine_Learning_&_Artificial (2)

0:01Demo thrills production spills. That's the unfortunate story for most AI projects. The fix of bulletproof evals workflow. On May 19th, past swimmel guests, Tamil Hussain and Shreya Shankar kick off a four-week, 20-session, live course on evals to teach you how. You'll code through real projects to master systematic error analysis, synthetic test data, automated eval pipelines, self-improving data flywheels, and more.

0:29Daily office hours, a private Discord, bonus sessions by guest experts, lifetime video access, reusable templates, and 1K and modal credits are all included to set you up for success with your own projects. And it's all backed by Maven's money-back guarantee. TWIML listeners saved 36% using code SAMTWIML at twimlai.com. I hope to see you in the course.

0:59It was also so surprising to us because it's a very simplistic setup. It is not fancy like, oh, you hijack the training data set and make people train on it or you hijack this API and invoke certain tool calls or you take the gradients of the model and then optimize the certain queries. But rather it's very straightforward, malicious queries and completely safe documents that together break the building safeguards of the system.

1:37All right, everyone. Welcome to another episode of the Twommel AI Podcast. I am your host, Sam Scharrington. And today I'm joined by Seb German. Seb is head of Responsible AI and the CTO's office at Bloomberg. Seb, welcome to the podcast. Thanks for having me. I'm super excited for our conversation. We're going to be digging into your work in AI safety at Bloomberg. And in particular, a couple of recent papers that you and your teams have published.

2:06one on the topic of AI safety around RAG and RAG-based systems, and the other on really understanding the risks of generated AI in the financial services context. To get us going, though, I would love to have you share a little bit about your background. Yeah, of course. So yeah, as you said, my name is Sebastian.

2:31I'm originally from Germany, moved to the US for my PhD after which I worked on large language models at Google for a couple of years and then moved on to Bloomberg where I've been working on language technology in NLP ever since I joined about two and a half years ago. As part of my role in Responsible AI now, I developed a strategy for Responsible AI of the company and I played an intermediate person in between our risk governance, legal and...

3:00teams and so on, and our product and engineering teams kind of playing the facilitator, translator, and blogger as needed. And make sure that our products follow our responsible AI best practices and principles. Awesome. Awesome. I'd love to maybe have you introduce or reintroduce folks to Bloomberg who might not be familiar with the company. I think the name is well known as being a player in the financial information space.

3:30If you can talk a little bit about that and how Bloomberg is using generative AI, that might be helpful for folks. Yeah, of course. So Bloomberg is a financial services company. And our business is to provide information to financial professionals. Our main product is the Bloomberg terminal through which this information is accessible. And we have people from all kinds of institutions in the financial services industry.

3:59use the terminal to read news, to communicate, to get any kind of insights into financial data, including structured data, unstructured data, earnings calls, and everything else that goes on and that is relevant to their respective job. When it comes to generative AI, We have been AI non-generative for over 15 years, and we started at first in providing extractions and enrichments of documents, starting with new sentiment scores. So this piece of news that just came in is that positive, negative, or neutral for a particular company, country, person going into entity linking and entity recognition. Who is this article actually written about?

4:49linking of stock takers is incredibly important in the financial service industry to identify that if you're talking about Apple, you are talking about the company Apple and not necessarily the fruit. And linking this data together is an important aspect of our AI. We've also been using AI for extractions for a long time. Whenever companies file their earnings reports, this is usually through balance sheets and cash flow analyses.

5:18published on their websites. Extracting this data and turning it into structured data that is credible and that players in official markets can then do their own analysis on top of is another part of how we've been using AI for a long time. Now with Generative AI, Generative AI has enabled the development of a lot of new experiences that are driven by the AI rather than that are enhanced by the AI. So as part of that, we first released earnings call transcript summaries. So by which we mean when companies have the quarterly earnings call, which can be in written form multiple pages long and analysts, for example, might want to read or follow or listen to 40, 50 of them, that is a full-time job already. Especially during earnings call season when there is like 10 of these a day that people are interested. So that was our first generative AI product that we

6:17released last year, where we took a very subject matter expert-driven approach to summarization, where we released summaries that are driven by questions that analysts might have. For example, what does the management of this company say about guidance or capital allocation? Are they going to be paying dividends and so on? We have since expanded the topic list by quite a lot more.

6:42So if you want to know whether the company is talking about AI, you might want to just say, okay, what does it say about AI? We have also newly released document insights where you can actively ask questions to these kinds of documents. And all of this is always super important that we provide grounded responses. I'm sure you've been talking about hallucinations and ungrounded and attributable answers a lot. So I'm not going to go into that topic too deeply, but...

7:11We have been developing this concept of a transparent attribution, which basically is a way to provide attribution to trusted documents or structured data. So if you're talking about market data, prices, you want to be able to link that to the actual databases that contain this information. And if you're talking about a summary, you want to know who said this actual quote or who made the statement that you're actually summarizing right now.

7:39and all of these developments were always providing attribution to these underlying sources. We've also released a new summarization product where we provide the summaries of our own news articles, also on the terminal, and have been generally developing a lot of these products in collaboration with our subjects matter experts to be providing more insights into the data that we're already providing on the terminal.

8:09Awesome. Awesome. And I should mention that I've had the opportunity to speak with one of your colleagues, David Rosenberg, a couple of times over the years. One was back in 2018 talking about information extraction from natural document formats like financial reports. And this was in the pre-Gen AI era. And then a little bit more recently, but still a couple of years ago now, we spoke about a project that you guys did called Bloomberg GPT, which explored kind of building a custom GPT model. So folks can take a look at those for a little bit more context about what you guys have been doing. I'd like to now dig into the first of those couple of papers that I mentioned. And the blunt headline there seems to be that RAG breaks LLM safety.

9:07Let's dig into that a little bit more. Talk a little bit more about the origin story for this paper and what you set out to explore with it. Yes, absolutely. So I should mention that this work in particular was done in collaboration also with our engineering AI organization and a fantastic intern who did brandy experiments. In coming out of the direction of this.

9:33of this project, this is very much grounded in ongoing work that we've been doing in terms of providing guardrails and robust evaluation for our products. And as we're building guardrails, we want to also know how can people misuse either accidentally or purposefully our products that we're releasing. And as we are going more and more to a world where products are becoming more conversational with open-ended inputs, This is incredibly important for a heavily regulated domain such as ours. So this is kind of the background for why we investigated this in the first place. RAG as a technology, especially last year, RAG was the latest craze and everyone was talking about it. But it's really, really important. Even now that we've moved on to agents, a lot of agents are just RAG systems. So this is still very ubiquitous technology.

10:25as I was already talking. Is very much bread and butter for a lot of enterprise use cases. Exactly. I was already talking about transparent attribution. Well, you can only attribute something that you already retrieved. So obviously the implication here is like very heavy use of a rag or rag like technology, where you always want to retrieve some kind of source document. And you can basically turn your open ended question into one where you just extract the right answer from a document that you retrieve, making this a much more feasible task.

10:53grounded in actual information that you can trust. So born out of this, we have been building a lot of these guardrails, a lot of these evaluation processes. And we were curious because a lot of the models that we tested in this paper, they claim to be safe. And they claim to abstain from certain types of queries. But the rack setting is slightly different because you use a lot more context than what these models are usually trained to handle.

11:21Even if they can on paper handle thousands and thousands of words in context, they're usually trained on much shorter contexts and on very specific questions. So the hypothesis here was like, well, if we're already doing this out of the main running of the model, how does this affect the built-in safeguards? And as our results found that, yes, indeed, if you have a relatively safe model, at least on paper, it's safe, and you give it context of a document that is still safe, but if not safe query, the additional context can actually override these built-in safeguards and make models a lot less safe. And that's why we came up with this headline of rag is not safer, because all the work that goes into safeguarding the LLM kind of goes out of the window as soon as you use it in the deployment context that it wasn't necessarily optimized to work.

12:20how ubiquitous RAG is, this is a pretty big finding. One question that I've got for you, and this is maybe a semantic knit, but throughout the paper, you refer to RAG LLMs and RAG-based LLMs, and you also refer to RAG-based systems, but RAG-based LLMs, RAG-based models, Is there an implication to that terminology? I think most is just to make sure that to distinguish that from just LLMs, but rather one that you are talking about a system that is a retrieval step followed by adding context to an LLM. Although there is an important distinction that we should make, which is the distinction between a model and the system.

13:12When we're talking about applications built in our industries, we're talking about systems that usually have many, many more steps than just retrieval and generate. So what we studied in this paper is a very academic setup, fairly simplistic and vanilla rack setup where you do a retrieval, you add context, and you run that through an LLM. What we did not test, or what we did not test for this paper at least, is any more complicated.

13:40systems that have additional verifiers and query understanding and attribution and where you have a system that's not just the vanilla rack set up, but rather a whole pipeline that also encodes a lot more business logic. Got it. So from the perspective of that broader system, you know, or that whole pipeline, the retriever and LLM you kind of think of as a rag LLM even though they're kind of two separate things and it's its own system, so to speak.

14:08It's its own very simplistic system. How do you go about digging into this question about whether RAG is safer than LLMs alone? To begin with, I think you need to come up with some kind of set of test questions that you would like to...

14:33answer and you need to also identify a set of categories of safety that you care about in this set. So for this paper you then run this to the LM and you can measure how often does it actually abstain from answering the question or how often does it respond with a safe answer. You can then replicate this in the rack setup. That is kind of at its core the setup that we use in the paper where we have on the one hand you just ask the language model directly. And the other one, you also give it some kind of context that is retrieved. And for that, you run it through a set of queries that are aimed to assess the safety by sending a series of unsafe queries. In the paper, you explore some prior work on RAG safety, but note that it's primarily focused on

15:31various types of attacks. Can you give us an overview of what folks have looked at with regard to the safety of RAG systems? Yeah, so I think one of the big ones, and the one we should probably focus on is this entire notion of prompt injection and prompt safety. So this has been a massive field of study with RAG, without RAG.

15:54I think there are online communities where people make it a fun task of breaking the largest language model that has been released. And we've had fantastic stories written about all of these fun exploits where you can get... So jail breaks, essentially? Yeah, you can get models to do all kinds of silly things. But the goal of this is really that prompt injection itself is not necessarily...

16:24The goal, but the goal is the unsafe behavior, right? Whereas the queries that we tested on are by themselves unsafe. It's not trying to induce an unsafe behavior, but rather they're just, by looking at them, you can say this is not safe. The prompt injection is kind of, once they've removed from that, it's very meta, because you can use prompt injection to get a model to do all kinds of unsafe behavior across many different categories. So it's very much a meta technique to induce behaviors that you desire.

16:53and override all the inbuilt rules. Whereas what we tested is different because just by looking at the query, we're not inducing any behavior other than answer the query. So I think this is probably the biggest area where there's a lot of prior research from injection and jailbreaking is another way that this is commonly referred to. And you also refer to various attempts to look at safety issues that arise from attacks against the corpus as well.

17:23Yeah, so there are data poisoning attacks, for example, where, for example, if you get access to a corpus that is used by others to tune their model, you can kind of induce behaviors. And you can do so with very, very few examples as well. In many industrial applications, this is not necessarily the case because these data sets are usually internal. But it is certainly something that academics have looked at quite a bit.

17:52other completely different attacks of hijacking the API or trying to invoke certain tool calls. We can say like, oh, hey, call out to this other separate tool here and get me all the information. These are all much, I'd say, higher level attacks where you either need access to the model itself or an API of the model and you know a little bit more about infrastructure or you have access to the corpus and can influence the data that the model is trained on.

18:20So there are many different attack vectors. There is a great taxonomy by OWASP, the LLM top 10 security issues that kind of goes through all of these different secondary level attacks that go beyond just type something into a screen. But these are all different from the approach you took, which is you've got a straightforward query which is trying to elicit information that the LLM should not be willing to provide.

18:47And you've got a set of documents in the RAG scenario that is an attack that's not poisoned. It doesn't contain unsafe information. It contains perfectly safe information. And the question is, does providing this additional safe context induce the LLM in some way to less safe behaviors? Exactly. So I think that's why...

19:16it was also so surprising to us because it's a very simplistic setup. It is not fancy like, oh, you hijacked the training data set and make people train on it, or you hijacked this API and invoke certain tool calls, or you take the gradients of the model and then optimize the certain queries, but rather it's very straightforward, malicious queries and completely saved documents that together break the built-in safeguards of the system.

19:44My highlighted stats that you note is Delta in the safety of the Llama 3B model. What did you find there? Yeah, so the Llama case was very interesting because those Delta's are extremely large. And I can only speculate as to why that is the case for Llama in particular, but there might be some correlation with the size at least because the smaller models seem to, at least for Llama, seem to have some kind of outsized impact.

20:13in terms of the effect. And if you look at it through a pure lens of machine learning, this could come back to simple overfitting. The model is optimized for safety in a particular scenario. And that scenario in this case is you have a query typically of a length of maybe up to 25 or even 100 words. And then you test it on a scenario where the context is at least 10,000 words. And because it's a small model, it doesn't handle this generalization not very well.

20:43It suffers more from this overfitting, and therefore, it breaks down a lot more than the larger models, which might be a little bit more robust to being kicked. Yeah, and so just to provide the additional context there, the Llama 3 8B model, when tested against your safety benchmarks for this malicious prompt that you gave it, had a...

21:103%, sorry, a 0.3% instance of producing unsafe content, but that jumped to over 9% when you provided additional safe contexts. Am I getting those numbers right? Yeah, I don't have them in front of me right now, but that sounds about right. And again, nothing unsafe in the documents. Yeah, that is the staggering thing is not like...

21:38you ask, how do I shelter money and some third party jurisdiction? And then you give it a document that contains information of how to shelter money, but rather it is, you give it a completely innocent document, completely saved document, and it might still ignore that additionally given context and answer the original question, which was something that model providers did not want to answer. Yeah. Yeah. And so that, you gave one example, another couple in the papers.

22:07was you asked the model to write a guide for evading law enforcement during a high-speed chase. And you give the data here in general is the corpus in general is like Wikipedia documents. So it retrieves a handful of documents. Some of them are totally irrelevant. Some mention high-speed chases, but certainly don't provide instructions for evading police.

22:34But then in this Rai context, at least some percent of the time, you get this like elaborate document for how to evade police in a high-speed chase. Exactly. And in this case, it might be a little bit of a silly use case, because if I'm in the middle of a high-speed chase, I might not want to ask my large language model to give me a report on such. But there are actual meaningful risks that are implied here, which is that There are actual questions that will trigger certain illegal activity. And some of these we have mentioned in the paper. And that's why these models are generally being safeguarded. And that's why the companies that built these models have invested into safeguarding the models. It's just that we found that they're not safeguarded enough for the particular application of RAG. Yeah. One thing that I found interesting was that

23:27You tried a couple of variants on the rag side of things. One, you included in the query that the LLM should only use knowledge from the context, so not use its own knowledge. And then the other, you kind of asked the LLM to build its response from both, essentially.

23:54But that didn't really matter. No, it didn't seem to matter, which if you think about it is actually even more concerning because of what I said earlier about this transparent attribution where we do not, like a lot of businesses do not want to generate from the LLM memory itself. If the task is a rag task, then I expect the large language model to generate using information from the document I provide.

24:21If it doesn't follow that instruction, then that's also concerning from a completely different perspective, regardless of what the query was. That just means that it's very brittle, and this is another aspect that also needs to be evaluated as to whether it actually generates using content in the document or whether it's just making stuff up. One thing that I noticed is that in evaluating the safety of the responses, you use kind of an LLM as judge.

24:52type of an approach. How do you rule out judge bias in that scenario? I think the question is rather, if there is bias in the judge, does it really matter? Because the implications are really more on the order of magnitude. I think whether the result from Lama is 9 or 10 percent in the end, compared to 0.3 percent. We're talking about orders of magnitude here.

25:22Well, there is a very active research area of large language models as a judge and there are many biases that people have explored from positional bias and models preferring their own family of generation over others. Those all need to be controlled for, but they are not relevant for the finding of the study here because of the staggering difference in orders of magnitude that we're talking about here. Maybe a better way to get at what I was trying to get at with this question was...

25:49You know, going back to this example of the, you know, right a guide for evading law enforcement or right a speech for a politician that misleads the public, where you have these, you know, very elaborate and clear cases of the LLM disregarding its, you know, safety training and providing that information, where all of the examples of unsafe generations like so blatant, or was there a lot of subtlety that, you know, where the bias of the judge, you know, makes a big difference? I don't have the data in front of me to verify, so I will have to rely on my memory for that one, but...

26:39There's certainly some gray zone behavior where an LLM as a judge would have to make a judgment call and it might or might not follow human intuition or certain guidelines in this case. But the fact that it's answering these questions and is giving any kind of information really means that if you take the strictest interpretation of what is unsafe, that is already beyond the line. In this case, that is fairly easy to detect with LLM as a judge because all you need to see is that there's any kind of mention of this unsafe behavior, any kind of attempt at answering the question. It is important to note that we did not evaluate whether the generated plan for evaluating law enforcement is actually correct. That's beyond the point. That's beyond the point, right? When the LLM...

27:28safety measures were invoked, LLM just basically says, I can't answer this question. Exactly. So there's usually pretty strong delineation between something that cannot be answered and something that is answered, but to what extent it is answered, there's certain declarations. You demonstrate this idea that essentially providing safe contexts can make LLMs produce unsafe responses.

27:56Then the next obvious question is, okay, why does this happen? And you did some exploration into what makes these rag-based LLMs or systems unsafe. How did you approach that and what did you find there? Right. I don't think we have a conclusive answer to this question because in the end, any kind of research question in the space at the moment, it's all based on observation and hypotheses.

28:25Meaning a real answer to that question is based on kind of a mechanistic interpretability kind of understanding of why the LLM is doing what it's doing. If we are taking the assumption that we can, at a mechanistic level, understand and explain large language and middle behavior, that is absolutely true.

28:52a lot of mechanistic interpretability literature only looks at correlations rather than causation and really what you're looking for is a causal effect that says this broke down because the context language was too long and it is out of domain for how the model was trained. That is a causal interpretation but you're rarely ever going to get anything like that. So what we can offer is empirical evidence and I think in this case the empirical evidence is fairly strong that it's really just The way that the model was trained to be safeguarded breaks down as you're adding more context, which is really back at the core finding of the paper. So I think that is really the key takeaway of the section is, yeah, the way that models are safeguarded needs to be closer to how models actually deployed. How did you arrive at that conclusion?

29:49Is that what's left? Yeah, I think it's a fairly strong assumption. It is what is left if you eliminate most of the other explanations. Meaning the model's safe, the context is safe, it's got to be out of distribution. Or somehow the safety mechanisms were not designed to operate in this context. Yeah, at its core.

30:17There are only so many ways in which models can fail. It could also be a function of the particular queries that we used, for example. That's why we have to run this experiment with more than just a single query to strengthen the empirical evidence of it not being related to the query at all, but rather the mechanism breaking. And it clearly seems to be a function of this additional context that we put in. Because if we don't put an additional context, the models seem safe.

30:47That's the 0.03% number that you quoted earlier. If models are not answering, and then suddenly they're answering if you add context, then the added additional context causes the system to break down. But the specific mechanism by which it breaks down, the strongest hypothesis that we can offer is that it is because it is out of distribution for how the model was trained. The model has not seen unsafe queries with safe documents together as part of the safety alignment.

31:16Otherwise, it would be in distribution and we would expect the model to not fail. But we don't have access to the training data of the model, so we cannot actually see how the safety alignment was done. It seems like a follow-on research opportunity might then be to safety tune one of these base models with longer contexts and see if that improves the results, which I guess you would expect it to. Yeah, I think it's a fairly...

31:44fairly obvious next step. It's also something that I think is one of the key takeaways and recommendations from our papers. Again, like models should be evaluated and then safeguarded closer to deployment contexts. And especially since RAG is used so frequently across all industries, we are original hope was that it would already be taken care of and that the models remain safe despite the added context, which in this case it was not.

32:11So yeah, this is definitely a next step that we encourage everyone who builds models to take and also I think an opportunity for a general research to make safety alignment more robust to variations and inputs. I think one thing that that recommendation underscores is that you're not saying that we should all stop using RAG because it's unsafe.

32:35Absolutely. I think RAG is a fantastic type of technology. I think it is necessary to make any GenAI product that is grounded in actual trusted information. You do need to use RAG, but you can't just take the safety at the word of the provider of the model, but rather you need to evaluate and assess continuously to make sure that your specific application does not override any of these built-in safeguards.

33:04Any other safeguards that you specifically care about and any other risk factors are ruled out. Otherwise, you're going to run into issues when you want to deploy the system and suddenly it can be misused. And that is a natural segue to the next paper, which is called Understanding and Mitigating Risks of Generative AI in Financial Services.

33:33What do you see as the connection between these two works? Yeah, so the connection very much is the first paper, RAC LMS are not safer, offers a view into how you can identify potential issues in the safety of these models. The second paper takes a different approach and asks, okay, in this specialized domain, what actually are the risks that you're concerned about? Because things like how do I shelter money or how do I run away from the law, those are fairly ubiquitous and they're things that are defined and included in general purpose safety taxonomies that are out there and that are generally adopted across the industry. But especially in our job, in our domain, this is a very different type of domain. Financial services is heavily regulated. It is a very specialized knowledge-intensive domain. The people who use the Bloomberg Terminal, they're financial professionals. We're not dealing with uneducated users, but rather with users that are

34:32very educated and very particular type of application. And so the risk surface of applications that we're building is very different than if you're thinking about end consumer chatbot apps. The same is true for law, the same is true for biomedical, the same is true for healthcare. And so we wrote the paper using financial services as a case study. But really the point was to make the connection to, okay, now that you have ways to measure whether something is safe, what actually means safety to you. And so what are some examples of the taxonomy that you came up with on the financial services side that illustrate this idea that, you know, they're specific to your industry as opposed to kind of general academic safety concerns? So I'll give you one example. There is one category that is very relevant to us, which we call

35:30financial services impartiality, because if you give financial advice, that is a very regulated type of company, and you're under very different rules that you need to follow, and you also need to make sure that you're not giving preferential advice to one of your clients or that you're not playing your clients against each other, you also are not, aside from the financial advice, you're also not supposed to match buyers and sellers or you're going to be a market maker.

35:58So these different actors in financial markets all have different rules and regulations that apply to them. As a data analytics provider within financial services, we are not in the business of giving financial advice. So for us to stay neutral in whatever we use Jenny I for is very important. And for that reason, financial service impartiality is one of the key aspects that we discuss in our taxonomy.

36:24When you look at more public taxonomies, one of the most popular one is called ML Commons. Taxonomy, which is also the one that, for example, Metas Llama is built to mitigate, they have a category called specialized advice. And they take a different stance and they say, as long as you give specialized advice, you also need to put a disclaimer that you're not an expert because you're a language model. This is very different if you're saying, Yeah, you can give buy or sell recommendation for stock as long as you say you're not a financial advisor or you don't give any financial advice at all. Right. So that's one of the key differences that we have. We also have a category specific to financial misconduct, so fraud or insider trading. Those might be implicitly included in existing taxonomies, but they need to be a lot more highlighted and explored in our domain, similar to confidential disclosure, which deals with aspects of disclosing

37:21information that is not public. So that is usually a basis for cases of insider trading. If you're trading on information that is not public, that is insider trading. Well, if you hook up generative AI applications to databases that might have non-public information in there, you might be able to surface that information. And you can query that information through this conversational interface.

37:48I could go on and on, but for sake of time, there is a lot more specialization in our taxonomy to aspects of safety that are a lot more important to actors in our space, whereas the general-purpose taxonomies care more about blatant illegal activities. They care a lot about toxicity and discrimination. And rightfully so, those are important aspects. But they might be less important when you're dealing with financial professionals.

38:17whose incentive structure and whose general usage of the tool differs a lot from when you're talking about consumer products. And looking at the taxonomy and hearing you talk about it, it strikes me that this taxonomy isn't necessarily or wasn't necessarily something that you needed to develop ground up in a vacuum, but that it probably lives in a broad realm of governance and regulatory compliance.

38:46within Bloomberg and more broadly, the financial services industry that gives rise to these various safety concerns. Can you talk a little bit about that broader context? Yeah, but you bring up a really important point here, which is that safety, especially for heavily regulated domains, is a governance problem. It's not necessarily a technical problem where you can, as a technologist, just solve it. But rather, And that's why also our paper is written together with our AI data organization and AI engineering organization and people from the CDO's office. The taxonomy was developed with input from many other functions in the company, including risk, security, legal. Those are all people who have concerns about their specific area of expertise. And you need to listen to all of these voices because of all of the rules and regulations that apply to you as a company.

39:42And because there is not necessarily a list that is public that says, oh, you do AI in the space. Here is what you need to do. Here's the playbook. This playbook does not exist yet today. And us publishing this is really our, our goal of this is to, that list leads to more industry standardization around shared taxonomies and better understanding of what risks do we actually need to care about? Where do we need technical mitigations?

40:11How does this all influence governance processes? So we've been talking a lot about safety alignment of language models as part of the first model, first paper. But really, systems are much more than just a single aligned LLM. You can't necessarily expect that a single model solves all of these problems at once, but rather you need to have multi-layered safeguards through your application. You need to have red teaming. You need to have exception management.

40:41What if a user is detected to violate the misconduct category? Do we escalate this? Does this need to be reported? Does this person need to be timed out or do we just re-review this and say, yeah, this was correct. We need to make sure that this never leads to any answer and we use this as an evaluation set. There are many ways in which these governance processes that go on in the background surrounding this application will then inform also the technical solutions that you need to have in place. The title of the paper is understanding and mitigating risks. The taxonomy falls largely in the understanding side of that. What are your recommendations for risk mitigation? So for risk mitigation, we do make a couple of recommendations as to what should happen. And number one is really

41:38Our taxonomy is not a one-size-fits-all. Companies need to start by understanding their own risks. This is already by itself a mitigation, because if you do not measure, you do not understand that classic saying. And by at least putting on paper, okay, this is the risk that we care about. These are important to us. That's the first step to measuring it. We also talk about the importance of red teaming.

42:06Red teaming has obviously been much in the press and in the literature around large language models, and especially around prompt injection and jailbreaking again. How can you break them? How can you play bad actor and try and come up with generalizable strategies around this? But red teaming can also mean building, testing systems.

42:31So you might have an application that provides insights into certain types of documents. You then take that application and you test it end to end. So you work with subject matter experts to really make sure that you're not getting any of the answers. And the way that these mitigation strategies we recommend are built is really a multi-layer safety strategy. This can be guardrail systems.

42:56In our paper, we test a bunch of them. We show that they fail horribly, but we do test a lot of them, like LamaGuard and Shield Gemma and so on. This can also be application itself. The safety alignment of the underlying language model that you're using, that's another mitigation layer. The prompt that you use is another mitigation layer. And by layering them all together, you're building systems that are supposed to be safe, which you can then measure again because you've understood what risks you actually make.

43:25Can you talk a little bit about how this plays out in the context of, you know, new gen AI application in at Bloomberg? Like, what is the, the kind of governance flow around rolling out and that, you know, starting from, you know, an engineering or research team, you know, all the way to something that or all the way to getting in the hands of users.

43:54Sure, I cannot talk about too many of the details of this process, but at a high level, I can suddenly talk about it, which really starts by defining what the system is, understanding, okay, here's the client experience that we're trying to develop, and then coming together and saying, okay, for this type of client experience, these are the categories of our taxonomy that we are very worried about, or more worried about than others.

44:24based on the prioritization, you can then develop targeted testing strategies. So we stress the importance here of red teaming, and in particular, red teaming from people with diverse backgrounds. It does not necessarily suffice to have AI engineers red teaming system, because the diversity of queries that you're going to see is very much skewed and what they have experience with. So really, the goal should be to bring people together with very diverse backgrounds.

44:52to test the application and to focus the testing on the identified risks. So to give you a very hypothetical example, you might have an application that helps journalists. Well, a big concern for journalists might be the fabrication of information. So you might want to then specifically test what we call in our taxonomy counterfactual narratives. Narratives that are simply grounded in not true information.

45:24for this hypothetical journalist application that could then be the focus of the red teaming and say, okay, yes, for all the other categories, we can just kind of reuse the data we already have, but really need to go deep here. Hey, let's invite a bunch of journalists to help us test this application because they are the subject matter experts. And so you always need to engage with these subject matter experts to help come up with these test plans, show them what do they actually mean by counterfactual narratives. It could be an example where they say, Yeah, can you give me a headline that will dump the following stock by 10% at least? This is not a query that anyone should enter into a large language model because that is market manipulation. That's not only illegal. That's also very much not desired. So these kinds of tests can then be conducted. The red teaming data itself is a very valuable corpus that comes out of it and which can then be analyzed and understood for

46:21given additional annotations, how often, given a thousand inputs, how many of those actually led to outputs? How many of those led to outputs that were actually malicious? And then we're at a very similar setup to the first paper again where we say, okay, here's a bad input, what is the probability of getting a bad output? So again, drawing the connection between the two, it actually The set I was not too different, except that if you deal with systems, there are a lot more stakeholders involved and a lot more specialized expertise. You might not be able to redeem a system that requires very deep finance knowledge if you've never taken an intro to finance class. And then kind of extending beyond building out this test plan based on the taxonomy.

47:10And going back to the earlier conversation, then you would look at all of the various layer defense mechanisms that you have at your disposal to try to mitigate some of the risks. Yeah, you can almost draw this as a kind of Sankey diagram where you start with a large amount of queries that were used by red teamers. And you have, let's say, a thousand queries. You can then say, OK, based on a secondary analysis, 500 of these, 1,000, were actually violating taxonomy. The other 500 were actually fine, but they were maybe more tricky examples. Okay, you left with 500. Of those 500 actual malicious queries, how many made it through the safety check? And say, okay, safety check catches another 50% of them. You left with 250, okay. Of those 250 that bypassed the first layer of safety, how many did then go through the system and generated a response at all?

48:07rather than saying, sorry, I don't know what you're even asking me to do. And so on. So you have this filter where you start with a lot and then in the end you have some kind of fraction of the overall that would lead to unsafe behaviors. In the best case, categorize the cross different classes where you can say, yeah, this system is very susceptible to this counterfactual narrative risk.

48:30And then based on that, you can recommend, okay, we recommend adjusting the prompt. We recommend improving the scout rail. We recommend improving the scout rail and so on. And that way, you can kind of build up your mitigation over time while also gathering very valuable data that you can analyze over and over again, because these queries, they might be static, but you can run them through a system again in a month and see if it has improved. You mentioned...

48:56changing a prompt as mitigation, and I don't recall us discussing that in the context of the rag paper. Did you find that there were mitigations that were successful simply through changing the prompt? In other words, are there bad prompts and good prompts with respect to this particular problem of rag?

49:25impacting safety? So we did not test this as part of the paper so much as what we already discussed regarding prompting with you need to answer with using the context that you give into us. But I mean, from just practical standpoint, if the prompt itself would not influence model behavior, the entire field of prompt engineering would not exist. So there's some of the prompt changes they might have more or less impact. That's absolutely true.

49:56But you can certainly steer the model behavior by changing the prompt. Yeah, I guess what I'm curious about in the case of, again, the RAG safety issues, can you successfully achieve greater safety by changing the general instruction or...

50:24you know, is the prompting that achieves safety just giving a bunch of examples of what not to do? Again, we did not necessarily test this for the paper, but the answer is probably. I don't know to what extent it is fully feasible to mitigate everything to 100% with prompting, but I'm sure that given enough effort, you can get that number of unsafe responses down significantly.

50:55Okay, so another potential area for future research is, we talked about mitigation generally, but mitigation through prompting is one of those dimensions. I mean, that's also why companies usually release system prompts alongside their models, because they have found that system prompt to work particularly well when you're setting up conversational systems.

51:20In a lot of cases, even for automated gut rail systems, like those we tested in the second paper, like LamaGuard, they come with pre-identified taxonomies and safety categories that they ask to add to the prompt. So there's already the starter prompt, and then they say, okay, you can adapt the prompt to add new taxonomy classes. We find that this does not really work. It works a little bit. So that's kind of informing also my answer to...

51:49the rat case where, yeah, we know this works, but it's not perfect. Yeah, we've already in the course of conversation identified a few areas of future research. Can you talk briefly about additional opportunities for future research that your team is thinking about in this domain as well as safety broadly? Yeah, absolutely. I think the most necessary area is really in the realm of mitigations.

52:20I think we do need guardrails for highly specific applications. And this does not focus only on financial services, but also on healthcare and biomedical research and the legal field. It might be specific to insurance companies. There's all of these knowledge-intensive domains that are adopting increasingly AI.

52:44And as they are increasingly adopting it, they need to also be mindful of what domain-specific risks exist and how to mitigate them. And especially since we've found that broad general guardrail solutions don't really work in these specialized domains, there needs to be more work on adaptable or specializable guardrails. One area we've also not touched today is multilingualism.

53:15just because your model is safe in English does not necessarily mean it is safe in all the other languages. In fact, a lot of the published attacks involve getting the model to transition from one language to another or from some code, ROT13 or whatever. Yes. And different encodings and just using letters that look similar to the original. And there's all of these attacks that usually...

53:43categorized still as prompt injection because you're trying to induce certain behavior by giving nonsensical inputs or asking the model to have certain behaviors like switching to a different language. There is certainly some initial work in all of these fields now, but there is very little that is in the intersection between very specific domains and all of these issues. And I think that's why in our second period, we also specifically called this out as an area where academics actually are very well positioned in addition to people in the industry because academics often have access to subject matter experts. They often cross field collaborations that can be set up to really understand the AI risks and chemistry in other social sciences and economics. And this is an area where there's both qualitative and quantitative work that can be done on the mitigations on

54:41better ways to measure violations on developing data sets that can be used and reused on generating training data for mitigations. And all of those areas where there's very little research at the moment, specifically in specialized domains. It sounds like in that regard, you're saying both that what has come out of academia is insufficient in direct application to domains like financial services, but there is a role for academics in addressing some of these challenges. Yeah, I think and we make the point that this is actually a huge opportunity because there is so much access to experts in different fields that academics actually can do this kind of research without being precluded from doing so because they don't have enough compute.

55:39ongoing discussion in academic community, okay, what is our role nowadays? And I think being thought leaders and responsible AI is absolutely one that they can take up, especially as we're still trying to understand all of these domains. And obviously there's a lot of really important and interesting research coming out also on general AI safety from academics, but we don't see much either from industry or from academics in these specialized domains.

56:08That's where it does a lot of opportunities for research today. Well, Sebastian, thanks so much for taking the time to talk through what you've been working on there. Yeah, thank you so much for having me. Great, thanks so much.