CIO Playbook:
Agentic AI in the Enterprise

How AI is breaking enterprise IT assumptions

Anthony Scriffignano: One of the fundamental tenets of testing is don't change two things at once. Well, we're changing thousands of things at once right now.

Michael Krigsman: AI agents are running wild inside the enterprise, and CIOs are on the hook. Anthony Scriffignano is a PhD data scientist, and Tim Crawford advises CIOs at the world's largest companies.

Anthony Scriffignano: The way that we used to think about application development, the way we used to think about testing, the way we used to think about data stewardship and ingestion, the way we used to think about governance, every single one of those things is changing. And then there's new domains that are coming into this sphere, like context engineering and understanding black boxes that are being culled through open AIs that are being used by people in the enterprise that you don't even know are using them. So there's a whole new set of challenges for the CIO that we don't even have words for yet.

Tim Crawford: If you go back and look at, for example, social media, social media was really great because it gave everybody a voice. It lowered the playing field. Everybody had a megaphone, had a platform in which they could share their perspective. The downside is it gave everybody a voice. And so we're starting to see the same thing play out with AI, and especially when you look at not just vibe coding where someone can vibe code an application that is not a... They're not a developer. They're a line of business user. But they're able to actually create code that gets deployed within the enterprise or used within the enterprise all the way through to tooling that, when you look at things like OpenClaw and NemoClaw and Claude Cowork and now Amazon Quick and even SAP's Joule Work, that they just announced this week.

From single voices to autonomous agents

All of those products are starting to enable every user access to AI with varying degrees of guardrails and protections around it. And so for the CIO, this opens up a vastly wider threat paradigm that you have to get your arms around, not just in terms of what's appropriate and what's not, but also in terms of the risk profile and how do you start to ensure that you're still maintaining the right protections for your organization?

Anthony Scriffignano: One of the tenets of, let's say 10 years ago, the little thing called regression testing. You just did something new, and you've proven that the new thing that you built works, and you've proven that it integrates into the environment. You've done all the things you need to do. One of the things you have to do is also prove that things that used to work still work. That's a really difficult thing to do when all of these foundation models are changing, when the training data is changing, when the way in which the environment that you're relying upon is changing at the same time that you're making changes to how you use it.

Tim Crawford: There's another dynamic in addition to what Anthony mentioned, which is just thinking about identity and credentials, so authorization as well as identity. When you start thinking about agent identity and who's responsible for that and which agents have access to which credentials.

Reframing AI risk and value

Michael Krigsman: We're talking about this shift from essentially automation managing systems to autonomy. Let's talk about risk. How should CIOs reframe AI risk and also think about the value of AI? How do you balance that? Tim, you want to jump in?

Tim Crawford: We're still figuring this out. You know, this isn't something that we can say, "Okay, it's just like we did with cloud," or, "It's just like we did with the internet," or, "Just like we did with distributed computing." There's no similar framework in our past that we can use to apply to AI. It's a whole new space. And so one of the challenges for CIOs is kind of working through this process, both internally in terms of governance models and how we think about governance, but also with their partners, both their software partners and hardware and services partners, as well as their consulting partners. So it's a really complicated question that has to be answered.

Now, this is one of the reasons why quite often you see some of these AI projects being much smaller in scope, with the exception of some very sizable outliers. But it's smaller in scope because that's the way that they protect themselves against exposing the blast radius and potentially opening themselves up to undue risk. There are some sort of learned tenets that we have as leaders in this space, and they haven't gone away. So governance is still there. The need to have a consistent experience unless you intentionally are changing that experience. Provenance, knowing where you got your answer from. Permissible use, knowing that you're allowed to use it for the purpose that you're using it. The difference between a risk, which is something that might happen, and an issue, which is something that is happening. These are still there.

So there might be a tendency to say, "Oh my gosh," and I'm not accusing Tim of saying this, "Oh my gosh, LLMs are in the room now and everything is blown up, and we need a whole new set of capabilities." I think it's those things and some new things. We don't have to throw away the experience that caused us to all lose hair and get gray hair. We need to build on that a new set of capabilities around being able to understand agency, being able to understand authentication and validation in the context of a non-human on the other end of that. The ability to understand the impact of shadow AI and what people might be doing, and we shouldn't necessarily try to stop them because it's like trying to stop oxygen. So there's some new capabilities that we need to name. But those old ones are still there, and they will serve us well if we pay attention to them.

The shadow AI problem

Michael Krigsman: What about this issue of shadow AI and the fact that AI is proliferating, the use of AI, AI coding, every aspect, the proliferation is very rapid and essentially uncontrollable.

Tim Crawford: You're right. The way that AI is being used has expanded greatly. You know, everything from ChatGPT all the way through to things like Claude Cowork. And so it's not just vibe coding code, which has its own set of challenges, but it's also how users are using this technology, and the ease of use to use this technology such that now we can give agents... And this kind of gets to Anthony's point about agency. We can give agents our credentials and have these agents do work on our behalf. The challenge with that is that can happen outside of the enterprise walls, meaning the enterprise may not have visibility to it. And so that's part of the concern, is that what do these agents have access to? How are they working? And there's a whole downstream piece of this, which is agents have access to more than what the user thinks they have access to because of the way that we structure our back systems and protections within the enterprise.

Anthony Scriffignano: As I'm listening to you talk, I'm thinking about earlier in my career when there was a big push to outsource. Outsource development, outsource everything. If it's not nailed down, outsource it. We had some of these same issues of, how do you know that they're doing what we asked them to do? How do you know that we're not giving them access to more than they need or more than we realized? Some of these are not new problems. Obviously, they're happening at a much larger scale now.

There are some best practices that are emerging for shadow AI. One of them, I don't necessarily love them, but I get them, and I'll probably come to love them. You know, create a sandbox, put the tools there, put trusted tools there, put trusted data there. Teach people how to use the AI that they're trying to vibe code their way through life with so that they get better at prompt engineering, they get better at testing. Teach them some of the 5 biggest things that you can do to make a mistake. So instead of trying to stop this, because I don't think it's stoppable, nor do I think we should, maybe we embrace it and find better ways to, control is the wrong word, but to walk alongside it and understand the change that's coming along rather than constantly reacting to it. It's a Whac-A-Mole game right now, and that is not a sustainable strategy.

When business users become coders

Michael Krigsman: Now let's just think about Joe or Mary in the finance department, or Joe or Mary in marketing, who are coding their own tools, and the lawn is growing, and the mushrooms are coming up in the lawn, and they're using corporate data, which is fantastic. As a company, we have so much data, we should share it with everybody.

Anthony Scriffignano: Let me give you an example. A person whom I greatly respect, who has been a guest on your show, Steve Daffron, once said, "The most fundamental thing in data science is counting things." And I just love the simplicity of that statement. So if we start with counting things, if you just let people have at it and give them the customer, the CRM list, for example, and they want to count how many customers do we have, and they just count the rows, they're going to grossly over count because there's more than one row per customer. So you start by teaching them about relationships.

Now, at the end of the day, there's going to be a reality here that they don't want to be taught. They don't have time. That's why they're vibe coding in the first place. They don't want to hear you. It's a lot of wah, wah, wah, wah, wah. And so you have to put some controls in place. And certainly one of the biggest controls I would have in place is how much data are they pulling in? How much data is going out? Where is that data going? The good news is we can count tokens. We can look at things to see who the sort of mega users are. We can put guiderails in place, like no PII. You know, I'm not just saying take whatever data you want from the enterprise and do whatever you want with it.

Michael Krigsman: Now I understand. So you're going to issue a mandate, and everybody's going to do what you want.

Tim Crawford: Nope. I'm going to have

Anthony Scriffignano: controls in that environment, including AI agents, by the way, that watch what's going on and understand, and I'll put understands in quotes 'cause that's anthropomorphizing, but contextualize the biggest risks that we've identified. So let's just take one, PII. We want to make sure personally identifying information is not somehow leaking out of the enterprise. I can easily put a control in place to watch for that. I can put a control in place to watch for data that has value leaving the organization without money that has value coming back into the organization. There are controls that we can put in place, but we have to give people a way... We have to have a way to watch what they're doing and not just hope that they're following the guidelines and the edicts, 'cause we know they won't.

Why AI agents are not like humans

Michael Krigsman: Let's take some questions, 'cause we have a bunch of questions that have come in. And the first one, I'm going to go over to Twitter, X, and this is from Arsalan Khan, who says, "If we treat AI agents as, quote, 'almost human,' why can't we use existing governance models, apply HR models, for example, to AI agents?" Tim, thoughts on this one?

Tim Crawford: They are not like humans. And I think this is one of the challenges. I mean, there's been this conversation for some period of time, you know, and Moderna's the poster child of this. You know, should we be taking HR policies and applying it to agents because they're just digital agents versus human agents? The answer is no, because it's a whole different ballgame. So, for example, you have everything from just scale and speed as well as complexity that come into play. When you get into autonomy and you start talking about complex agents and complex workflows, now you have this concept of super agents that can go through an orchestration layer and call sub-agents, and you have to have governance layers at each of these steps, as well as understanding, and it's not always for nefarious purposes.

That's the other piece that I think we often forget, is that it's not just because someone said, "You know what? I'm here to screw the corporation, and I'm going to do something nefarious or just completely go off the reservation. I've been wronged, and this is the way I'm going to right it." But it also could be because agents and AI is imperfect. I mean, let's not forget that we are still learning how AI works. So kind of to one of Anthony's points, I mean, we have to be able to have explainability, especially for audit routines, in terms of how AI works. And so we don't necessarily have that in these algorithms today. It's more than just putting policies in place and the guardrails, but we also have to think about how we're setting up these workflows, what data it has access to, but what it could and couldn't do as well. And that's a far broader reach than any one individual.

Anthony Scriffignano: Let me add to that. Couple things in my practice that are very interesting in this space. One is to have adversarial agents that are trying to break things, and they can do that at scale, and that can be very interesting. So the red team concept. Yep. Another thing is ethical checks. So we can look at, but this is very helpful when you're generating text, when you're generating words. You can look at mean shift and sentiment. You can look at multimodality. Are the leaders leading the group, or is the group leading the leaders? Those are very valuable things to look at. Provenance, making sure that there's a way to go back to a fact for any kind of new nouns that you start talking about. Veracity, looking at misinformation and disinformation. When people repeat things that seem true, they're not necessarily lying if those things are lies. Those lies seem believable. So getting at the source of those lies. There are lots of very interesting things that we can do in this space that don't really have names yet, but they have agents, and we can get those agents to do some very clever things if we teach them.

Testing and release notes in an LLM world

Michael Krigsman: This is from Chris Petersen. Totally different topic here now, and he... And Chris says, "Historically, we could read the release notes and hope to know important changes in software, in a new software version. Is there any useful analog in the LLM GPT world, and if not, how do we," and he says, this is to Anthony's point, "how do we regression test for safety?" It's a really important point.

Tim Crawford: The large LLM providers, and it's not just the Anthropics and OpenAIs, but when you look at what IBM's doing, when you look at what Amazon's doing, when you... what Google's doing in terms of building their own models. So whether it's Granite from IBM, Nova from AWS, Gemini from Google, for example, they are putting safety measures in the mix. Is that enough? No, absolutely not, because we've already seen examples of where these models have kind of gone off the rails and done some pretty bad things. And so we have to be real cautious about this. And the same goes with the agents, too, because we have to understand how these agents work, and that's something that is a new skill that we are just learning about today.

Anthony Scriffignano: The old approach to regression testing, you build a bunch of scripts. You say, you know, all of these things should still work when we're done. Here's the data that you use with these scripts. When you're done doing whatever it is you were doing, run these scripts with this data and make sure it produces these answers. You can still do that. And to Tim's point, the major models at least do tell you when they're changing. They don't tell you what they're changing, but they tell you that they're changing. So that's a good time to start doing this.

There are some other dimensions in an LLM, in an agentic world, I'll say in a gen AI agentic world, there are some other dimensions that you need to think about. One of them is misuse. Are people able to, either through prompt injection or stupidity or kicking the tires, able to get your environment to do something it was never intended to do, and have you just made it easier for them to do that? Another one is misadventure. You can't say hallucination anymore because everybody just gets triggered, but it is absolutely possible that you may have enabled your system to speak for your organization in ways that you never intended. There are other dimensions like that, that we can test for. We have to name them. We have to build modalities for testing. Note that I didn't say scripts, because this will not be linear. This is very much a non-deterministic kind of thing we're trying to do. But if you say that the first thing we're trying to do is test against misuse, all right, that's a good start. At least now we know what mindset we're in when we're building out regression approaches to misuse. So it's absolutely not a linear process anymore. You could still use that linear process, but you need to build onto it a sort of a non-deterministic approach to some of these other ways of messing up with gen AI.

What aspiring CIOs need to learn

Michael Krigsman: Tim, this is a CIO career question, and this is from Preethi Narayan, who says: For aspiring CIOs, what capabilities and mindset shifts will matter most in an AI agentic enterprise world, and how difficult will it be for today's CIOs to transition from traditional IT operating models to AI-native ways of working?

Tim Crawford: I often talk about this differentiation between the traditional and the transformational CIO. A CIO is not a CIO is not a CIO. If you're an aspiring CIO, you need to focus on the behaviors of the transformational or business-oriented CIO. In an agentic world, you have to understand your business, and I don't mean just the superficial layperson's explanation of, "We're a clothing retailer, and we sell shirts and pants and socks," and that sort of thing. But you need to understand the underpinnings of how the customer engages with your business. You have to understand the entire workflow of how your business operates end to end. You have to understand some of these dynamics and nuances within how your company operates, not just the cultural and institutional knowledge, but also the industry and where it's going.

And so part of this answer is that you need to be a business leader first that happens to have responsibility for technology. When we move into this agentic era, it has to understand the business. I mean, if you don't understand your business, it's going to be really hard to truly lean in on these agentic workflows more than just the simple, "Hey, we've taken a process, and we've automated that process." So for the aspiring CIO, first and foremost, understand your business deeply. Second is reimagine your business. Think about how you would do it differently and where technology can play a role, and I think this is where some of the agentic workflows, especially the more complex ones, become really interesting and, surprise, surprise, more valuable to your organization.

Designing solutions with red, blue, and green teams

Michael Krigsman: Anthony, here's a question from Hue Hoang, who says: How important do we see the need of approaching a demand to have a very clear and comprehensive blueprint in our solution designs in order to prepare teams to be operational and red team ready?

Anthony Scriffignano: Talk about hand grades, hand grenades. That's a fascinating question, and you could probably write a book on the answer. But the essence of it is how do you get people who are in this mindset to be ready for this red team concept while everything's changing at the same time? And that is a very, very big question. So the first thing I would say is, going back to the comments that Tim just made about transformational leadership, one of the essences of transformational leadership is not to just be changing things for the sake of changing, to be going from someplace to someplace with overt intentionality and understanding why you're doing it.

So if you're bringing red teams into the room, you don't want to just bring red teams in the room and say, "All right. See if you can break it." That's a ridiculous mandate. What you want to say is, "We're worried about 5 things. We're worried about data exfiltration. We're worried about malware injection. We're worried about changes in the..." You know, whatever those 5 things are. And there should be 5. And then the red team is focused on those 5 things. The non-red team, the good guys, know what they're sort of defending against, and it's a bit of a fair fight because they're not just coming in and trying to break stuff while you're trying to build stuff and fix stuff.

There's got to be some discipline to this. It can't just be a food fight. What happens when you let it just be a food fight, it's fun. People enjoy it, and especially the red team enjoys it 'cause they don't have any rules. But it's just so disruptive, and I think we've all worked in large corporate environments where you're just trying to get your job done, and emails are coming in that are fake, and they're trying to trick you, and there's pen tests going on, and yeah... And at the... You get to a point where you don't trust any email, and then you don't respond to a customer because it looked like a pen test. We got to be a lot more thoughtful about this, and I think it starts with Hui's question about how do we get them ready to do this. We've got to teach them what they're doing and not just charge them with breaking stuff.

Tim Crawford: I want to build on this a little bit because there's an additive piece, especially when you start talking about AI, that I think some people are kind of missing along this pathway, which is the automation that's coming soon is not going to be optional. It will be a requirement whether you want it to be or not. And what I mean by that is that the sheer speed in which we need to operate is increasing so quickly that we just can't put a human in the loop on some things.

And so when you think about red teaming, and if you're not familiar with the concepts of blue teaming and green teaming, definitely get your arms around that. But for those that aren't familiar, you know, a red team is the attackers. We're simulating those attacks and trying to understand kind of what's happening. The blue team are the folks that are looking to put up the defenses around those attacks. And then the green team says, "Okay, we found these attacks, these vulnerabilities. How do we fix them, and how do we start to ascertain, well, we have 2 different ways we could go about this. What's the right way to do it?" That's no longer a group of humans doing it, nor is it a group of tools doing it. It has to be a group of agents that are automatically working on your behalf within your constructs and are doing this almost continuously. That's the speed in which we have to get to.

And so here's the thing that I would pose to your audience, Michael, is that if you had to automate any said process, whether it's in cybersecurity, whether it's in a business process, if you had to automate that fully, no caveats, no ifs, ands, or buts, you had to automate that, how would that change your thinking around all of the other aspects, everything from how you develop the application, how you use the application, how you update the application? Like for example, patching. We were having this conversation with a group of CIOs just this morning. What happens if you have to take patches automatically? You no longer can do testing before you deploy it, because that might be a reality for us in the very near term. This is something that I think people have to start getting their head around, is if these automation routines become the norm, meaning we have no choice but to have to do it, how does that change our thinking around other aspects and other decisions and choices we've made within the enterprise? And that's what I would challenge the audience to think about.

Michael Krigsman: What could possibly go wrong with automation that you, over which you have no choice?

Tim Crawford: We've all been there. We've all run organizations. I get it. But we may not have a choice. We may have to try and figure out how to... So for example, thought experiment here, just pick one piece off. Pick off the patching, for example. If you had to automate all of your patches because everything is a 0 day, how does that change your thinking within the organization and how you operate?

Anthony Scriffignano: I would suggest that there's some of that going on without your knowledge anyway, if you're using anything in the cloud or any API or any external black box. Stuff is getting patched, stuff is changing. It's part of your stack. You're relying upon it, yet you may not even know it changed. And there's that other dark side of, I don't want to name organizations, but there's some very big ones in the news recently where they didn't do a patch for whatever reason, and it created some huge vulnerability, and now you're explaining why you didn't do the patch. So, you know, the answer isn't to patch or not to patch. Your point about the red team and the, at least the red team and the blue team being automated clouds of agents, I think it's both. So there's also orchestration from, you know, Lex Luthor in those cohorts that are people that know how to think new thoughts. Otherwise, we only learn how to protect ourselves against preexisting vulnerabilities that we knew about. And some of these new things that come up, I am amazed at the incredible creativity of bored 14-year-olds. Some of the stuff they come up with is just genius in its simplicity. One of the most recent major oopses that hit a big part of the world happened by releasing an empty file and it caused all kinds of problems 'cause nobody ever thought of that. So yeah, we need people and machines doing this.

The social media analogy revisited

Michael Krigsman: Let's go to another question, and this is from Swami Vaidyanathan, and he says, "Earlier, we spoke, one of you mentioned social media." And he says, "It was a great analogy with social media where the floodgates opened, sort of. But if we don't have a close parallel from history that can be used as a model, do you have a specific path or thought process you can share for CIOs and teams to approach this?"

Tim Crawford: It was my comment that social media gave everyone a voice, and that's the upside. And then the downside is social media gave everyone a voice. We have something similar with AI. I think the construct here is, unlike social media, number one, there's more at risk, and the speed in which it's at risk. Because with social media, it took time to build an audience and have people follow you, and then there's a human in the loop. Now we're talking about agents that can run off and potentially do tasks and do a lot of damage before anyone is even to the wiser. It's actually one of the challenges of agents right now is some of these more sophisticated agents are moving at such a speed, they're breaking APIs, they're breaking websites because they were never designed for that kind of scale and that kind of speed.

There are constructs you can use fundamentally to understand kind of how agents work, how agentic workflows work, and then step into it. You know, I often say that one of the things you have to think about when you're thinking about your organization and how you accelerate your organization in an agentic era is how do you start with one thing and then move to a second, but with each iteration of this flywheel, you're accelerating the organization. The same thing is true with agentic workflows, is that you need to start small, get your sea legs, but then build upon that and accelerate your knowledge and accelerate your understanding of it.

And the reality is there is a training aspect of this. We need to educate people on the importance of both the opportunity and the risks so that they can make good business decisions for their organization as well as their role. And that's something that we haven't necessarily stepped into as much today. It's not as much of putting the policies in place and bringing the hammer down, but rather, you know, we've got some intelligent people within the organization. Let's lean into that. Let's help them make good business decisions on our behalf.

Anthony Scriffignano: When we think about making mistakes faster with agents, we also have to keep in mind that the reason we call it gen AI is it's generating its own content, and a lot of the content that AI is consuming was produced by other AI. So there's a multiplicative effect of this problem. It's not just making mistakes faster, it's making more chaotic mistakes faster. There's a geometric effect here that we have to keep in mind. We can get out of control at a rate that we cannot recover from at the same rate. And so it's really important to have these conversations about guardrails and red teams and all of that to sort of thwart that kind of multiplicative effect that those are the things that kill us.

Human in the loop: meaningful or theater

Michael Krigsman: Let's shift gears and talk about one of the great buzzwords of our time, which is human in the loop. So when is human in the loop meaningful, and when is it theater?

Anthony Scriffignano: If you look at the regulatory response to this right now, if you look at the application and what the application is doing, the more critical the use, the more mandatory it is that you have a human in the loop. That is the general theme of all of the governance that's out there. There's a little bit of blinders in this because I think Tim pointed it out earlier, there are things that happen way too fast to have humans in the loop, and that's happening more and more now. And so requiring a human in the loop might make you feel good, but humans can't make decisions fast enough when the environment is changing at certain rates.

And so the response to that is sort of audited humans in the loop, the humans will look at some percentage of the decisions being made. That's not a sustainable answer either. The reality is that even humans in the loop, all humans don't have the same reaction when they look at things. So the gold standard, this is called a heuristic frame, the gold standard is to have similarly instructed, similarly incented people looking at things. Well, you can't get that in the real world because people don't understand things at the same level. They don't have the same background. They don't have the same stake in the game when they're making a decision. Some are trying to get done, some are trying to get done right.

So the important thing in this is not to say, "Well, gee, Michael, it's really complicated, and I wish I could explain it to you, but there is no answer," which is the answer, by the way. The answer is that wherever you are in this human in the loop, be it because of regulatory constraint or because of ethical AI or because of good design principles, be where you are on purpose and constantly reevaluate that in terms of the rate of change in the environment and the elasticity that you have for risk.

What AI governance looks like in practice

Michael Krigsman: Tim, given that, what does AI governance actually look like in practice?

Tim Crawford: It's complicated, and if you think it's simple, full stop. You probably don't understand all of the aspects that are needed in complex agentic workflows because you have a level of governance, for example, and I'll just use a very simple example where Michael has access to certain things, Anthony has access to certain things from an agent, Tim has access to certain things, and that's what it presents. And we often talk about RBAC as the method to say, based on what Tim has access to, that's what we're going to present. Well, then the agent governance layer sits above that in terms of what the agent itself has access to be able to do. Above that is a knowledge governance layer, which is how do we start to amass this knowledge and where does it come in and what agents have access to what from a knowledge perspective. So that's just 3 layers by itself.

Oh, and by the way, where does this data that we're going to tap into reside? Because when we start talking about business processes, we're talking about processes that span multiple departments and therefore multiple systems. So we might have some data in ServiceNow, we might have some data in Salesforce, we might have some data in SAP. And so each of those systems have their own governance layers. So how do you start to amass and rationalize these different governance models to be able to present the appropriate data to both the agent but then also to the end user? And this is where things get really complicated because it's not just a really complicated math problem, but there are a lot more aspects to it 'cause it's not just simply bits on the page of data elements. We have to even look beyond metadata and semantic data and start looking at context around data, which is something of a new concept for us. So how do you start to put all of that together?

If that already makes your head explode, you're not alone. And so this is why we're starting to work through how do we think about it, but in the meantime, we have to kind of dumb down the architecture a bit because we can't solve that problem today. If you live within a given ecosystem, you have additional opportunities that you don't have when you start working across different systems. But that's also why you have to kind of wrap your head around, okay, so opportunity risk here, I'd like to be able to do that, but I have to understand how do I start to play these other pieces into the equation.

Anthony Scriffignano: One of the first things that I do in my practice is, everybody wants to rush to the tools. You know, what AI are we going to use? How are we going to generate the code? And I usually try to slow it down and start with, what's the problem we're trying to solve? How would we know we solved it? What are the things that can happen that are sort of the unknown, unintended uses of what we're about to do? So I have some fundamental questions that I ask, and if we can't answer those questions, there's a bit of malpractice in taking that next step. So first principle's really important. There are certain approaches in... I'll just use LLMs as an example. You can use a RAG approach if you're trying to do something and you know that these documents represent the truth that you understand right now. Start here instead of starting with the internet. You can look at veracity adjudication. You know, how would we know that something is true? How would we know that it doesn't contain part of the truth to mislead us? So there are definitely mindsets, and there are definitely epistemological models, you know, ways of knowing that you can ground yourself in that help defend against the chaos that Tim is talking about. It's not hopeless, but you do need adults in the room when you're doing this. You can't just throw tools at it.

New roles and organizational structures

Michael Krigsman: So are there new roles and new organizational structures that CIOs need to be thinking about now or starting to build right now? Okay, so what

Anthony Scriffignano: are they? The one we keep talking about is context engineers. People that understand the way in which the AI is given context of the problem that it's trying to solve. That's definitely an emerging role. You know, we're talking a lot about red team. That's a role that preexisted, but it's way more important now. Tim, I'm sure you have a few more.

Tim Crawford: Even beyond red teaming, blue teaming, green teaming, from an automation standpoint, and then look at the finance piece of this. We haven't even touched on cost. There's a whole talk track around cost and understanding what the true value, right? Let's be focused on this. Value is an equation that is value equals opportunity minus cost. So it's not just about the cost. It's about what you're able to do for that cost or what you're trying to accomplish. And so there are new frameworks that are starting to come out. FinOps is working on versions of FOCUS and their framework to be able to accommodate understanding the true AI cost. There are some startups that are just about to announce some of the things that they're working on around this. The TBM Council, from their perspective, kind of that umbrella of all IT, understanding value and being able to work down. But this isn't as simple as just tagging resources and saying, "Okay, well, this resource is this application, and this resource is this application." We're talking about agents that, let's say the 3 of us all have access to this agent. How do we build those resources to the appropriate person? If this is an autonomous agent. And if it's a complex agent where it's an agent that is learning, and it's also discovering other agents, which, oh, by the way, might not be within your organization, might be a third party, and it's able to build ephemeral agents, there's a whole nother layer of complexity that comes into the mix. We've largely been talking about a very simplistic way around agentic AI, but these other pieces are actually in market today they're just in different forms, and it's only a matter of time that we're going to see them all weave together.

The future of IT talent and jobs

Michael Krigsman: Anthony, we are just about out of time. Very quickly, you're going to have the last word. What happens to talent and jobs inside IT as a result of all of this? Very quickly, please.

Anthony Scriffignano: I don't think AI's going to take all the jobs away. I think that we're going to have lots of new, exciting jobs, and I think that it's very important that we don't try to just vibe code our way through, you know, evolving our career, that we learn these first principles, we learn how things are changing, we learn how to take advantage of it. We start working on new problems that we couldn't solve before because we have this abundance of capability. Don't just look for the easy button. Look for the next thing that you can do to add much more value to the organization in a responsible way.

Michael Krigsman: Well, that's pretty good advice, and unfortunately, we're out of time. A huge thank you to Tim Crawford and Anthony Scriffignano. Thank you both for being here. I'm very grateful to you.

Tim Crawford: Thank you for having us.

Anthony Scriffignano: Thank you.

Michael Krigsman: And a huge thank you to everybody who watched, and especially you folks who asked such excellent questions. Before you go, subscribe to the CXOTalk newsletter. Go to cxotalk.com. 2 weeks from today, we're talking with the chief technology officer of Mozilla. So get our newsletter so we can notify you of upcoming shows, and we'll see you again very soon, everybody. Thank you so much. Take care.