From ad hoc to autonomous: The AI content ops maturity model

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Spotify | Amazon Music | Email | TuneIn | RSS

There are five levels of maturity for AI-driven content operations. Which level are you in? In this episode, Sarah O’Keefe and Bill Swallow walk through the AI content ops maturity model, from ad hoc experimentation to fully autonomous workflows.

Sarah O’Keefe: We want this automation, right? We want the ability to go in and extract release notes and do something with them. We have to have a certain level of maturity on the software development process so that we can grab the appropriate information. The same thing is true on the content side. You have to have a certain level of maturity in your content development processes, in your content management, so that you can identify the right things to process and the right things to access.

Related links:

• Want to know more about Sarah’s nifty little side project? Register for our upcoming webinar.
• AI in the content lifecycle
• Enterprise content strategy maturity model

LinkedIn:

• Sarah O’Keefe
• Bill Swallow

Transcript:

Introduction with ambient background music

Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations.

Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it.

Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change.

Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off.

End of introduction

Bill Swallow: I am Bill Swallow.

Sarah O’Keefe: And I’m Sarah O’Keefe.

BS: And today we’re going to talk about AI in content operations, or more specifically, a maturity model for AI.

SO: Everything needs a maturity model, even AI.

BS: Even me.

SO: I have no comment.

BS: My maturity model is written in crayon, what can I say? So okay, so we need a maturity model for AI as far as content operations are concerned, and probably in you know, many different degrees, but we’ll focus on content operations. So what might that look like?

SO: I’ve been thinking about this and what it looks like to employ AI as a tool to help you with content. And as I was thinking about what this looks like, you know, you always fall back on that standard five-step model where one is basically mass chaos, and five is the perfect world, generally. also, one is nearly always cheap, and five is nearly always expensive enterprise things. But, you know, let’s go a little beyond mass chaos versus governed, regulated, etc., and sort of sort of back up a little bit and talk about what this might look like. So level one in every maturity model typically is ad hoc. And what that means is that in this case, AI is being used sporadically by some people. It’s inconsistent. And I would say that when we look at AI and content specifically,

BS: Mm-hmm.

SO: This is going to be things like reprocessing your content using public-facing models. So I wrote a draft of something, I shove it into ChatGPT and I ask it to shorten it or tighten it up or identify areas that are problematic. Or I just say, hey, you know, write my article for me. The outcome that you’re gonna get on an ad hoc model is going to depend on an ad hoc level one.

BS: Mm-hmm.

SO: AI thing is going to depend on how good you are on the individual’s expertise and their level of interest. So if you want to just go in there and say, hey, I have a bio and it’s too long, and I’ve been asked to produce one that’s only 50 words for a particular conference, for example, then, you know, this is this is actually a really good example of ad hoc, right? 

BS: Mm-hmm.

SO: We have these long multi-paragraph bios and every conference I’ve been to has a different requirement for how that bio needs to be shaped. And the fastest way to success is to just shove it into a chatbot and say, give me a 50-word version. And and then read it and make sure it didn’t invent things or give you a PhD or anything like that, and then ship it off to the conference organizer. But this is much, much faster than rewriting it from scratch by hand. And also I think, it’s a good example of something where I have the extended version and I’m going to summarize down. And that usually works pretty well. So level one is ad hoc. It’s kind of sporadic. There’s no standard across the organization. It’s just me saying, this looks useful, or you’ve probably got some use cases in this space as well.

BS: Right. So it it kind of aligns with I guess level one of the the content maturity model that we talked about a while back, where level one is is simply content exists. Could be, you know, someone typing stuff up in Word or, you know, using a myriad of different tools, no style guide, just kind of getting content out there because people need it.

SO: Yep. So level two is tactical. And tactical is sort of like we’re using this tool to solve some specific problems. And what you’re going to see here is something like that Bill has invented some nifty time saving tool and he has shared it with other people. Or in a larger organization, maybe somebody invented a nifty validate or or something like that and they’ve rolled it out across maybe the department, probably not the entire organization. Aomething like AI support is being rolled out. Maybe the organization has created a chatbot internally for customers, right? So there’s a chatbot, it’s sitting on the company website, people can use it to get answers, but it’s really bad. And the reason it’s really bad is because nobody thought too carefully about the content going into the chatbot, because again, we’re tactical. So probably this looked like the AI team just raided the local SharePoint, grabbed a bunch of content, did not pay a whole lot of attention to the question of whether this content was up to date in release status. Those things don’t exist, right? It’s just, look, a bucket of PDFs. Cool. Let’s dump them into the AI and go for it.

BS: Mm-hmm.

SO: And tragically, in many cases, the techcomm team is sitting on rigorous, structured, vetted, approved content, and nobody remembered to go ask them, can we have your content? Or where is your official content source? Or how do I know what version belongs with which document?

BS: Right, because you know, in in their point of view there’s a PDF of it, so I don’t need to ask them.

SO: Yeah, it’s it’s just PDF. How hard could it be? So something like AI was support was rolled out, but nobody really thought about it. Maybe it’s at a departmental level, probably it’s not enterprise-wide. And nobody has really thought about connecting this AI thing to the assets inside the organization in a reasonable, rigorous, governed, organized kind of manner.

BS: And I suppose that’s where you get to the next tier.

SO: Right. So the next tier after tactical comes strategic, right? So we have an actual strategy. Now, one of the difficulties in talking about AI is that AI is a tool and it’s kind of like talking about electricity. You can apply it to lots of places and it’s more sensible in some places than others. But when we say what’s your AI strategy, like how do you use water? I mean, come on, and the answer is of course to drive the AI and you know destroy the environment. But there are things that you can do with AI that are useful for content. There are also things that you can do that are not. So if you have a strategic approach to this, a strategic approach to use of AI, backing up to the authors again rather than the delivery side, maybe this looks like a collection of prompts that have been built that are shared.

BS: Not with electricity.

SO: Maybe this looks like saying this is the workflow that you employ. These are the kinds of things that we do to actually test whether this thing is working. these are the metrics that we’re following. So there’s an actual overarching bigger picture that somebody’s thinking about that goes beyond, let me go shove this into chatbot of the day.

BS: Mm-hmm. Right, right.

SO: So there’s an actual strategy for the public-facing chatbots. Somebody has thought about the back end. The authors have useful AI tools that add to their you know their productivity. One of the things that I’m hearing a lot now, you know, low-hanging fruit, release notes. Nobody wants to write release notes. It’s a terrible drudge task. It’s and it needs to be done. Well,

BS: Mm-hmm.

SO: There’s now there are now a lot of solutions that look like look at the diff in the code, look at the delta from you know version one to version one dot one, find the diff in the code, find the changes that have been made, look at the JIRA tickets that have been addressed, that have been solved in release one dot one, and then consolidate that all into a set of release notes that say, here’s what’s been done. And that’s probably 90% of the work, and the last 10% of the work is read that and make sure it’s accurate. Right? Don’t please don’t skip that step. Like actually look at what the thing is generating. Now, what’s interesting to me about level three, this sort of more strategic approach, is that what you’re gonna start to see is that you have prerequisites for this. You can’t do this. 

BS: Yes.

SO: So release notes are good example. Let’s say that hypothetically, and this is gonna sound insane, but let’s say that hypothetically, you have software development and you have no source control.

BS: Hmm.

SO: Everybody’s screaming, right? Because this is nuts, and why would you ever do this? Okay. But hypothetically, you have no source control. Okay. How do you know what’s changed between version one and version one point one?

BS: It’s up here in my head.

SO: Excellent, great. Ha okay, cool. so I’m gonna need to connect the AI to your head so that we can pull those changes out of your head.

BS: That sounds fun.

SO: Yeah. Amazing. Right. So all of a sudden, because we want this automation, right? We want the ability to go in and extract release notes and do something with them. We have to have a certain level of maturity on the software development process so that we can grab the appropriate information. Now, the same thing is of course true on the content side. You have to have a certain level of maturity in your content development processes, in your content management, so that you can identify, you know, the right things to process and the right things to access. And why it is that, you know, we know that software has to be governed, but we’re not so sure about content is a mystery to me.

BS: I had never understood that.

SO: Yeah. So there we are. Okay, so that’s kind of like a level three. With there’s some sort of strategy emerging across the enterprise. There are some useful tools and they’re shared. This is kind of like in content when you start thinking about templates. We’re gonna have some templates and we’re gonna give them to people and they’re gonna use them and it’s gonna be great. All right, so level four is governed, managed. And so now good understanding of AI.

BS: Mm-hmm.

SO: It is being applied in a useful, intelligent manner, by which I mean don’t apply it to the wrong problem sets, right? Apply it to the things where it makes sense to apply it. Thinking about governance, thinking about metrics, thinking about success. And then your data sources and your content sources are being managed in such a way that the AI gets good input and can actually generate good output. So I’m actually not a big fan of the term human in the loop because human in the loop implies that the AI is doing all the work and then like the human eventually gets around to QAing it. You know what? We’re terrible at QA. You know who’s good at QA?

BS: Mm-hmm. AI.

SO: No, computers, not AI. AI is all about probability and whatever. It is actually not very good at QA. What’s good at QA is traditional software, right? One plus one is always two. In an AI, one plus one, sometimes it’s not two. So you manage that stuff and you put those guardrails up and you start putting up the guardrails that say, okay, when the AI kind of wanders off into the wilderness, we’re gonna like bring it back to reality. We’re gonna have, we’re gonna put it in a box, right? Make the AI think inside the box, and we’re gonna govern what that box is. AI is great at thinking outside the box. Unfortunately, that’s usually not what we want from technical content. So it needs to be in in the box and it needs to be consistent and needs to be managed and all the rest of it. 

BS: Mm-hmm.

SO: So we govern it, right? We go in there and we make sure that the processes and the tooling that’s being put in place and the automation that’s being put in place where we’re leveraging or using AI to do things is managed. And so the human in the loop thing. I don’t want the human in the loop to fix things on the back end. I want the human in the loop to fix things on the front end so that what goes in is better, so that there’s less work to do when it comes out. You know, fix it beforehand. Don’t remediate it afterwards. That’s a boatload of work and it is not fun. So fix it ahead of time.

BS: Right. Yeah. And likewise you probably wanna have, you know, some guardrails in there so that, you know, your AI, whatever it is, doesn’t go playing around with content that has been approved and released and is not slated for updating.

SO: Yeah, you know, don’t fix that. That one’s done. That one’s and you know, we’re not even talking here about what it means to be in a regulated industry or in a regulatory environment. there if you are shipping or sorry, if you are a large organization and you are doing things in Europe, then you are likely subject to the European, the EU AI Act.

BS: That’s a completely different beast.

SO: And you have to think about what that means for what you’re doing, because the fun gold rush wild, wild west strategy of just throw AI at everything is not gonna fly in Europe. Okay, so that’s governed, you know, hypothetically. And then level five is agentic, which is basically that everything, everything or a lot of it is running autonomously.

BS: Mm-hmm.

SO: You know, the layman’s explanation of what is agentic AI, the difference is that instead of saying I need to put a prompt into the chatbot, it does it itself because you’ve built out the systems that drive all of that happening. 

BS: It understands what needs to happen at what point in time.

SO: Well, let’s not say understands, but yes. I’m trying so hard. 

BS: Well, yeah, not understands, but there’s a workflow in place that the AI is following.

SO: And it’s so difficult. I think that, you know, there’s, as a side note, the why do we think, why do we impose personality on the chatbots? And the answer is I think that psychologically it’s very, very difficult to interact with something that play acts at human interaction. What a great idea! Good for you. I love your thinking, blah, blah, blah. So it makes you think you’re interacting with a human. And I don’t think that our brains are equipped to say, no, actually, this is a machine.

BS: It’s like a scary version of Teddy Ruxpin.

SO: It, well, it passes the Turing test. And so we just can’t separate if it feels like you’re interacting with a person, you know you’re not, but it feels as though you are, and feeling is always gonna win over knowledge. So

BS: Mm-hmm. Well yeah, the interaction is a lot more organic than you get from, you know, traditional tools.

SO: Or, you know, yeah. I mean, think about the difference between a search, typing in a search string, and you know, a conversational search, a conversational interface. It’s quite, quite troubling, actually. Yeah, so this is kind of the five-level model, right? From big mess ad hoc, some things are happening, th some things aren’t, up to it’s completely autonomous. Now, if it’s going to be the more autonomy you want, the better your inputs have to be, which circles us right back to, and therefore, you have to do the work on the content side, because if you don’t do the work on the content side, the AI is going to go off the rails in interesting, unexpected, and potentially disastrous ways.

BS: It will play with the mess you leave it.

SO: Yep. So that’s where we’re going with this. That’s the AI content ops maturity model as it stands today. I reserve the right to change it tomorrow.

BS: Today. Of course.So this model came out of I guess some little nifty side project you’ve been working on recently.

SO: I am working on a nifty little side project. We’re not quite ready to announce it. but I’ve got a a co-author and we’re working on a thing.

BS: Fair.

SO: I could say more but then I’d, you know, be in trouble.

BS: When might you be able to say more?

SO: I believe that we have a webinar coming July 22nd, where we will say some more things.

BS: Alrighty. Well we will learn more things then.

SO: I too will learn more things and probably we’ll have to we’ll probably we’ll have to change everything we’ve done up until this point because everything will change by then.

BS: Of course. Guess that’s a good place to leave this podcast. Thank you, Sarah.

SO: Thank you.

Want to know more about Sarah’s nifty little side project?

Register for our upcoming webinar.