The limits of automation and AI in content workflows

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Spotify | Amazon Music | Email | TuneIn | RSS

What are the limits of automation and AI in content systems? In this episode, Sarah O’Keefe and Bill Swallow dive into traditional formatting, workflow challenges, and what happens when AI is introduced.

The limit of automation here is not really the automation. It’s the people who say, “I don’t like the way this looks, and I’m gonna find a way to fix it. I know all sorts of ways of bending this stuff to my will.” If you’re generating content at scale, you can’t afford to do any of this stuff.

Related links:

• Digital sovereignty in the age of AI
• From ad hoc to autonomous: The AI content ops maturity model
• Subscribe to our newsletter for updates on Sarah O’Keefe and Carlos Evia’s book: The Uninvited Author

LinkedIn:

• Sarah O’Keefe
• Bill Swallow

Transcript:

Disclaimer: This is a machine-generated transcript with edits.

Introduction with ambient background music

Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations.

Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it.

Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change.

Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off.

End of introduction

BS: Hi, I’m Bill Swallow.

SO: And I’m Sarah O’Keefe.

BS: And today we’re gonna talk a bit about automation, and more specifically what the limits are for automation.

SO: Yeah, they let us out of our cages. And so here we are. And I think that in this world of  I’m not going to get through the first sentence without saying AI. In this world of AI, we are suddenly facing, you know, increasing automation across every facet of everything that you could imagine. And I think it’s important to take a step back and talk a little bit about limits and what automation cannot do and maybe more importantly, why it cannot do certain things.

So before we get to, you know, the AI in the room,  Bill, what are the limits when we talk about generalized publishing in the pre-AI world where we’ve done a ton of formatting automation and we talk about all those kinds of things, what are some of the limits that we’ve run up against in terms of automation?

BS: Yeah, automation is something that we’ve been doing quite a bit of  for the past, what, twenty years, I think, or more. I’ve only been around for thirteen years. I’m thirteen years young,  so…

SO: Yeah,  this is lies.

BS: But no, a lot of our work around automation is around publishing.  That is, I think first and foremost, the most common case.  And with automation and publishing, you can save a lot of time and save a lot of effort, but a lot goes into building that capability because you can’t really work around edge cases or you know, fancy hand-cobbled formatting,  hand page breaks,  all of that thing. You have to kind of rely on the software to understand the content that you’re feeding it and it’s going to produce something that you know is a finished product: a PDF, a web page, what have you.  And you know, if you deviate from the structures and conventions that it’s expecting, then all bets are off as to whether or not your automation will be successful.

SO: Right. And so, from our point of view, doing lots and lots and lots of structured content work,  i setting aside the tools and the technologies, what structured content really does is limit variability. Right? It limits the variability of the information or of the markup really going into your system, which then means in turn that the system can be automated and can automatically output all the outputs, all the deliverables, all the file formats that you need. And so  when you have weird edge cases, and I am of course the worst offender in terms of actually finding ways to bend the software to my will,  but only when I’m an author, right? As a system configuration, I’m, yeah, I think you should all follow the rules. Yeah. 

So here we are, and we basically say we’re gonna limit the variability of the input and standardize the input and therefore we will get standardized output. Cool. But as you said, that only works if you don’t have so what is a weird edge case or what are some of the things you’ve run into that just bollocks up automation?

BS: One good case is working with content that maybe has valid structures in place. So they’re not, you know, the content itself validates against a validator. So there’s nothing wrong with it. But they’re using, let’s say, different in this case, DITA elements in a let’s say creative way. You know, so they’re you know  I was on vacation last week, so that wasn’t me. 

SO: I see you were looking over my shoulder last week again. Yeah, that’s true. Well,  what I actually ran into was I had a DITA map; it was valid, it validated, and then I used Oxygen’s validator, which you know really goes a little bit deeper and looks at things. Everything was fine, but it crashed. And eventually what I realized or what I found after some digging was that somebody who was definitely me had inserted a draft comment and the draft comment was in a table, maybe under the title, but before the table group kind of thing. It was in an unusual location and it just the processor just died. It just laid down and gave up.  now I don’t know exactly whether that was because of the, you know, the core processing or something that we did in the the  plugin that I was running. It doesn’t really matter. The point is it was it was valid. But it didn’t pass the processor because the processor was like,  Why would anybody put a draft comment in this location? This is dumb. And then I, you know, felt berated by the processor and I moved it and then it all worked.

BS: Interesting. Yeah. I would have enjoyed being there for that. 

SO: Yeah. Mm hmm.  So anyway, we fixed it, and it was fine, and you know, and off we go. Let’s talk about pagination!

BS: Pagination’s a fun one. 

SO: Pagination’s my favorite.  I get very upset with bad line breaks or bad page breaks.

BS: Yeah. And you know, we can build in rules that say, you know, only, you know, to control widows and orphans, that type of thing. You know, tell it, you know, break a table leaving X many cells. If you don’t have X many cells to break, then move the entire table to the next page, all that fun stuff. But if you have, you know, specific places where you need to break to a new page for whatever reason, that always can’t be necessarily automated, depending on what the rules are for producing your output. You know, the processor’s not going to know that, you know, you may arbitrarily want to break a page at this particular location. So you know, in many cases we cobble together a little tag that says, you know, essentially break the page here,  and the processor knows to you know, when it sees that to say, okay, stop processing this page, move to the next one.  And you know, it works most of the time. 

SO: I would never. Yeah, the problem with inserting page breaks, as I’ve learned to my great sorrow, is that, of course, later you add more content and now you have a page break a third of the way down the page because you hard-coded it in. So this is bad, and you shouldn’t do it. But if you insist on doing it, do it l as late in the process as possible. Don’t try to fix your pages when you’re 80% of the way there, because you will have to reinsert them over and over and over again.  also.

BS: Right.

SO: Inserting empty tags that have a non-breaking space in them to introduce vertical space is wrong, and you shouldn’t do it.

BS: Ha ha. I will agree with that one. Also, I mean, doing these types of things to force a page break or to force extra space, it really flies in the face of automation because technically you have to create the output in order to know where you need to insert your page breaks and then go back and add them and then automate your output again.

SO: Ha ha. It looks bad.

BS: Yeah.

SO: So yeah. So really the limit of automation here is not really the automation, right?  It’s, well, it’s me, right? It’s the people who are like, I don’t like the way this looks, and I’m gonna find a way to fix it. And I have lots more demented tricks up my sleeve. I know all sorts of ways of bending this stuff to my will. And if you’re generating content at scale, you basically can’t afford to do any of this stuff, right? It’s one thing if you’re producing, you know, one document and it’s short and/or it’s marketing content, and we’re really concerned about the appearance because of people making buying decisions. But if you’re producing, you know, ten, twenty, fifty thousand pages a year, then you just need an engine that produces this stuff. So

BS: Mm-hmm. Yeah. And the page break problem is actually a good one to speak to at scale because if you’re in an environment where you’re sharing a bunch of different content and you’re reusing pieces, if you insert a page break for your own personal preference, suddenly that’s going to be in everyone else’s document that also uses that particular topic.

SO: All right. So I’m hearing that page breaks are bad and I shouldn’t do them.

BS: No, they’re great. It’s just that you have to be smart about it.

SO: Okay. Sneak them in. Don’t get caught. Got it.  So, while we’re on the subject of recalcitrant authors. Such is definitely not me. What about automation in a scenario where your content production system is not being used by the authors?

BS: Yes. That’s a completely different problem. Yeah, that’s an entirely different problem because then you’re in a situation where essentially you need a production team that are also, you know, data entry people, you know, and they need to somehow bring the content that’s being produced into the system in order to produce the output.

SO: Okay, but can’t we just use AI for that?

BS: Sure, why not? AI’s great for that, right? 

SO: Huh. Right. Yeah. So ultimately, I mean, sure. If the authors are authoring and they follow some sort of reasonable template or or they’re reasonably consistent in some way, we can bring the content in automatically.  I hope you heard the first part of that about the reasonably consistent part because that’s where this breaks, right? You’re gonna get all sorts of things.

BS: Mm-hmm.

SO: And our experience on this has been that this is a problem or a challenge across a variety of different factors.  Are your authors volunteers or are they paid? If they’re paid, you have, you know, some degree of leverage to say you need to work in the system. So are they volunteers or are they paid? Is content their full-time job or is it a small part of their responsibility? So, you know, 90% of the time I’m an engineer and 10% of the time I’m a writer. Or 90% of the time I’m a subject matter expert of some sort, and 10% of the time I’m writing content. The less time a given person is spending on content production, the more likely it is that they’ll push back on learning your weird content system. Right. 

BS: Mm-hmm.

SO: And then the third question, the third factor that goes into this is sort of level of expertise. I mean, we make jokes about rocket scientists, but it’s kind of that question. If you have a person who is responsible for contributing content and they are a world expert in their field,  unless that field happens to be something like content markup and formats, they’re not gonna be that interested in spending their time learning about content. They’re focused on,  you know, curing cancer. And I’m not prepared to argue that, you know, learning some tags is actually a useful use of their time, right? So you take the content that this world expert produces in whatever format you get it. Scribbled on a napkin, yucky Word file, whatever. We’ll take it. And you find a way to bring it into the system. Now, I will say there are some subject matter experts who are actually very interested in the content production process and will jump right in and kind of, you know, play around with it and learn what they need to learn and contribute directly in the system because they think it’s interesting and fun. But when you start talking about volunteers versus people that are being paid and/or experts versus, you know, people who are dedicated to this role and/or it’s part-time, the intersection of those three, the Venn diagram or actually the anti-Venn diagram of those three things is where you will find people that will not be working in your system.

BS: Yeah. And you know, with any kind of an external contributor to content, you also run the risk of having content that doesn’t necessarily follow the conventions that you’re expecting, you know, or that your automation pipeline is expecting. So you may get something from someone who absolutely knows their subject matter inside out and sideways, but they’ll give you a document that has, I don’t know, three levels of nested lists, and your system only accounts for two; then you have to figure out how you’re gonna get that content to play well within the construct that you have. Otherwise, you have to reengineer your pipeline.

SO: And then you have to balance the cost of reengineering the pipeline for the edge case against the value that the edge case brings to the situation.  And, you know, very early in my career, I ran into somebody who has since passed away, unfortunately,  who was terrible and I mean awful at  I I shouldn’t say it that way. He wasn’t terrible at following templates. He followed the templates until it didn’t suit him. And when it didn’t suit him, he would come up with some workaround to do what he was trying to do. So your nested lists are a good example, stuff like that.  And I was, I don’t know, 20-something,  and now I’m slightly older than that. It was my job to push back on this guy and say you didn’t follow the templates. And he would literally yell, which was a fun experience. 

BS: Mm-hmm.

SO: But the thing was that in every single case, he had thought about it. He never deviated from the templates by accident. It was always, I need it this way because of this other thing. I also learned the hard way that he had thought really carefully about word choice. And so if I ever changed any of his words, we would have well, we would have words.  And he would explain to me why I was wrong and why changing his word to my word or, you know, follow the style guide, whatever, was wrong in that instance. And he was always right. Always. Now, I’ve been doing this a really long time, and I’ve only run into one person like that in my entire career.  But I have not forgotten that at least once I ran into somebody who would work outside the provided, you know, color outside the lines in order to make his document better, and always had a defensible reason now. Do I wish he had yelled a little bit less? Yes, but you know, that’s life.

BS: Mm-hmm. All right, so we talked a bit about the automation pipeline and what people can inject into it. Should we turn our attention back toward our good friend Mr. AI?

SO: Mm. All right. So it’s interesting because a lot of the edge cases that we’re talking about can actually be addressed and flattened out by AI. So AI and AI tooling actually provide a potential solution to all of this. But I wanted to focus a little bit on a different problem with AI and automation. So when we talk about the limits of automation in, like a scripting workflow or a traditional non-AI workflow. A big part of the response to that is, well, the AI could actually address what you’re describing and solve for those things. Okay, fine. But now let’s talk about what AI can’t do, the limits of automation with AI. And it’s actually a really different problem than what we’ve been talking about. We’ve been talking about sort of following the template, don’t color outside the lines. With AI, what we’re running into is that.

BS: Mm-hmm.

SO: AI and AI systems only have visibility into the digital world. And so if there’s information floating around an organization that gets passed around in a hallway conversation, typically, and I say typically with great caution, the AI is not listening. And so if I, you know, if I run into Bill in the hallway and I say, which would be unusual because we’re always remote, so but never mind. We run into each other in the hall, and I say, hey, don’t forget to update that one thing. Or  I ran into so-and-so at lunch, and they told me this other little tidbit of information. Okay. Well, I’ve now transmitted that information to you and you’ve got to get it into your whatever you’re working on, right?

BS: Right.

SO: But the AI is not aware of that because it didn’t make it into a JIRA ticket or anything else, any sort of digital workflow. Now,  this is also in many cases, it’s similar to the problem that you actually run into with remote employees that they miss out on the so-called water cooler conversations, right? Like the conversations that happen on break or at lunch or this or that. But so the AI can only see what it has.

BS: Mm-hmm.

SO: And what it has access to is digital content. So your existing documents, your meeting recordings, your digital footprint, your email. I mean, it’s super creepy,  but what it doesn’t have access to is the person-to-person interactions that are not being recorded,  or conversations that are out there, or a note scribbled on a piece of paper or a couple of these Post-it notes that are sitting in front of my laptop, right?

BS: Yep.

SO: The AI doesn’t see those and therefore it can’t automate them. And because it can’t automate them,  they don’t get injected into the workflow.  And all of that sort of detritus, because if it’s important, I’m going to pass it on, right? I’ll put it in an email, or I’ll put it in a Jira ticket, or I’ll put it somewhere where it belongs.

BS: Right.

SO: In some sort of defect tracking. But it’s those little like, hey, I just saw this kind of side conversations. That cannot be automated. I mean, unless we move into an always-on surveillance cap surveillance world, which I mean I’m sure somebody’s working on it, but at the moment, if it’s not captured digitally, the AI can’t see it. And I think that we underestimate the amount of work and the amount of information that is sitting out there in the not-yet-digital world.

BS: Mm-hmm.

SO: So that’s the limit of AI automation, which is very different from “It can’t do really nice page breaks.” It probably actually can do really nice page breaks.

BS: So basically the quality and the efficiency of what you’re gonna get out of an AI automated workflow really depends on whether or not the information itself is available and being captured.

SO: Right, which is really quite different. I mean, AI is much better at taking sort of questionable inputs and doing things with them, right? Because it’s not a pass-fail script,  but it can only see what it can see.  And so we have to understand that there’s still limits there. And and it’s it’s a different limit. It’s a qualitatively different type of limit.

BS: Mm-hmm.

SO: And I think that’s important to recognize as we move forward.

BS: Right. And of course all that stuff that does need to be captured, it needs to be done in a way that AI can access it and can access it in the right way. So if it’s input wrong or whatever, then AI is not gonna have the right info.

SO: Yep. And so,  yeah, this is where I’ve been spending a lot of my time because, as you probably know, in the world’s worst-kept secret, Carlos Evia and I are working on a book on content and AI. 

BS: That’s my shocked face. 

SO: And yeah, I know, please contain your shocked face. It’s coming soon. She says hopefully, with many fingers crossed. We’re getting there. And what we’re hoping to do is get something out there that talks about some of these issues, right? We’re not gonna write a book that tells you how to optimize for a specific AI model or specifically how to build agentic workflows.

BS: Mm-hmm.

SO: What we want to do is give you the tools to ask the right questions so that you can start looking at how to put these things together and how to move forward with a reasonable content operations approach that also involves AI as appropriate.

SO: I look forward to being done. Also, for the record, the AI is not writing the book.

BS: Like, yeah.

SO: That seems important to say in this context today.

BS: That is quite important to say, given all of the AI-generated stuff we’re being force-fed at this point.

SO: No, I can assure you that my pain is human and personal when it comes to this writing. It has been real. It has been a slog. As perhaps it should be, because the process of putting something like this together forces you to really think carefully about what’s going on.  And the process of doing that with a co-author really forces you to think carefully about what’s going on. 

BS: And real? 

SO: And so it’s been great working with Carlos and also extremely painful because that’s just how books are.

BS: Yep. Cool. I think we’ll leave it there. Thanks, Sarah.

SO: Thank you.

Conclusion with ambient background music

CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links.