Conversational AI: The cost of ignoring structured content (webinar)

Conversational AI is everywhere, but reliable AI responses depend on reliable content. So, how do you ensure your content is reliable? In this webinar, guest Rahel Bailie, Content Solutions Strategist at Content Seriously, and host Sarah O’Keefe, Founder & CEO of Scriptorium, examined how the intersection of structured content and conversational AI has evolved. They also share practical next steps that organizations can take to create a successful AI content strategy.

Rahel Bailie: How do you know your content is ready for AI? The level 1 test is, “Is the AI agent working well?” If it’s working well, then you go to, “Why isn’t it getting the right answer?” Then, you go to the content. The content can be good or bad and can be measured in a couple of ways. Is the source content marked up well? Does it have the right semantics on it? Does it have the right metadata? Do you have a knowledge graph in the background that’s making these relationships, so that the AI can pull out the right content?

Resources

• Content Seriously
• tcworld magazine: Eight content structuring techniques for smarter information
• Rahel Bailie: Content Integrity Model ebook
• Altuent: Why structured content helps knowledge managers maximise value from AI
• The true measure of success for AI initiatives (podcast)
• Want to stay updated on key industry insights? Subscribe to our newsletter!

LinkedIn

• Sarah O’Keefe (Scriptorium)
• Rahel Bailie (Content Seriously)

Transcript

Christine Cuellar: Hey, everyone, and welcome to our webinar today, Conversational AI: The cost of ignoring structured content. Our special guest today is Rahel Bailie, who’s the senior content consultant at Content Seriously. And our host, as always, is Sarah O’Keefe, the founder and CEO of Scriptorium.

Sarah O’Keefe: Hello. Hey, everybody. I’m Sarah O’Keefe, and hey, Rahel. There you are.

Rahel Bailie: Hi.

SO: Nice to see you.

RB: Good to see you.

SO: So, today we wanted to talk about conversational AI and the implications of structured content. So, I really wanted to start with the very, very basics, which is in fact, Rahel, what is structured content? For those people that might be on this call and maybe they’re new to this particular space or this particular discussion.

RB: Sure. So, when you’re deep into techcomm, everyone thinks of structured content as something that’s structured to the DITA standard, but there are really nine ways of structuring content. So, you can think of structure as starting with things like editorial conventions. We always put a title and then the description comes under the title or the paragraph comes under the title, or the section head and then the text. That’s one way of structuring things editorially. Then there’s markup, the markup that goes on. It’s a heading one, it’s a heading two. Even in Word, you can structure. It’s not as semantically structured as some of the more advanced techniques, but it’s a structure nevertheless.

Then you’ve got things like putting in synonyms for search purposes, and having some vocabulary that gets synchronized across all of your products so you can standardize that. So, that’s another form of structure. You’ve got information architecture, which is another form of structure. And then you’ve got things that are things like taxonomies, ontologies, and knowledge graphs, which are a different type of structure, because they’re structured relationships between concepts. And then you’ve got the newest type of structure is context, so context engineering or context graph. And that’s another layer on top to help deal with AI. So, there are many ways that you can structure content, but it basically means that you constrain, you put guardrails around it. And the reason for those guardrails is so that machines can process the content more easily.

SO: Right. That was the next follow-up question. While I’ve got your ear, it looks as though about three-quarters of the audience that’s here today is self-identifying as belonging to techcomm. Most of the rest are content design and just a small percentage in marcom, marketing communication. And so far nobody’s said conversational design and nobody has said other, which is unusual because usually we get a lot of other.

RB: Other. Okay.

SO: So, given this broader or maybe multidimensional way of looking at structured content, I would argue it’s like saying content, there’s lots of different ways you can standardize content across all these different axes or across all these different dimensions. The premise of this chat today is that that matters for AI. So, why does it matter for AI?

RB: Well, there are a few reasons and that is that AI loves consistency. The more information that you give to AI, the more it understands the intent of what you are looking for. So, when we talk about things like hallucinations, which we would call just mistakes, so in our industry we would just go, “I’ve been hallucinating for years, et cetera.” You want to give AI more context. This is why we talked about a context graph or context engineering. So, for example, if you’re talking about the word bank, am I talking about a financial institution bank, or am I talking about the bank of a river? So, AI, we might know, we might be able to tell, but AI may get it wrong. Not all the time, but they may get it wrong sometimes.

So, the more information you can give, then the better you can find the right answer when you’re asking a question of the LLM or when you’re, I hate to use the word, I hate anthropomorphizing AI. But when you’re chatting with the LLM, when you have input into the chat field and you get an answer back, you always have to check. So, we know that there are lots of times when, for example, in court cases, where lawyers haven’t checked the sources and it turns out that the LLM made up a source. So, why did it make up a source? Because it’s going by some formula and it’s trying to figure out what the right answer is and it gets it wrong. And so it doesn’t have enough context to get it certainly right. I don’t know if that makes sense.

SO: So, it sounds as though you’re saying that taking a random collection of PDFs and just dumping them into the AI is not the path to fame and glory here?

RB: No.

SO: No.

RB: No.

SO: But that’s what everybody is doing.

RB: Yes. Now, I have to say that not all PDFs are created equal. We have to keep this in mind because there’s the old PDFs where you have a stream of consciousness Word document, there’s no formatting. Maybe there’s some inline formatting where you highlight it. Instead of saying it’s an H1, somebody has highlighted it and then said it’s a 16 point bold and now it looks like an H1 to the human eye, but not to a machine.

And so when you look at these things, those are unstructured documents. Then you’ve got semi-structured documents, where you’ve got basic things like H1, H2, heading one, heading two. We know that it’s a list, but we don’t know why it’s a list or what the list is about. And so when you have a whole bunch of lists in a document, how can you tell what it’s about? So, the newer PDFs, if you are taking a Word document or an XML structured document, maybe even in DITA, and then you are outputting it to a PDF, there’s going to be some metadata behind it and it’s going to be better. So, there’s non-structured, semi-structured and better structured. So, it’s not great still, but that’s better than no structure at all.

SO: So, how do you go about measuring that and understanding how good is this content or how good is this information I’m feeding in to the LLM?

RB: So, that definition of good is changing vastly, right? So, we have to think about, like in the case of PDFs, you’d say, “Okay. What’s the version number? Is it up to date? Does the AI only have access to the up-to-date one? Do you need to have three versions out there, because version one refers to the software or hardware from last year, and version two is from six months ago and version three is from now?”

But you need to have them all out there because some people still have the version one hardware on the go, and other people have the version two hardware on the go. There’s all those kinds of things you have to think about. And then you say, “How do we indicate to the AI model that these versions pertain to what?” So, that’s going back to context.

But I think it’s when we think about content quality, we are actually thinking about more than content quality. So, we’re not talking about just editorial quality. We’re talking about, and I called it the content integrity model. I’ve put it in the attachments if anyone wants to go into depth about it.

But it’s basically saying you can’t think about only editorial quality, because AI is this new gatekeeper. So, it used to be you created your content SEO, you SEOed it, you made it really friendly for Google search or a search engine search. But now you’ve got your content wherever it lives, the AI models go and check it and then pass that to the search engine, and then that passes it to the human who reads it.

So, now AI is this gatekeeper and so we need to have more than just the editorial standards, if you will, so a tone and voice and so on. Now we have to think about, and it goes back to those intelligent content principles that Ann Rockley and company talked about 10 years ago. Is that content is not just the copy, it’s copy plus semantic structure and metadata.

And so when we think about what makes content good, it’s got to be good for the humans, but before it can be good for the humans, it’s got to be machine deliverable. So, machine-understandable, and machine-readable, and machine-deliverable, so that it can get to the humans. So, putting in your intelligence after the fact isn’t really a good way of going about it. You’ve got to think about it right from the beginning.

SO: I think right now the model seems to be dump everything into the AI, see what you get out, and then try to build guardrails around it, fix it, remediate, do what you need to do. Now, I can see why you would want to do it that way because that looks and smells an awful lot like an easy button that you can just drop it all in there and go, but the work is the work. And if the content isn’t good for values of accurate, up-to-date, predictable, consistent and a whole bunch of other things we could talk about, the AI will not produce good output. And so this is just like anything else.

You can either do the work upfront one time and have predictable results, or you can do the output and then clean up the output over and over and over again. And so just from a pure efficiency standpoint, fix it ahead of time, then drop it in, and then do your remediation. But do your remediation on the backend, not over and over and over again. In that context, looking at the polling results that are coming in, we asked people what their attitude is towards AI. Love it was in the lead for a while, but it has now fallen behind. So, roughly a third are saying, love it. Roughly 10% are saying hate it and the rest tolerate it. So, over half, almost like 55%, 60% are saying, I tolerate it. It’s there, I tolerate it, I put up with it.

RB: So, I think that that might be an overly simplistic way of looking at it. I understand that this is why we set the poll up that way, but they love it or hate it. When it’s working well and I don’t even really notice it, I love it. When it gets in my way of doing what I want to do, I hate it. So, it depends what we’re using it for.

SO: It depends is always the answer.

RB: Yeah. It depends is always the answer.

SO: Always.

RB: Spoken like a true consultant, and I do that all the time.

SO: That’s exactly right.

RB: But it is that when we’re being told do AI, do what with it? So, when we are being expected to apply it inappropriately, of course we’re going to hate it. It’s like saying, “I want you to write a manual. Now put all the content, each sentence into a cell of an Excel spreadsheet, and then we will assemble it by having the developers write a script.” You even be like, “I hate this.”

SO: Yeah. So, I’ve got a bunch of questions that I want to get to that are related to this question of structure. I wanted to touch briefly on provenance, which I heard you say earlier. I think that one of the biggest challenges we have going forward is the question of provenance, which comes from the art world. Where did this piece of art come from? When I’m buying it, where did it come from? Who owned it previously? Was it stolen at some point? Provenance nearly always is related to the question of, is this a stolen piece of artwork? Or can I prove that it’s not a forgery? Because I know that it went from, I’m definitely buying Rembrandt. It went from Rembrandt’s workshop to here, to here, to here. We can trace that chain of custody, that provenance, down through the years, the centuries in that case, and prove that this is an original.

I think that ultimately with AI content, we’re going to have a very similar issue around provenance of what we’re feeding into the AI. Is it up to date? Is it accurate? Is it well-structured and all the things you’re touching on? So, in that context, I’ve got a couple of people asking fundamentally the same question, which is how do you know that something is AI ready? I’m going to try and package these all up and drop them on you and see what you can do with that. But the question is, so first, let’s start with how do you evaluate content for AI readiness? And then I’ve got some DITA-specific questions that people are asking.

RB: Sure. Yeah.

SO: So, evaluating for AI readiness, let’s start there. How do you do that?

RB: Well, that’s a really interesting question. I have to say that last week or was it two weeks ago now? Anyways. At the end of April, I was at a conversational AI conference. And it’s interesting how they’re testing because they’re the ones that inherit the problems of inappropriate content in the repository. I don’t want to say bad content because it could be very, very accurate. It could be really well crafted, but it’s not working. So, content that doesn’t work. And they’re the ones that have to deal with it because that’s who their compliance department will come to first to say, “Hey, why are you giving people this answer? And why is it not working?”

So, how do you know it’s ready for AI? Well, you can have Altuent, where I am a fractional strategist. They’ve developed this reliability score and it doesn’t test the content itself, but it tests how well the content can be delivered. So, it’s testing the agent itself, like how is it pulling out the right answers? So, it could be that level one test is, is the AI agent working well? And if it’s working well, then you go to, now why isn’t it getting the right answer? And you go to the content. So, the content can be good or bad, can be measured in a couple of ways.

So, one is, is the source content marked up well? Does it have the right semantics on it? Does it have the right metadata? Do you have a knowledge graph in the background that’s making these relationships, so that it can go and pull out the right content? So, that’s one level of it. If you’re answering straightforward questions like, what’s the part number for blah, blah, blah, nuclear reactor do hickey.

SO: Maybe not that.

RB: But the part number is the part number. So, it’s not like one of the examples was for a bank where it says the person asks, and they’re doing this as a test, so they’re testing. “I want to invest money. My grandmother died and left me some money, I want to invest it.” And the reply was, “Great that you got this money, what stopped you from investing it before?” Inappropriate, right? But it doesn’t have to do with the source content. It has to do with the LLM interpretation. So, they have to fix it at that end, but they need to also figure out what the source content is. Because if there are, I don’t know, three options for investing, they need to know, are those three options there and can the LLM pull them out appropriately? And if it can, then you’ve got the problem at the front end. But if it can’t, then you have to go back and remediate the source. So, now instead of looking at SEO for our content, we have to be looking at AEO and possibly GEO. 

SO: Which are?

RB: A GEO is generative engine optimization, and AEO is answer engine optimization. So, the GEO is when you want your company to get mentioned, you want it to have some authority. So, you’ve got what’s the best widget maker out there, and it lists five companies and yours isn’t there. That’s a GEO problem, because your company isn’t recognized as an authority.

So, how do you get it recognized as authority? You have to have your company name there. You have to have it tagged up with the company name, and you can ask an LLM to do this. This is where I love it, because I can put in a bunch of content and say, “Can you mark this up for GEO?” And then I can see what the markup looks like and go, “Ah.” I did this for a government.

And so they said it’s a different country that keeps getting mentioned, because they asked the question and that country is more prominent. So, how do we get you more prominent? So, we have to put in your country name, we have to mark it up as a country. We have to mention the legislation, and then we have to mark it up that this is a piece of legislation. We have to mark it up that this is a government service. And so now it’s going to bring your rankings up.

Then there’s the other, the answer engine optimization, and that’s more on when the LLM wants to get the most appropriate answer out. So, it looks maybe in a big long paragraph, but there’s that one sentence and that’s the actual answer if you need to answer it in one sentence. That’s where you mark it up as a microfact.

And so microfacts, when you mark those up, they’re more likely to be what the LLM will pull. And so you can mark up your content in a way that’s like maybe you need to have that context.

But when someone asks, I don’t know, “What’s the deadline for filing my taxes?” Well, the deadline is the deadline. So, if you have that thing about you have to file your taxes by this date, and then if you don’t, here are the penalties and so on and so forth. But maybe that’s the one piece that you mark up as the microfact, because you know that that’s going to be pulled.

SO: And so then I’ve got two questions here that are specifically related to DITA and source content that are interrelated and tie quite nicely into what you just said. Because one is, so if the source content is in DITA, then two questions.

One is it’s in DITA, but we generate PDFs to feed it to the AI, is that a good idea? And then secondly, if the source content is in DITA, is that already sufficient structure? What additional essential metadata should be added in addition to what the DITA elements provide?

RB: So, that’s actually a really good question. So, AI actually loves DITA content. Ironically, our time has come. So, the DITA content is… So, I suppose you could mark it up with such basic elements and attributes that it can’t recognize. But if you’re using DITA in a way that’s got those inheritances, so that you know that it’s in this language, it’s for this product, it’s for this product line for this product and so on and so forth. It has all of that information because that’s what DITA is built for, then when you’re delivering your content, it’s going to be that much richer.

If you combine that with a delivery standard called iiRDS, it loves it even better. So, depending on where it’s being delivered. So, if you’re doing a public knowledge base where a search engine comes in, I don’t know about that as much. But I know that if you’re delivering content, for example, you’ve got a knowledge base and then someone comes to your knowledge base and types something in, you’re delivering directly to consumer basically, then you’re going to get even better answers.

So, now, do you put it into a PDF? It’s missing the point. If you need to have a PDF, you can create a PDF as well as having your content in topics living in the knowledge base, so it’s much easier to read the topics. But doing one doesn’t mean you can’t do a PDF as well, because there are still people who need to download a PDF for whatever reason and it should be available to them.

SO: I would argue that the question of, do I produce a PDF from DITA is different than the question of, do I connect my AI to my DITA source content via PDF? You could still output PDF for consumption, but output something different for consumption by the AI, or use an API to connect it or something like that.

RB: I don’t even know if you would connect it to your source DITA. You would output, but you would output in HTML or markdown because AI likes that.

SO: Well, there’s some other things we can do there, but yeah.

RB: It’s complicated. I think you would output what you want the AI to see, but you don’t have to output it in a PDF. You can output it to HTML and keep it in the knowledge base and let the LLM draw from that.

SO: I think the big picture answer to the question, which was it’s DITA on the backend, but we’re converting to PDF and shoving that into the AI, probably because the IT team said, “PDF is easier for us.” Is that the optimal solution? No, it is not because the content is getting flattened when it goes into PDF. You’re losing a lot of structure. So, is it optimal? No. Is it better than a hot mess? Yes. Because you’re going to have some inherent implicit structure, because of the DITA files, because of the DITA structure itself. So, it’s one of these if that’s the best option that you have today, given your tool set and your organization, then that’s maybe the way to go.

RB: Sarah, by the way, you’ve disappeared off the screen. Oh, I can always-

SO: Amazing.

RB: Agree.

SO: Yeah. It comes and goes. Welcome to internet fun. Rahel was the one that had hail apparently just now. But the hail in London has affected me in North Carolina, clearly.

RB: There we go.

SO: Somebody asked about types of structured content and whether there is a method for measuring maturity of structured content in an organization?

RB: Yes, there is. And that’s probably an entire webinar on its own. So, you set benchmarks, and different organizations set different benchmarks because they have different business goals. I’m going to go back to this bank that had the chatbot.

They handle millions of conversations a year through their chatbot. The situation is that it’s pulling from various sources and the conversation designers control those. So, they do it by feeding in bad examples as well as good examples, and then saying like, “When you answer, don’t sound like this, but sound like this.” Because it’s hard to measure to say you don’t have enough empathy or those fuzzy goals.

In a manufacturing environment, very different because they want concrete answers that come directly from the content source. So, you can do tests, you can do certain kinds of tests. The most basic one being if I ask this question, what comes up? And do we get that consistently over the course of a period of time? One of the things that we have to keep in mind is that every time you add to your repository, now you’re changing the composition. So, it’s like when you have a recipe and then like you add more salt, now it’s going to taste different. Well, you have that same thing. You have this big soup of content and now you’re adding more ingredients.

And so it’s going to change what comes out at the other end from the LLM slightly, depending on how fast and a lot of factors. But so you have to keep testing, it’s not a one and done. You’ve got to keep going in there and saying like, “Are we still getting consistent answers? We’ve added 10 more documents, or 100 more documents, or 500 more topics. Are we still getting the consistent answers?”

There are different ways you can test it, particularly if you’re using PDFs. You can ask questions that pull from the PDF where there could be ambiguity. So, for example, I’m going to use an example that a vendor has said to me. Is that if you’re in life sciences and this says you have this dosage, now you have this dosage for adults, this for teens, this for babies.

Now, if those were tagged subheadings, then you would probably get a better answer than if somebody had just said, “That’s a level four and we don’t have the level four in whatever system we input our content into. So, I just make those a bold heading.” I just type it in and I make it bold and now it’s a heading.

So, if you want to ask what’s the dosage for this medication? And then you ask, “What’s the dosage for this medication for an infant?” And then see what it does. Does it pick the first one or does it actually understand that it’s down further? So, you have to be careful with your question answer pairs when you’re testing, and then you can get more sophisticated as you go.

And as I said, you’ve got this bank who’s doing good answers and bad answers, so that you can now instruct the LLM to do something more. You can build yourself a dashboard based on all sorts of criteria. When you look at the content integrity model, there’s a business, does it meet the business goals? Does it meet the editorial goals? Can you operationalize it? Do you have the right infrastructure? So, you start to go like, “How does it measure on quality of metadata? How does it measure on updateability and provenance? How does it measure on editorial quality?”

And so if you combine them all together, you could get a score, but I don’t think you’ll ever get to 100% just because of the nature of AI. You can get pretty darn close sometimes, but it’s just the nature of the beast at least now.

SO: Yeah. So, related to that, we have a side question about disclosure. What degree of responsibility exists for disclosure of AI use for structuring or organization? Do you tie it to individual pieces of content, or also to the library or the collection?

RB: Oh, such a good question.

SO: I left a response on this. I’m not aware of any legal responsibilities. I think the question is responsibility from an ethical point of view, and perhaps there’s something in the European AI Act, but I’m not sure. What’s your take on that?

RB: I just had a chat this morning with someone, or was it yesterday, that in the EU AI Act, there’s actually an entire section on technical documentation. So, go and read the EU AI Act. I can’t remember which section it is, but it’s probably Section 9, 10 near the back somewhere.

There was also a very interesting post today on LinkedIn from someone, I can’t remember her name. I think it was T-H-Y-S was her first name, and I can’t remember the last name. It’s not somebody I follow, but somebody had reposted it and I happened to notice it. And they said it’s basically, it’s all fun and games until something goes wrong and then the regulator wants to know whose fault it is?

And then everyone is scrambling to look at the provenance; where did it come from? How did it get out there? What did you do to remediate it and so on? So, there are regulations around it. The EU AI Act is a good place to start because even if you’re not in the EU, if you’re serving customers in the EU, you have to think about this. Generally, you have to disclose in the EU AI Act. You have to disclose if you’re a deployer, if you are a creator, deployer or you use it, but you use it to pass it along, pass the results on. So, you have a duty to declare.

So, it gets a little bit complicated because do you say, “Well, I use Microsoft Word with Grammarly and so I used AI for spell checking.” That’s probably not a declarable reason. But if you are using it to do anything where the AI results, it’s drawn conclusions or it’s created something that you’re using as is, then you have a duty to disclose.

SO: So, we did ask people about how much work they’re doing with AI and content, and the most popular answer here was with caution. How much work are you doing with it? I use it with caution is about 60%. Our 10% haters are represented only when required to at work. And there’s a strong third that’s saying as much as possible. They’re using it as much as they possibly can.

So, that’s maybe not too surprising given this particular subject matter. So, I wanted to change gears a little bit and talk about the question of the roles of content creators, technical writers, and then conversation designers and how they can collaborate? I think probably the first thing we have to start with, I’m not going to make you define technical writers, but conversation designers. What is the conversation designer?

RB: So, conversation designer, even that role has changed recently because before, it was people who wrote the scripts for the pre-generative AI chatbots. So, there’s these things called utterances and entities and so on. And so you would actually design the question path. So, if you think about any of your interactions with a chatbot in the past few years, that’s what they did. Now, it’s up a level. It’s more managing. So, it’s prompting. It’s making sure that the prompts are good, so that’s part of it. But I don’t want to call them prompt engineers because everybody is writing prompts. But they are managing the outputs and all those things I was just talking about. You are looking at where the content is coming from, how it’s getting processed. You’re working with the technical folks to tweak algorithms and so on to make sure that you’re getting good output. And you’re making sure that the output is appropriate, that it may be accurate, but wildly inappropriate. So, that’s where conversation designers are right now.

SO: Okay. And so they’re the people responsible for framing this up. So, then what does it look like for that role to be interacting with the people creating the content on the backend?

RB: Well, I have to tell you, when I did my presentation at the conversational AI conference, I thought nobody would really care much, because I’m conversation AI adjacent as a content strategist. But people were actually really interested.

The reason I say this is because every presenter that day, on day one, was mentioning content debt. How do we reduce content debt? And how do we structure the content so that we get more reliable results? That became this running theme and they can’t do it on their own. They’re working with whatever is there.

So, the people who are feeding them the content, in other words, the tech writers, the content designers, are the people who actually can make a big difference. So, you have to collaborate now. You can’t just say, “I’m just going to work in my little silo.” We actually have to have those cross-discipline collaborations. When you think about cross-discipline collaboration, you’re thinking about connections. So, you’re developing stakeholder relationships to increase the value of whatever solutions you’re working on. You’ve got cooperation, so you have to work across the entire content ecosystem. And that’s looking at your corporate culture and what incentives do they have to work with you or that you can work with them?

It’s like high school, the power dynamics of this group doesn’t want to work with that group. And you have to think about what’s the end goal? And so how do we work towards that? And then you’ve got that coordination because you have to figure out if we do this and they’re doing that, what’s the most efficient process for us to get this into the pipeline for them to use and so on? And then there’s the capabilities’ aspect of it because if you don’t have the right skillsets, then you can be trying and trying and trying and you’re never going to get anywhere.

So, when we think about all of these together, we have to go, “Well, let’s figure out how to create the content in a way that’s going to be useful to be delivered in however we’re delivering it.” More and more now there’s going to be some AI enabled chatbot, so the folks at that end need to be working with the folks at the backend.

It’s going to be iterative because you’re going to create something that you think is going to work, then you have to test it, and they can do a lot of the testing. And then they’ll say, “Okay. Here are the results that work. Here are the results that don’t. Let’s troubleshoot.” And then you go around and round until you figure out what the magic formula is for your particular organization.

SO: Yeah. The troubling thing is as little as maybe five years ago, the chatbots were, it was almost like a decision tree. You go in and it says, “You’re here for tech support. Would you like to return something or buy some more?” You could only click A or B, and then it would take you through this flow and it was pretty predictable. It was also for the most part being hand built. So, we had this whole issue around, we have this enormous amount of content, but it’s being rebuilt in the chatbot system, because the compatibility between that and the backend content was non-existent for the most part. But now-

RB: The conversation designers that I was working with a few years ago… So, I had someone on my team who was a conversation designer and well, we had a couple of conversation designers, and it was painful to watch them work. Because they would work in Excel, so they would create their scripts in Excel, and then on the other side they would have a flow chart.

If they realized you missed something, you’d have to go back up and you could insert that, but then you’d have to redo the entire flow chart. Just working back and forth, back and forth, it was quite laborious. And you’re right, it was like, “What’s my baggage allowance?” So, it would be like, “Are you in economy, or business class, or first class?” And you’d pick one and then they would make this, it was a decision tree. It was the conversation flow.

I’ve done that myself, not a lot, but I did a project where I was doing that and I just thought, “This is not for me because that’s not my core skillset.” But they did literally just map out the flow and sometimes with crazy results. I had a travel thing where it’s like, “Where are you going?” I was going to Reykjavík and I couldn’t remember how to spell it. I went Iceland and it said, “You can’t swear.”

SO: What?

RB: You’re a travel chatbot. Anyways. It turns out I looked up on Urban Dictionary, you don’t want to go there.

SO: Don’t do that. Not for show notes.

RB: Yeah. But now that’s not the case. Nobody is writing those scripts anymore. What they’re doing is they’re saying, “What are all the things that people are bound to ask?” And they’re going through logs and so on to find out what people are asking. And then they’re saying, “Where is the source content, and how can we make that source content available?” And then the LLM will do the actual conversation. So, instead of just going, “Show me your document. What’s my baggage allowance?” “Let me transpose it into my little chat flow here.” Now you just say, “I need to have a baggage allowance document.” And the document has to say what is the baggage allowance for each different cabin type and each different ticket type. And now when the LLM goes in, it will pull it from the source content. So, who’s writing that?

I say this and I apologize in advance to the content designers in the audience. But everything that conversations designers do, tech writers used to do, but with better software. Content designers are using Word and Google and they don’t have that ability to mark up semantically. But if you’re a tech writer and you’re using an XML based editor that’s to a DITA standard, you know when you’re writing, you know why you’re writing it. So, you’re going, “Baggage allowance, tag with economy class, tag with business class, tag with,” because you know why you’re writing it. So, you’re writing it and now that becomes that definitive source for the LLM to pick up.

SO: Yeah. I would say for that specific example, even a table is probably enough to give people that context because now the table provides information on effectively two axes, where one of them is class of service and the other one is baggage allowance, or number of people, or time of day, or plane, or whatever. And so if you query, it should be able to figure out which it, should be able to figure out, there we go again. Which cell in that table has the right answer in it now.

RB: Well, should’ve, would’ve, could’ve—

SO: Should’ve, would’ve, could’ve.

RB: … because it depends on the LLM that you’re using. It depends on how the table is formatted, depends on so many factors. So, that’s where the iteration comes in. So, you send over a table and you go, “This should work.” And then they test it and they go, “It’s giving us false information. Let’s go back and redo how the table is.”

SO: Okay. I’m going to ask you to draw some conclusions. Our last poll, it looks like there’s an even split between, I’m pretty good at this AI thing and I have more to learn. There’s a few people saying they’re experts. A very small number see a lot of limitations and nobody is actually flat out refusing to participate because again, if that were the case, they would not be on this call. They are already boycotting. So, for those of you on the webinar, if you want to get your questions in, I would say do that ASAP and we will try to get to them towards the end. Because Rahel, what I want to ask you is what are the next steps? As a practical matter, what is the next step for someone to start thinking about successful AI outcomes?

RB: Okay. Well, first, you need a strategy. Everybody is going through this transformation and they’ve been doing transformations for years, but now we’re into the next phase of the transformation. So, you could be calling it a digital strategy, a conversational AI strategy, a delivery strategy, call it what you will. Each company has its own jargon. It doesn’t matter what it’s called. What you do need though is some direction, like what do we want to get out of this? Once you don’t know what you want to get out of it, then you can start saying, “Now we’re going to shape the content to be able to deliver.” So, it’s all about the delivery of the content in a reliable way. So, we assume the content quality that we all know how to write, we write well, we know how to be consistent, et cetera, we know the importance of that. We’re professionals.

The delivery part has always been the weak point. The whole thing of we’ll do this in a headless CMS, and then somebody else down the line will figure out all the magic to make it happen. It’s not that easy anymore. So, we have to take a bit more responsibility for doing our bit to make it happen. So, some of that is just working with the technologists as well to say, “Well, what do we need to do at our end so that you’re more successful at your end, so that we’re all successful for the customer and the business?”

SO: So, I had a question early on, I’ve lost the context for it, which in the context of this conversation is ironic. But somebody asked, what does evidence-based actually mean for AI?

RB: Wow. It depends. It depends. How can I even start to enter this? I feel like I’ve got this big grapefruit and I’m looking for a way to stab it, so I can start peeling it. If you’re in a non-regulated industry, it’s going to be different than if you’re in a regulated industry. So, the way I’m interpreting it is how reliable is your output? And so if you are a pharmaceutical company, you have to have very high accuracy results, because you don’t want anyone overdosing or hurting themselves inadvertently. If you are a manufacturer, non-regulated industry, and you are more worried about just getting accurate results in terms of the source content is this, the questions are more straightforward and we have pretty well concrete answers to give them. Then you may not be as stringent and evidence-based may be something very different. It could be, do I always get the rights from the specifications?

If it says that, I don’t know, with a car that you have to inflate your tires to a certain PSI, does it always give you that certain PSI? That’s pretty straightforward. When people ask about things like medication, they could have comorbidities, they could have all sorts of things that affect the answers. And so you wouldn’t want to go like, “It says half a teaspoon and half a teaspoon, done.” So, I think it really depends on the strategic direction, and that’s why it’s so important to have the why and the what and the how of putting it all together.

SO: I will say, we’ve talked a lot about different resources. I did a webinar last week and we were talking about what it means to produce a user interface that is supportive of AI content? There were a lot of things flying around there, but some of them had to do with sourcing and citations even within synthetic generated content. So, I think that is a question or maybe a direction to look at to consider this. There’s another one while I’m plying you with all these big, big questions. When you designed help content for accessibility, does that help make your content more digestible for LLM modules?

RB: Anything done for accessibility is usually better for machine readability in general. So, I would say yes. Even those things like alt text. We’ve known for years that alt text is also better for SEO. So, it’s going to be better for AI as well, because it can read that and it provides more context. But that depends on the quality of the alt text. If it’s woman eating an orange, but if it’s woman eating whatever your brand of oranges that you’re selling are packed with vitamin C, then that’s going to be a different level of alt text. Citations, I don’t know if any of you use Google search with AI enabled, and I have done this because it’s my job. I have to experiment with these things and figure out what works and what it doesn’t. They will give you citations and then on the right-hand side, if I click that, this will open and it will tell me which website it’s from. It doesn’t mean that the website is necessarily right, but it tells you where it’s getting it from. I think that’s a very important aspect of it to get reliable results.

SO: I will say I went down a lengthy rabbit hole on this because I asked a question and got a response. I said, “Where did you get this information?” Which is different from what are your citations. But I said, “Where did you get this information?” And got a company name, an invented company basically. I kept probing like, “Who is this? Who’s in this company?” Actually, the first question I asked was, “What’s the URL for this?” It said, “Well, it’s more of a consortium and there’s not really a URL. You should go focus on the people that are part of this organization.” Okay. Great. Who are the people? And it said, “You are so smart to focus on the people.” Thanks. And then it said, “Well, it’s really hard to figure out who they are.” But eventually it gave me a name and of course it was a name of somebody in our industry who does not in fact run the company that was specified because it doesn’t exist. I spent a decent amount of time following this rabbit hole down. I could never get it to admit that it had invented this reference, but that’s 100% what happened. In an ironic twist, the company name was Synthesis.

RB: Oh, interesting.

SO: Yeah, it was super fun.

RB: So, one of the things that thanks to Charlie Southwell, who’s the marketing guy at Altuent. He took a course and then he passed some information along to me, which I found very helpful. And that is to set up a skill in the AI model where you say how you want it to work with you.

So, the things are if you create anything for me, it’s got to be in my tone and voice, and here’s an example of, or a couple of examples of my writing. You are not to make anything up. You are to cite everything. There’s a whole bunch of criteria, and you are to give me an assumptions log at the end. And then every time you open the AI model, it runs that skill and then that’s the way to work with you. I found that that saves me a lot of that back and forth of the rabbit hole, because it’s going to cut down on that a lot. Now, you never know if it’s going to cut down 100% on it, but it’s pretty good.

SO: Okay. So, this is all super fun and interesting. We’ve got a couple of minutes, but we need to throw it back to Christine to close us out. Do you have any parting words for our audience before we jump back to Christine?

RB: I would say that our world is changing and changing rapidly. Our professions are changing and changing rapidly, and to keep up we do need to figure these things out. Just like we had to figure out the web, and we had to figure out structured content, and we had to figure out all these other things. So, it can feel like a daunting task, but I don’t think we can close our eyes to it. So, it sounds like everybody’s got their own, either they are using it at work or they’re using it as little as possible. But I think that being able to be those people who go, “Don’t worry, I know the answer to this. I know how to structure the content, or I think I have a solution for you.” I think it’s going to raise the profile of what we’re doing from tech writing. Which really, when you say you’re in techcomm, it’s 20%, 15%, 20% of what you do is actual writing. The rest is all what they call soft skills. It’s stakeholder management and research and so on and so forth. In that way, we may be elevating ourselves to garnering more respect when we can get other departments out of mind. I hope that didn’t sound too judgy. I didn’t mean it to be judgy.

SO: Seems like a good closing statement. So, Rahel, thank you again. It’s always a pleasure to see you. And I will throw it back to Christine.

CC: Yes. Thank you both so much. I’m going to turn our slides back on for a minute. So, thank you all for being here today. If you can take one minute to go ahead and rate this webinar and provide us some feedback, that really helps us bring you the content you’re looking for. So, if you can take a minute to do that before you hop off, that would be fantastic. Also, if you want to check out the resources that Rahel and Sarah mentioned today, a lot of them are in the attachment section. We’ll also be sending out an email to attendees tomorrow with more of them. But a lot of them are in that attachment section, so I would go and I would check that out. Thanks for being here today. And if you want to stay updated on the next episodes, the upcoming episodes in our webinar series, go ahead and subscribe to our newsletter. It’s the Illuminations newsletter by Scriptorium that is also linked in the attachments. Thank you all again for being here, and we hope you have a great rest of your day.

Stay updated on future webinars by subscribing to our monthly newsletter.