Charlie: Welcome to the CMO Series podcast, where we discuss all things marketing and business development in professional services. I'm Charlie Knight, and in a special episode of the podcast, we're taking you back to CMO Series Live in New York this summer. In this session, Ali Bone is joined by Joe Green, Chief Innovation Officer, and Monica Rodriguez-Kuniyoshi, Chief Marketing Officer at Gunderson Dettmer, to discuss their experiences of integrating generated AI with their experts. As early adopters of AI, they share insights on how to effectively navigate the fine line between automation and authenticity. During the discussion, Joe and Monica share real-world examples of how they've successfully deployed AI tools at Gunderson Dettmer, while maintaining the human voice behind their brand. So let's dive straight into the session.

Ali: So to open up, we're going to keep it really nice and simple, just to make sure that everybody's on the same page. What is the difference between generative AI and AI?  

Joe: Well, based on all the hands raised in this room, maybe this is the 201 crowd, but I do a lot of 101 sessions. So we thought maybe it'd be a good idea just to level set. So a lot of the conversations that I have around this on the technology side, we've built with this technology, we look at a lot of products that incorporate the technology. And we talk a lot with folks who are thinking about policy setting for, you know, organizations like big law firms around how they use AI. And working with our clients. A lot of times the conversation gets lost in the terminology. So we thought it would just be helpful to start with something simple.  So AI is really kind of an umbrella category for a whole lot of different things. It includes things like self-driving cars, as well as a bunch of legal tech tools that we've been using for 30 years. Westlaw, LexisNexis, they had AI in the 90s. They also had a lot of the underlying  technology that has kind of led to this generative AI revolution, which is natural language processing.  That's old too. That's been around for decades. So what's really new here is, you know, within this broader kind of umbrella category of AI, we now have kind of a new flavor of AI, and that's taking language models,  models that are built to understand and be able to create language. Those models are very large. That's the name, large language models. They see lots and lots of data, scraped data from all over the internet. And based on what they've seen, what they've learned, now those models can generate language. So when we say, is it AI? Is it generative AI?  Really, if the model, if the thing that you're looking at is generating language, then it's generative AI. If it's not, it may still be using the same kind of underlying technology, especially if it's extracting language, manipulating language, things like that. But having a bit of an understanding of the nomenclature, I think, can be helpful, especially as we continue in this conversation. 

Ali: So actually just feeding into this, wanted to continue along that theme of understanding a little bit more around it. So why do you think it should be something that's really important to everybody that's in this room in terms of understanding AI? 

Monica: It's funny because when Ali asked this question originally, I thought this is the why should you care question.  And there's a couple of things I started to reflect upon. One is that 25 years ago, I walked in the doors at Fried Frank. 25 years ago was a different world in legal marketing. How many of you were in legal marketing 25 years or more ago? Absolutely. It is unrecognizable as an industry, not only the kinds of things that we have to do and are expected to be able to do, but the way that we need to leverage data and information in new and amazingly interesting ways. And for those of you who maybe weren't in this industry 25 years ago, email was new, right? We're really talking revolutionary changes. And I've often said to my team, you know, the bots are coming and they're going  to come for everybody's job, not just the jobs of the the folks who are the easiest to automate. And they're absolutely coming for our jobs too, right? But now the bots are here. Thank you, Joe.  But now the bots are here and it's like, how can we enter into a marriage between these technologies and this intuition, this human intelligence, emotional intelligence, the ability to transform organizations? That can't be done by the bots, not yet at least, but it can't be done by them. And that's where we bring something really special. But if we still act like email is new, we would have never gotten here. And if we don't embrace and figure out a way to marry the human intelligence with the artificial intelligence, we're not going to go where our lawyers need us to go in the future. And so that's why I think the moment is now. What did someone, what I think Connor said, you know, you can either plant the tree 30 years ago, or plant it now. Now is the time, and I'm excited about the potential for how we can use AI in effective ways to move this industry forward and to move legal marketing forward as well.

Ali:  Yeah, 100%. And actually, one of the things that's going to be a real key theme in this conversation is AI is obviously a big, big topic at the moment, but something that we've been discussing a huge amount between the three of us is actually it's that intersection between human intelligence versus artificial intelligence. And talking about being the very forefront of things, Gunderson & Dettmer clearly are leading the way in this in many ways. You've created your own systems within the firm. It'd be really interesting to hear from you around what you think the advantages are about being those early adopters, because actually often within the legal industry, we see there's that race to be second. So I would love to hear from both of you on what you find the advantages are of actually being those early adopters and leading forward.

Joe: I think Monica might have been our, maybe one of our earliest adopters. Even before our risk management folks realized there was this thing called ChatGPT, Monica was an expert prompt engineer, coming up with all kinds of great ways to leverage it in her work. And like many of your firms, I'm sure, you know, when GPT-4 came out in what, March of 2023, you know, I was at a legal tech conference and here in New York City, actually, and our malpractice insurance provider, I'm sure many of you share the same malpractice insurance provider, had a blog post that said ChatGPT is not ready for prime time, and you should all ban it. And before I even got back to my desk, our risk folks had a memo banning ChatGPT for anything, including marketing, not just using it for practice, not just using with client data, just no, nobody touch it ever. And, you know, we had to do some soul searching, we are a, you know, exclusively focused on the innovation economy as a law firm, we represent 1000s of venture backed startups, and all the the top marquee, you know, venture capital investors, you know, we don't want to look like Luddites. It's a bad look from a marketing perspective for us. And so we had to have, you know, a meeting kind of all among our top, you know, our general counsel, our managing partner, our technology management committees, our risk management committees, and kind of hash out what the right approach for us was. And so the way that we ended up going with it was we actually allowed all of our attorneys, as well as all of our business professionals to use ChatGPT from  that that point with some guardrails in place. And I'm sure we'll talk more about that. 

Where we kind of took it a step beyond that was, you know, because ChatGPT was this public-facing, you know, thing like Google, where, you know, anything you put into it, they're going to use it however they decide to use it. You know, that was a real concern for us. And while we basically told everyone they could use it, we said, you know, if you're going to use it, you need to make sure that you're not putting anything sensitive, confidential or anything, you know, that could be kind of untangled into that system. So what we decided to do is we were looking at some of the third-party tools that were out there in the marketplace. We saw what they were doing, and we had an intrepid engineer, maybe foolhardy, but it's only foolhardy if it doesn't work, who said, you know what? I think I could build that, and it wouldn't take that long. There's a lot of open-source software out there that is just kind of available, and I've been playing with it on the weekends, and I think I could build something. So we ended up building our own internal generative AI tool that had two features. You could chat directly with the open AI models, so very similar to ChatGPT. Or you could do something called retrieval augmented generation or RAG, which some of you may be familiar with. That's what a lot of the third party tools that use this technology, how they make it more reliable and better and less likely to hallucinate is basically by taking documents that you upload. Convert it into something that's machine readable. And then when you submit a prompt, instead of just going to this model and having it come back with whatever it comes up with, it goes to your documents that you uploaded and finds the most semantically similar text and packages that together with your prompt and tells the model, hey, use this information and then answer my question. So you just get much better results. So we built something that had those two capabilities. We launched it to our entire firm in August of last year. We had a beta period over the summer before that. Monica and our marketing team were a very big part of that beta process and to this day are some of our heaviest users of that tool.

Monica: It's funny. I often like the story. I'll tell the story of one way that we used ChatGD, which is our own system, in a way that accelerated how easily we were able to respond back to reporters. So we put Joe on a huge talking circuit. Joe was out there talking about ChatGD, I felt like everywhere. And obviously, as good marketers do, we gather up everything that has been said, that has been written about him that he has actually used. And we're talking on the phone. He's like, I got this reporter has a new set of questions for me. I don't know where to start. And I said, well, why don't we take everything that you've said and we'll feed it into ChatGD and we'll ask ChatGD to now answer the questions in your voice. It knows who you are because it has everything that you've said so far, all the transcripts of every presentation you've given, et cetera. And in the course of a 15-minute phone call, we'd gone from, I don't even know where to start answering these questions because all in my head, and there's so many things I've said, to I have actually pretty close to the answer. What did you guess it at? 90% accuracy?

Joe: I was very skeptical. I said to Monica, I was like, I don't know. I don't think it's going to do great at that. I'm probably just better operating it. She was like, no, no, try it. So we uploaded the documents and I was like. Yeah, I mean, I'm going to make a few tweaks, but that's about 90% of the way there. I was pretty shocked. And I've used it a bunch of times for that since then. 

Monica: Well, and I often think of, and I encourage my team to think of ChatGD or any of these AI tools as your intern. And just like you wouldn't give an intern, you wouldn't put an intern in front of your managing partner, I wouldn't put the content that it produces in front of Joe immediately, or I wouldn't expect, you know, I'm not going to send off to a reporter that way, right? And so it really does take the person who is the most knowledgeable about the output. It requires someone who has discernment, human intelligence, and certainly Joe to look at what it came up with and tweak it and finesse it and make sure that it really does communicate something also different to the reporter than the things you've said every other time. And it gets really, really close. And it means that, at least on my team, we use that to help us generate new ideas. Because unlike an intern where you say, can you come up with three alternative captions for this LinkedIn post?  Can you generate 50 alternative captions? And are most of them garbage? Absolutely. Are some of them wildly off? But there's nuggets in there. And that nugget may be in the 15th one where your intern would not have been able to produce that volume of output. And so that makes it a lot easier. Then we take the real people who are communicators to look at it and say, oh, I like that one line from here and that other. And now we have something useful that we can say that is actually also accurate. But captures the nuance and some of the human ingenuity that comes in. And so I think that that's really where the magic happens. It's an accelerant to being able to do the work that has to be done every day. And we're all stretched budgets, resources, etc. We're all trying to get a lot done with a small amount of time, much less money and people. And so how do we add or speed up that process? I have found that these tools are very, very helpful.

Joe: Yeah. And I think, you know, had we just sent the reporters questions using the ask the AI mode, just sent it to one of the GPT models from open AI, it would have written some stuff. Some crazy stuff. I mean, like, that's the kind of thing where it just hallucinates. You say like, oh, tell me, you know, right, answer this question about ChatGD, and it will make up an amazing story about ChatGD that will probably bear no resemblance to reality. But because we were able to give it this corpus of materials that actually did come from a human, then it's able to manipulate it and change it and tweak it and kind of go from there.So kind of the humans at the beginning, the humans made this, right? Like, you know, it knows how to speak language because it's read everything on the internet, right?  It's all built on our ideas and the way we speak and the way we communicate. But then on the back end of it too, having that human in the loop is really crucial because in many cases, again, the outputs are unreliable. The way these systems are built, they're a little bit random. And I mean, they can be very random depending on how you have them set up. And so you may ask a question today and then you'll get a totally different answer if you ask the same question tomorrow in a week in a month.

Ali: If I can just pick up on that leaning into sort of that you've got various different answers coming out of it at different times how are you leaning into the human side of it almost giving a little bit of an education to from your side you know the marketing the BD team but then you start to think about the associates. Within the law firm how are you giving them that guidance of going right this  is where this is the outcome that you can get from it it's going to make the grunt work that bit easier but we need you to you know put your knowledge on that and you know sprinkle some brilliance around how are are you going about doing that with your teams? 

Monica: From my side, we have a playbook. The team uses this playbook. I have a lot of very junior people. And so they're very excited to use these tools. Our ChatGD is not as creative as ChatGPT is, you know, it is, you know, we have to stay within certain guardrails intentionally.So there are times when I need, it's just not going to come out with the most creative and exciting responses. And so I have to, my team can jazz it up. But one of the things that we will also play around with is like, hey, each one of you try a different prompt and see what kind of answer we're getting. Because the prompt engineering is really a thing. And it requires a comfort level with the questioning, with the way that we craft the individual question in order to get a response that's interesting. So we learned California cool. You know, California is our base. We learned California cool is a little too cool. And so that's not going to work. And so when I'm trying to figure out how do we define our tone so we don't have to tweak quite as much, we know we're always going to have to tweak. It's never going to come up with something that sounds like Gunderson, but it can give us something very, very close. And experimentation helps us to get that much closer. And so we'll run like we have these various little challenges that we'll run for the team where we'll ask them to try to... Did you know this? No. You didn't know about it? No. Here we go. Then we'll try these little challenges where we'll have them go head to head with the same kind of prompt. See whether or not we can get at something really interesting. Because we're always trying to get ChatGD to be more interesting than it wants to be.

Joe: You can always turn up your temperature.

Ali: Is there anything there with those that you've got nice examples to lean into around campaigns that you've been able to maybe do that from a marketing standpoint, or if you're thinking about from the client side, how you're demonstrating that use of, say, ChatGD or other sort of AI elements to the firm?

Joe: So in addition to our prolific marketing users, We rolled it out to all of the attorneys at our firm. And in terms of the way that we tried to frame it for them and how they were to use it and kind of use it in conjunction with their own experience and judgment and creativity, basically, we told them, this is a learning tool for you. We gave it to them very early before there were a lot of third-party tools that they had access to that they could use securely. But we told them more or less that they are the first human looking at any of these outputs. The machine doesn't actually know anything. It's just stringing words together. Those words may look great and then still be terribly wrong or flawed in some way, which when you're talking about legal advice and contractual interpretation and drafting contracts and things like that, much higher risk profile, you know, for being precise and right. And so, you know, one of the cardinal rules that we have for using this tool is that, you know, nobody is to use it on anything that they can't personally verify the accuracy and completeness of. Because that's the most dangerous thing about this technology. It's not that different from many things that we've all used, you know, kind of in our work lives, except that you can't see how it came up with what it came up with. And often it looks really good if you have no idea. It looks very convincing. It'll even argue with you to tell you that it's right when it's wrong. And so that was something that we really kind of harped on with our attorneys in particular. In terms of the ways that they're using it, we really tried to focus them on not using it for the things that we knew it had trouble performing with. A general question answering machine. Lots of people want to use this technology for that. Just ask it questions and hope to get answers. Use it as a Google replacement. It's pretty bad at that. I mean, sometimes it works, but it's just not as reliable and you can't verify the information in the way that you can with a Google search, which we have more muscle memory doing as a society. So we really encourage people to use it more for language manipulation tasks. Take this chunk of legalese and turn it into some easy to read bullets that I can send to a client that explains it. It's not going to be perfect, but it gives me a starting place. Some other things that would, you know, lawyers have to do all the time, but that, you know, are kind of time consuming. You know, one example that we came up with that's different than just doing kind of a find and replace is taking like some resolutions, board resolutions for a client that were all gendered. He, him, chair, chairman, you know, that sort of thing, and asking to make it gender neutral. Now, normally, you would have had to go through and search for each of those terms individually if it was a long document, and you might have missed chairman, for example, not realizing it was in the document. It caught all that stuff. It did a really good job of that. So relatively simple tasks that are kind of, you know, scut work that nobody enjoys doing, it can be really great at that. And that it turns out is, you know, a large percentage of what we all do every day. 

Ali: And are you finding that the clients are really receptive to knowing that you're doing it? Because we were discussing this yesterday that often I'm sure people in the room are finding this as well. The clients want you to be using technology. They want you to be using something like AI, but they're like, we don't want you to test it on us. And you're obviously at the forefront of this and you are doing it. So they've been responsive to that. They like the fact that you have this internal system and that clearly some of the work, I suppose, is produced and put in front of them is coming out of technology and AI. 

Joe: Yeah, we're very fortunate in this regard, just given our client base and our business. I mean, most of our clients bring us their most sensitive company documents in a free Google Drive account that Google is mining for ads. So, you know, they and they build SaaS software like that's what they do. They build stuff in the cloud. Like we just don't have a lot of the same challenges that firms who represent JP Morgan and big insurance companies have to deal with. And so for us, that has not been an issue. In fact, our clients generally are excited. I've talked with, I can't count how many of our company clients who are building in the space and are interested to hear about what we did and how we went about it. We're building on the same tech stack that they are in many cases. And it's been really good from that perspective. I've been talking with a ton of our venture capital clients as well, just about the legal industry and the things that we're seeing and what the technology can do today. So it's been great from that regard. In terms of the kind of, for many of your firms, it's hilarious. I mean, I go to so many of these meetings with people in my kind of, you know, position.  And, you know, it's like the same client will be, you know, here's my RFP that says, tell me all the ways that you're using AI to lower my bills and be more efficient. And here's my outside counsel guideline saying you can't use AI at all. No AI, not generative AI. This is where that distinction comes up. You got to go back to me like, okay, so we won't use Westlaw because that has AI and we won't use Microsoft Word because that has AI now too, right? I mean, like it's getting a little bit ridiculous out there. You know, I think at the end of the day, this is all a transitory period where people just need to kind of understand and get comfortable. I mean, Copilot's going to be in Microsoft Word for all of our firms in a very short amount of time and for all of our clients. Everybody's going to realize that, you know, there are things that they may not want us to do with AI that may be understandable. You know, they may not want us to train our own models, which none of you are doing. I'm fairly confident, you know, with your client's data, where it might get spat back out in some way that, you know, is kind of unintended. But the vast majority of the things that we're actually doing with AI, clients shouldn't be concerned about. It's for their benefit. It's just going to take a little time and a little bit of education. Often the people who are submitting those outside counsel guidelines don't understand it themselves and aren't really in a position to change it in a massive institution.

Ali: Yeah. Just building on that, it'd be really interesting to take the conversation in a slightly different direction. One of the concerns for so many people around AI, if you look internally, is that it's going to dumb down the voice of the firms. Everyone's going to become a little bit more sort of the same and sort of maybe slightly antiquated around that. What are the sort of risks that you see? Do you think this is something that is genuinely a concern or do you think it could be used in other ways to ensure that you're moving forward to something slightly different?

Monica: I absolutely think that if every law firm here used ChatGPT to write their website, it would sound exactly the same. It can only do so many different things. It can only present so many different kinds of perspectives. And yes, California cool and pirate sound very, very different. I totally agree with that. And so that's where that human piece comes in. So if we imagine that the technology can help us get the table stakes down, then our teams can focus their attention on the piece that makes us different, the piece that makes us special, that little bit of difference that allows us all to be in a room together talking about how we all are facing the same challenges, but we all have different people in our organizations. We're trying to lead transformation, but each one of us has a different combination of personalities and perspectives that we need to navigate. And that's something that ChatGPT, Copilot, Gemini, none of them can replicate—that intuition that we have as human beings, as people with emotional intelligence, with social intelligence. Forget even the ability to then drive real change in an organization because it needs people. The systems can tell you the what and how very often, and even sometimes the why, but it can't really get it all done because it needs that extra piece. And so if my team is using these tools to help get us from zero to 60, from 60 to 100, that takes us. That takes their creativity, their knowledge, their judgment, and their understanding of all of these humans that they happen to be working with who have these just different modes of being in the world. And our client base is going to be different too. And so that's where I think we can then spend the majority of our time being different instead of the majority of our time trying to take care of table stakes. And so I think, yes, if we were to rely entirely on the box to run our organizations, yeah, we'll all be the same. But then there's such a loss because there's value in being different. And now through the use of these technologies, we can spend more of our resources, energy, time, et cetera, in that place where we are actually differentiated and can communicate a different value proposition to our client base.

Ali: So how are you overcoming some of those challenges in the risk? Because the risk can be perceived in multiple different ways. It might be the dumbing down of attorneys. It might be other examples as well. How are you kind of getting around overcoming that from your perspective? I'm sure you've faced a few different challenges.

Monica: I think we think about this all the time. 

Joe: Yeah. How will we have partners in our law firms if we take away all the work that young lawyers, like I once upon a time, cut my teeth on doing mindlessly in many cases? I hate this conversation that 's happening so much. It also kind of feeds into the whole back-to-work thing. You know, everybody has to be together. It was the only way to be a lawyer. I think it suffers from a lack of creativity. There's a whole bunch of things that lawyers used to do that they don't do anymore, that when they stopped doing it, senior lawyers said, "There's no way they're going to be able to be a lawyer." Redlining, blacklining, we all know, that used to be done by hand from a Harvard-trained young lawyer sitting there with a pen and crossing out the words that were changed from one draft to another. And as soon as the software that does that came out, everybody said, "Well, they're not going to learn how to draft contracts. There's no way." Before generative AI, when I started building document automation tools and making it so you can fill out a web questionnaire and not have to put the company's name in 15 different places in the documents, and it just, you put it once and it ends up all throughout the documents, how are they going to learn? They're not going to be in the contracts. This is the same thing. However, the more that you suck up of the work that, you know, actually was detail-oriented in kind of getting people, you know, to become more technicians, you know, as lawyers, the more intentional you have to be about how you teach those skills if they're not going to do it just through brute force repetition. And I think this is true for all knowledge workers. There are aspects of all of our jobs that have changed in the past. Some of those were useless. They didn't really teach us anything. They were just necessary evils that had to be done. Good riddance. They're gone. Do a dance. But there's plenty of things that do kind of seep in from doing that kind of work. And so how do we teach people to have the skills that they actually need to do the jobs we want them doing? I think that's the big question. And that requires investment and thought and revamping the way that we train our attorneys, our business professionals kind of across the board. But I think it's eminently doable. And I think everybody's lives will be more enjoyable.

Ali: Well, I think one of the threads between what both of you just said as well was that, Monica, you said AI could take you from 0 to 60, and then at 60 to 100 is that human, you know. Joe, yesterday, you should quote, made by a human will be the thing coming into the future. 

Monica: I bought that domain. 

Ali: Did you? 

Monica: This morning. Well, there we go. Made by humans. Because I think that's going to be something that will actually have value. It was only $43. Not a huge investment on GoDaddy, but-

Ali: But the point of that is that that sort of 60 to 100 is where all of a sudden in both realms, you can start to be very strategic, actually start to think about what's mattering, that grunt work's done for you. And all of a sudden, it's actually adding real value to the work that you're doing.

Monica: It's funny because I was actually thinking about, it can also work the other way around. I have some pretty junior people who prior to arriving at Gunderson had never written a professional email before. And they, so they also came on, right? When we launched ChatGD, I would say, look, we want to write persuasively because we need the attorneys to respond back and they need to use bullet. Let's, let's make it simplified. Why don't you ask ChatGD to write you this email? And so they would write an email and then they'd give it to ChatGD and say, ChatGD, can you make this more persuasive? ChatGD would give them back an email. And then they would ask, why did you make the changes you made? And then it would give some explanation. And very recently, and they write beautiful emails. And so very recently, I was asking one of my team members, wow, did you have ChatGD write that email for you? She's like, no, no, I wrote that myself. Made by humans. Hashtag. But what was amazing was that she was learning from ChatGD how to be a better writer because she was seeing good writing. This is what good writing looks like.

And that was the ability to then use these tools to help these folks who did not learn it the long way. I often use the example of like while we might have calculators, it's still valuable to learn math right because you understand the mechanics behind how these things happen. She didn't need to learn how to write a persuasive business email before she could learn from the tool itself. And that's actually opening up all kinds of ideas for me about how to train my team going forward so that they can actually augment the human learning from their supervisors with the tool to be able to be that much more effective and really go from being, you know, I'm two months out of college to being writing as well or communicating as well as someone who's been out of school for a couple of years.

Ali: Yeah, it's a really nice example. It's a big round of applause and thank you ever so much for being part of this, thanks for listening.

Charlie: Thanks for listening to this special edition of the CMO Series podcast. You can subscribe to the series via your favorite podcast platform and if you'd like to find out how Passle makes thought leadership simple, scalable and effective visit passle.net. That's all for this week. We'll see you next time. Thanks for listening.