00:00:00: You are listening to the Effective Statician podcast.

00:00:04: The weekly podcast with Alexander Schacht and Benjamin Piske designed to help you reach a potential, lead great science & serve patients while having a great work-life balance!

00:00:22: In addition our premium courses on the Effectives Staticians Academy.

00:00:28: we also have lots of free resources for you across all kind of different topics within that academy.

00:00:38: Head over to theeffectivestatistician.com and find The Academy, And much more For You To Become An Effective Statistician.

00:00:48: I'm producing this podcast in association with PSI A community dedicated to leading and promoting user statistics within the healthcare industry for the benefit of patients.

00:01:01: Join PSI today to further develop your statistical capabilities with access the ever-growing video on demand content library, free registration for all PSI webinars and much more.

00:01:13: Head over to the PSI website at psiweb.org To learn more about PSI activities.

00:01:20: become a PSI number to pay!

00:01:30: Welcome another episode of The Effective Statistician Today.

00:01:34: I'm super happy because two reasons First, I talk to very well-known statisticians Chris and second we talked about a very interesting topic AI.

00:01:48: And the topic of today is around paper.

00:01:54: Chris wrote what's future role of statistician will be in an AI world?

00:02:00: Do they actually need statisticians as their future all become redundant by solutions?

00:02:07: So before we dive into the discussion, Chris maybe you can introduce yourself a little bit.

00:02:13: Yes certainly thanks for the invitation to be here Alexander!

00:02:17: Yeah so I'm Chris Harbrun now that is an independent statistical consultant.

00:02:23: Before that i've been in the pharmaceutical industry for many years working through variety of roles and varieties companies most recently within the methodology group at Roche I guess within the AI world, by backstory of this.

00:02:36: Probably around about twenty years ago was one of the cycles round the omics technologies which people started thinking big data on how that could be used and techniques like machine learning were starting to be thought out in a situation And i got involved with that as it developed into thinking ideas around personalised health care.

00:02:57: yeah i've gotten involved those type things as well as biomarkers more generally.

00:03:01: And then, of course over the past few years there's been this explosion of interest in AI.

00:03:05: So it again is natural extension that to get involved and be involved with that type of activity?

00:03:12: Maybe I could start with some kind of provocative thing a protocol an statistical analysis plan The programs the report Then maybe parts of the manuscripts and abstracts.

00:03:29: I can't all do that with AI.

00:03:32: If i would be kind of a little bit worst physician, yeah couldn't or maybe just medical writer and assistant?

00:03:40: Or even some kind of AI agent could set the solution to everything around clinical study.

00:03:51: well Potentially it could in substances, so that I think the technologies today particularly when we look at a genetic AI which is still most recent development.

00:04:01: We should last for specialization.

00:04:03: So thankfully you can think of an agent within an AIS best as being a large language model like chat GPT Which has been specifically trained on one area of expertise whether it be amokia heading modeling or biomarkers whatever.

00:04:16: and If were to do that You could properly generate all of the documents that you've mentioned there And initially they may look quite plausible.

00:04:26: Yeah, it might have looked good but then if we started to read them I think that would start getting a little bit worried about things.

00:04:33: in normal case We've all had the experience when using the large language model where suddenly it traps out something with huge confidence which is clearly not true what we call hallucination and so you cannot be sure of your confidence.

00:04:49: I think there's also another subtle layer below that, is actually if we think back in the development of all these kinds of documents at any team i've ever been it has never been a straightforward linear process.

00:05:01: There are always huge amounts of circulation going around and I might write this statistics section based on what I understand they protocol to be about.

00:05:10: It goes through their physician.

00:05:12: They have got completely different interpretation from And it's only by going to that level of detail, we're actually able to come up with a resolution.

00:05:20: What do you want the study to be doing?

00:05:22: what Do We Want To Be Achieving in this Study?

00:05:24: If we adjust our source data core start to large language model or an AI system then were gonna lose.

00:05:30: that kind of discussion and understanding is just where ever the AI assumes we wanted to do which probably won't have agreement for everybody.

00:05:41: So In a sense these documents are starting with.

00:05:44: protocol is not just a document, it's a tool to help facilitate discussions so-to say.

00:05:55: Yes yeah and I think that the whole process of that um you know especially the way it forces a lot of rigor and real precision about what can be called at that was really critical in designing good study And I think the adoption of the S demands framework is a great example.

00:06:13: It doesn't matter for any wiggle room or misunderstandings, because we're being really precise what were going to do within that particular study.

00:06:20: Another point also... Any innovation will be very difficult to assume from an AI partner isn't

00:06:30: it?

00:06:30: Yes!

00:06:32: What an AI model does is just reflects the data its been trained on.

00:06:36: That's where they are good at doing So.

00:06:39: yeah, their starting position for any protocol will be all the protocols would have been written previously.

00:06:45: Now in some ways that's not different.

00:06:46: maybe we'd write it.

00:06:47: if you asked me to write a protocol and new area first thing I go do was dig out of previous one take parts.

00:06:54: Yeah, it's absolutely right.

00:06:56: Without a lot of additional input or work which probably has to come from the human you're not going to get an innovation or some different type with design towards what's gone beforehand because they are just trying to replicate their training data.

00:07:08: Another point that I have seen in designing clinical trials You always need to make compromises yeah?

00:07:16: You can't have everything!

00:07:18: You don't have all the money in the world, you're going to have also patients as well.

00:07:25: Yeah?

00:07:25: So it's always real life conditions that you need to have and under these conditions of course you need find some suitable balances.

00:07:37: how would I work with that?

00:07:40: so if for example say two AI want a half study set is not bigger than so many patients And it doesn't run longer than three years, let's say.

00:07:52: It doesn't cost more then ten million whatsoever?

00:07:56: Yeah I mean so again.

00:07:58: potentially you could do that if we could specify your limitations about things.

00:08:02: i've got a budget of five hundred million and this amount time.

00:08:06: what to maximize?

00:08:08: the?

00:08:08: yeah but the objective is to minimize in the time to market.

00:08:13: yes This PTS value and so on, it could probably come up with some compromise which would meet those criteria.

00:08:23: Now there's two challenges that one is really being explicitly able to state those criteria Which again this typically something you do lots of iterative discussions about things normally And the other is will it come to the trade off?

00:08:37: The balance though we want have.

00:08:40: It may be that it's huge.

00:08:42: That things but actually completely cuts out all of the biomarker assessments because they cost lots of money and not relevant.

00:08:48: Yeah, I did see them as being relevant.

00:08:51: The bio rockers scientists on the other hand is gonna get very annoyed about that.

00:08:54: And yeah There's a lot value.

00:08:55: which can we achieve from those?

00:08:57: so the balance?

00:08:58: Which it will achieve?

00:08:59: he'll try to do something sensible cuz yep As if feels it doesn't necessarily mean what you would want to be achieving or so optimal compromise.

00:09:09: I think that is also coming back to the discussion.

00:09:14: In team meeting you can then come up with all kind of different solutions, especially through these discussions... ...you better clarify what actually goal we want to reach here because very often it's not super clear and there are problems AI can't solve.

00:09:34: I think the other point is looking into a bigger picture.

00:09:40: For example, we might think about... We've just focused on the phase one study and all of our discussions are about the phase-one study.

00:09:48: that could be actually optimising if we also include aspects from the Phase One B Study into it.

00:09:58: or what kind of implications would this have for the Phase Two Study?

00:10:01: Can we optimise whole thing a little bit differently so that we get better through the phase two study and maybe safe in the Phase Two Study.

00:10:11: There's so many different aspects, I think it is really difficult to just get out of AI.

00:10:18: Yeah, and I think there's also something about just the experience which anybody working in your development has accumulated over time from seeing things experiencing.

00:10:27: Things maybe not going quite as they wanted them so on Which is again very hard to capture an any way input into thinks.

00:10:34: but if you're in a meeting You discuss some of things And you've got someone who's got that experience.

00:10:38: it really shines through what they can quickly identify where I could think it's going on in the future and other things.

00:10:44: And this speaks to something which you always found really interesting, that i don't see the future as being a completely AI world but AI will be part of it!

00:10:53: The real interesting challenge is how we get AI and humans to work better?

00:10:58: I've thought it was unfortunate actually that AI has got an aim at AI with intelligence.

00:11:03: because yeah... Intelligence is something humans have But its different type intelligence for want to be a better word, what we get from an AI.

00:11:13: And in some ways it'd be better if different words because people automatically try and think that there's been the same type of things.

00:11:20: but I think If We Then Thought Of AI as being something which is complementary capability... ...and how we work together at Complementary Capability instead of being competition used most intelligent then they give us other ways or best way working together?

00:11:35: I kind of think of AI very often as a librarian that has access to a huge library of all different things and can quickly pull out all kinds of answers from this library.

00:11:49: The problem, however is your answer will very much depend on the quality of questions.

00:11:57: I think it's another area where you need to ask the right question in order to do so And then understand whether what we get back is sensible or some complete BS.

00:12:12: And we know that there's a lot of BS actually out there, I just recently stepped over post talked about kissing skills and how that relates to the months of births of you... ...and said they were six specific months set for better kisser!

00:12:29: The other six are not as good in terms of kissing.

00:12:32: i was thinking like my gut feeling tells me this is complete rubbish.

00:12:38: Yes,

00:12:39: yeah.

00:12:41: Now absolutely and I think one of the clear takeaways at moment is that if we are using AI to be generating anything or any importance then having a really thorough review of it is absolutely critical.

00:12:55: keeping an act human in the loop as part of that review process and checking what's provided is absolutely important.

00:13:02: this is something and as we think more broadly about say medical applications you can use with hospitals Again, fantastic and useful but they realise that have to include those very rigorous review processes.

00:13:14: And now this self-directs other problems because one of the features about I think humans which may be less vulnerable than AI is a human's are fundamentally lazy... ...and actually if we've been using AI on its generated three outputs.

00:13:27: so we're happy with those.

00:13:29: often for me looked at four fifth in six.

00:13:31: you're not gonna look at it so rigorously.. ..and how do we keep up the quality?

00:13:37: That's all one of the big concerns I have moving forwards.

00:13:40: I think that speaks to the trust... ...that we have into the AI output, why should we trust a statistician more than what they get from an AI output?

00:13:58: That is really good question!

00:14:01: No one understands.

00:14:02: this is on the AI.

00:14:03: There are a whole range of risks which were involved with things and the outputs we hope from the statistician, I think that sort there's a role all of us have got to keep on developing pushing in there That there is an added layer understanding within that Which you don't get it form the AI.

00:14:19: We've got additional confidence which can provide people The understanding why reasons behind thing to be able to interact with them and understand their understanding is better.

00:14:31: And that can come at times, sometimes it may actually case there's a challenging which one of the old statisticians could provide Which AI we know congenally often doesn't like doing.

00:14:42: It can very agreeable in time But I think its really good underline question There always this challenge even pre-AI world about how we produce trust & confidence in people.

00:14:53: How do you build that?

00:14:55: I think for me if i think about trust, usually think of three different things.

00:15:01: The first is competence the second is care and third is character.

00:15:07: so competence well maybe AI can become pretty good at some point but AI doesn't really care.

00:15:16: it doesn't care whether you succeed on or not succeed yeah?

00:15:21: It's a machine.

00:15:23: Yes, yep.

00:15:24: And where it will not for example just generally tell you something that you don't want to hear like this is not feasible which nobody wants to hear that.

00:15:43: but of course there's some things people need here so they can come up with something that is feasible and meets the demands.

00:15:52: When you also something fails, who's accountable?

00:15:56: Yeah.

00:15:56: That is absolutely one of the unanswered questions at the moment and this has potentially some very big in a wider world.

00:16:05: so I'm very big question because it comes litigation legal standings and admins probably want to be blockers in adoption of AI in lotto critical systems because until he established case law to understand who will have to be responsible, nobody wants to take that risk.

00:16:22: Now certainly in an internal case I think inside a company companies are still very much structured on human reporting lines and so i think there's a very clear accountability lies within the human reporting line.

00:16:35: yeah i think it's fair say cobbly in every company that the person here is accountable.

00:16:40: yes he was a person as in a way we've always been accountable for the work, the outputs that we produce.

00:16:48: Yeah I have to be confident and get it right.

00:16:50: if they're produced by AI then i think it's still very much going through that same line of person who is the user of the AI has to ensure that their comfortable

00:16:59: with that.

00:16:59: yeah take accountability, take responsibility for it.

00:17:04: so if you get rid off your whole statistics department can't blame the statisticians anymore.

00:17:13: So let's talk a little bit about recommendations for the future.

00:17:17: Yeah?

00:17:18: For, lets say typical statisticians that work on clinical trials, reward evidence any kind of data projects pretty much hands-on.

00:17:27: what would you recommend to them?

00:17:30: learn and change or maybe think differently?

00:17:35: I think underlying this will be an idea of curiosity which is something we've always been encouraging in any sense.

00:17:42: Yeah, there's this world out there which is AI.

00:17:45: Just for people to explore it and read about it listen to information about it.

00:17:50: There's a huge wealth of information and stories by AI so there are huge opportunities to be learning about that And to experiment with it in any setting like that.

00:18:01: So I think the basic bottom line within that It certainly going to be a case one form or another is going to be with us moving forward.

00:18:10: So I don't think it's something we've got the opportunity to stick our heads in the sand that pretend it's not happening and over time, i'm sure in the same way as a Google search was quite revolutionary when they first came out four years ago... email!

00:18:25: Think of what change has made for life.

00:18:28: there will be parts of AI which get just adopted into everyday lifestyle every day being as humans, which we don't even think about in the future.

00:18:38: And you want to make sure your part of that and able to do it otherwise would be operating at a complete disadvantage.

00:18:44: completely agree I tried to test AI for all kind of different tasks said I'm doing so especially tasks where need to generate text or documents things like this.

00:18:59: I really like doing it.

00:19:00: Of course, read things and what i also liked is to basically ask CAI What additional information do they need?

00:19:09: And I think that was one of the best hacks whatsoever.

00:19:13: Yeah says that you can provide better prompts and then refinance and refinement and refine It.

00:19:20: That's also how I've written my book.

00:19:23: But if yours in also have testings out You could become more confident over time And what to do or use it and set can really accelerate things.

00:19:34: Yeah, I think that's a great thing of the experimenting.

00:19:37: there is an activity which you're doing Which is basically a bit dull feels a bit repetitive.

00:19:43: That is probably why AII can help You within that.

00:19:46: but again the experience Of Doing Things and i'm not sure if your experience in writing But certainly when I've used It?

00:19:53: I've always felt that Probably Half of the comments it makes, I think are really valuable.

00:19:57: They're things where they just structure the sentence better than i had and you say that as good feedback in future But probably about half their cases.

00:20:05: actually It makes a suggestion but it's not what I wanted.

00:20:09: I've written in that way because I had reason for writing in that Way And I think that understanding is going to be very important.

00:20:16: forgetting there

00:20:17: also used for complimenting me yeah.

00:20:22: so For example email.

00:20:26: that makes you very angry, yeah?

00:20:28: And we all get that sooner or later in our work.

00:20:32: In the past I would have kind of drafted something, slept over it and then refined it thought about do i need to send it at all... All these kinds of different things.

00:20:43: but now I can also use AI to write an answer with a very different mood than I have.

00:20:55: So, i might be really frustrated angry upset that they can tell AI to write it in a very understanding calm voice?

00:21:06: Very solution-oriented!

00:21:09: And which...I can't at the moment but know what you want and why not.

00:21:15: That's another kind of good thing.

00:21:18: Yeah, and no I think that's a great example.

00:21:21: But yeah i do think within that we've got to be careful about the authenticity of that.

00:21:25: Occasionally you get something which is clearly written by AI And you've lost that personal touch.

00:21:31: It's doing something like that but just not taking verbatim what's written by the AIs would be very helpful approach there.

00:21:38: Completely agree...I don't use AI for my social media actually anymore because some social media the great AI written posts, which is in a way kind of weird because all these different social media have built into it and suggest you to use that.

00:22:02: Anything else that he would like to listen or take away from this podcast episode?

00:22:07: I think there's.

00:22:08: the overwhelming position was actually Things will change with AI, but things have always been changing over time in any career and I think statistics is no exception within that.

00:22:21: But actually sort of if i tried to be positive about this why would hopefully going forward it's a maybe some other less interesting the less rewarding parts our job we can actually reduce when I've gone completely eliminate them.

00:22:33: AI will provide really good tools for doing that ,but absolutely at the same time opens up a range of opportunities for other things we're gonna do during few less need for statisticians, in fact I think there's going to be more needs.

00:22:47: We're in the world that is gonna be data new endpoints which come from AI technologies like an imaging or wearable devices other analyses which may become feasible because we've got a lot of data we can now use with AI sort of getting additional insights too.

00:23:04: so actually see our very promising future and enlightening future then And probably many careers, I think actually statistics is one of the best places to really write and thrive within the AI world.

00:23:17: Honestly we work in a very innovative area so there will be always new things that we are working on that AI hasn't worked before.

00:23:28: where need to think differently about... ...and lastly we talk a lot about human aspects which never go away.

00:23:35: No, absolutely.

00:23:36: Yeah and I think those are the skills which become more important to more useful in effort.

00:23:41: so yeah... Absolutely!

00:23:43: Thanks a much Chris for this interesting discussion about AI as we're recording his PSI conference And there's also of course a lot about AI when that gets published.

00:23:55: this i-conference is already going but this topic will definitely not go away.

00:24:00: That?

00:24:00: Absolutely!

00:24:01: Yeah thanks very much.

00:24:07: This show was created in association with PSI.

00:24:10: Thanks to Rain and her team at VVS, well-positioned background and thank you for listening!

00:24:17: Reach your potential, lead great science and serve patients – just be an effective statistician.