00:00:00: You are listening to the Effective Statician podcast.

00:00:04: The weekly podcast with Alexander Schacht and Benjamin Piske designed to help you reach a potential, lead great science & serve patients while having a great work-life balance!

00:00:22: In addition our premium courses on the Effectives Staticians Academy.

00:00:28: we also have lots

00:00:30: of

00:00:31: free resources for you across all kind of different topics within Z-Academy.

00:00:38: Head over to theeffectivestatistician.com and find The Academy, And much more For You To Become An Effective Statistician.

00:00:48: I'm producing this podcast in association with PSI A community dedicated to leading and promoting user statistics within the healthcare industry for the benefit of

00:00:59: patients.

00:01:01: Join PSI today to further develop your statistical capabilities with access to the ever-growing video on demand content library, free registration for all PSI webinars and much more.

00:01:13: Head over to the PSI website at psiweb.org To learn more about PSI activities.

00:01:20: become a PSI member to pay.

00:01:30: Welcome to another episode of The Effective Statistician And Today I'm so happy to have Gorana Kapkun from Merck on the line.

00:01:39: She is he head of the Rebuild Evidence Department at Merck, so German Merck and we will talk about a very interesting topic which I think all statisticians should have close eye because it's very close to data that is Integrated Evidence Planning Plans.

00:02:02: And so Gurana welcome to the podcast episode.

00:02:06: Thank you, Alexander and hello to everyone who's listening.

00:02:10: Okay maybe start with an introduction of yourself.

00:02:13: in the pre-chat we found that From

00:02:23: my studies, yeah.

00:02:24: So I left these pure theoretical mathematics to touch on computer science and then discovered statistics which fascinated me because they tried to describe the world by using measurements.

00:02:38: And i was first working in different actually areas outside of healthcare when I discovered epidemiology.

00:02:45: So after my PhD in applied statistics, I moved into the field of epidemiology and was really fascinated about data that are already collected or that we're collecting to understand what is happening in clinical practice.

00:02:59: And try to get some meaning out of depth with guidance as well especially when questions are geographically very diverse and wide.

00:03:09: And from there, so I spent some time in academia.

00:03:12: I spent sometime in public health.

00:03:14: So i worked at the Swiss federal office for Public Health when a really great invitation From Novartis to join their biostatistics department came and Novartes was A fantastic place To be because it Was like The best ever school For discovering all different facets and aspects of pharmaceutical industry That opened quantitative sciences different aspects of decision making, which is one of my passions.

00:03:42: So after being in bio-stats and health economics outcomes research market access real world evidence medical affairs.

00:03:51: an integrated approach to evidence generation attracted me to Merck healthcare where I'm now leading patient focused and real word evidence team.

00:04:00: so our team already integrates the classical more classical Real World Evidence with Patient Focused Evidence which is patient preferences, patients' appreciation of symptoms diseases treatments benefit risks and the creation of endpoints that a patient derived.

00:04:17: So I think this is little bit about my journey so far.

00:04:21: Very good.

00:04:22: Yeah, so you really combine all the different knowledge spaces basically be beyond clinical trials.

00:04:29: So if you can tell in a couple of words what actually is an integrated evidence plan and what does integrator about it?

00:04:37: I can tell your one definition but I can also tell you that since ten years or more, this term existed and was invented by pharma companies to respond the request for totality of evidence.

00:04:51: By our external stakeholders.

00:04:54: it has been implemented in so many different ways But my definition is a cross-functional strategic roadmap that coordinates all evidence generation activities across the product lifecycle, from very early development through commercialization and life cycle management.

00:05:15: And then you ask me about what it integrates?

00:05:18: This will depend on your organization.

00:05:20: so ideally it will integrate or bring together all functions as part of an organisation.

00:05:29: It will integrate needs for all geographies.

00:05:32: It will integrate needs of all stakeholders to whom internal or external, to whom we want share this evidence with.

00:05:40: And it will integrate all methods And this is the beautiful part.

00:05:45: So now clinical trials are not anymore alone, they're surrounded and swimming together with their friends of real-world evidence patient focused evidence modeling simulations machine learning AI everything is integrated into that plan.

00:06:03: so This how I see it

00:06:06: you mentioned kind off totality of evidence is needed by the different stakeholders.

00:06:12: Let's first start with external stakeholders, what kind of external stakeholders do we talk about for this integrated evidence

00:06:18: plan?

00:06:19: Our classical suspects in pharma industry once upon a time were all aiming towards satisfying regulatory needs and said okay if you get market authorization that it.

00:06:32: The situation got more and more complicated, and more interesting for us.

00:06:36: And if we do have market authorization... ...we don't necessarily get access to it.

00:06:41: so we need to satisfy needs of payers in the HDA bodies.... ....to give us reimbursement and to give access.

00:06:48: but nowadays becomes even more complex.

00:06:51: If you really want what we are all here for in pharma industry.. ..and reach a patient itself.

00:06:57: somebody needs to prescribe that medicine.

00:07:00: So we need to reach the hearts and minds of our prescribers, of our physicians.

00:07:05: They need to understand for which patient why and when do they need to prescribe?

00:07:10: And then lastly –and more presently now– We need to reached out patients with a clear narrative Why is that particular medication or combination of treatments Or whatever solution actually fit for purpose them?

00:07:27: And so these are all of our different stakeholders that we need to prepare our evidence and our story for.

00:07:34: Yeah, let's start a little bit about the life cycle how it actually develops.

00:07:39: when would you consider be good start for something like an integrated evidence planning?

00:07:45: So from me this starts at research on.

00:07:47: this is why I'm actually already preclinical in many companies may disagree with me at this stage.

00:07:54: a part of the Merck healthcare that took this strategically for faith and said, how would the world be if we start integrating at the very beginning?

00:08:06: At that very beginning.

00:08:07: We are distantly thinking about our external stakeholders later on And we're not outlining all of the plans in the exact sequence of execution.

00:08:17: with thinking of these questions surrounding us Is there even an unmet need?

00:08:23: What is the burden of disease?

00:08:24: what problem do we try to solve, Is that specific mode action actually most potent for what were trying achieve.

00:08:34: to the level of endpoints that we need to measure, how do we measure even future treatment success or burden on this disease?

00:08:43: So all these questions needs some kind integration with different experts and domains.

00:08:50: If you want design the best possible next experiment which can be a clinical trial it could have been a different nature.

00:08:58: And of course you need filter and select.

00:09:00: yeah... Which ones will kill ,which one move forward?

00:09:03: And if you see like, well that might look nice in the lab but there's no way we can translate into the clinic or they're high risk for certain side effects of whatever.

00:09:17: Yeah, indeed.

00:09:18: So this integration really goes beyond evidence itself.

00:09:22: it serves as a basis of cross-functional discussion serving the global product team that many pharma companies would have and this is like a mini enterprise guided by the Global Program Head or Lifecycle Lead depending on the company who's trying to see.

00:09:40: do we disease that we can really bring to patient faster and more patients.

00:09:48: Yeah, so I think Integrated Evidence Plan is really serving that purpose – serving patients in the end!

00:09:54: Yeah i think it provides clarity in terms of what needs to be done for everybody that's involved?

00:10:00: And also upper management... It's not just sitting on someone's head.

00:10:05: Yeah, that's things like okay I know all these kind of different things and i make the decisions but if i'm sick or if i leave then all the documentation is gone.

00:10:14: so to say

00:10:15: yeah.

00:10:16: No no in the ideal world it will give us all a whiteboard on which we draw and put together all of the ideas and draw the path forward And We All See The Same Picture and We Can All Comment On It.

00:10:30: Make Its Better Change Left Or Right?

00:10:34: And it's a living document.

00:10:36: Integrated evidence plan is a living documents.

00:10:38: every time something that could influence our thinking and decision-making happens external of the company or internally, Every readout... Every launch or read out if somebody else's phase one two three influences Our thinking and brings us back to that board asking this do you need change course?

00:10:59: Or are you good?

00:11:00: In a sense its also Decision Lock.

00:11:03: Yeah, so later on you can go back and say well we decided to cancel that because of x. I think that is very often not actually that easy to trace back.

00:11:16: if for example need to make a decision again Likely, if you're working the same indication over a longer period of time.

00:11:23: You will find similar challenges and then look back at why did we make that decision?

00:11:29: Why did it move forward?

00:11:30: What is now different decisions have been learned?

00:11:34: That's really interesting!

00:11:36: It is... I love this concept.

00:11:39: When do we move forwards into an early phase?

00:11:43: What kind of typical evidence questions arise now?

00:11:48: And how far would we look forward, to say?

00:11:52: I think once you find this good marriage between the compound and the indication then questions are How am i going to design my first experiment?

00:12:03: who are the patients that need To go in.

00:12:11: Can it be projected to the future success or even how this molecule will behave in the real world, and clinical practice?

00:12:19: So I think that's then a question we could imagine around that period of time.

00:12:26: The central point becomes your first experiment... ...and everything can help that experiment successful!

00:12:34: This is where the integration of different types of evidence happens.

00:12:40: speak about types of evidence.

00:12:42: Let's go a little bit through them, and you already mentioned some.

00:12:46: maybe I can summarize... Of course we have the clinical trials yeah?

00:12:50: And also not just see ones that we do for efficacy in safety but also one such video for PKPDs these kind of areas!

00:13:00: Of course from the experimental drug, but maybe other drugs.

00:13:08: Maybe something to show the burden of disease?

00:13:11: To better understand how patients are treated what standard care we have How long they're treated?

00:13:17: Comorbidities all these kind different things.

00:13:19: Then probably systematic literature reviews where we can look into kind of what is already out there and every kind of meta-analysis, network meta-analysts in their comparisons that we can do based on that.

00:13:35: And you mentioned more patient research like qualitative studies with patients where we have focus groups, things like that but also patient preference studies.

00:13:46: What other kind of data in broadest way I'm not just thinking in terms of numbers can be incorporated into our evidence planning?

00:13:56: I think there is methods, there are designs and they're also data types.

00:13:59: And sometimes we mix both.

00:14:01: but digital data from variables could bring us additional information – data from images or mixed data about specific biomarkers.

00:14:13: that now also LLMs giving access to some other type of data that are providing input into the overall knowledge and questions we're asking ourselves.

00:14:25: Data around outcomes, so patient-reported outcomes or clinician reported outcomes values or any kind of vital signs.

00:14:35: So basically anything that describes a patient, but I would even go and say the system through which patients needs to go though in order to have the most positive outcome is part and everything that is coming from preclinicals, so everything related to safety.

00:15:02: Toxicity initial signals of effect even if they are on types of biomarkers we're not going use in clinic.

00:15:09: I think all this part the story as a part our understanding.

00:15:14: Would you also think about aspects more for manufacturing pieces or would say it's completely different story?

00:15:24: It depends how far we want to integrate, but it is an important element.

00:15:29: as well as operational and feasibility aspects are on our paths to success.

00:15:36: But for the moment I'm more thinking of patient-level data or animal data or system data that telling us how to achieve the right patient at a right time receives the right medicine.

00:15:51: So at the moment I'm not thinking of operational and manufacturing aspects, but they are really important elements for success.

00:15:59: How about advisory boards?

00:16:02: So is it a payer advisory board or a community advisory board... ...or patient focus groups these kind of things?

00:16:09: how do they inform the integrated evidence plan?

00:16:13: They're important advisors and important data points that are guiding the path.

00:16:19: But if you allow me being a quantitative scientist, and I would have access to data like European Health Data Space or all integrated real-world data in the world.

00:16:31: I would somehow more trust this huge database that is filling with... That number of patients waits for that number time before being diagnosed then receiving treatment than put weight on opinion.

00:16:47: five fantastic key opinion leaders around the table because they probably do not have access to that vastness of data.

00:16:55: So I think it is complementary, but in a pyramid of evidence i would give it a little bit less weight.

00:17:02: and i can give you actually an example if you don't mind once upon a time in a neuromuscular space a rare disease and that disease was considered not shortening patients lives.

00:17:17: five out of thirteen world specialists in the domain.

00:17:21: People saw a patient aged seventy or seventy-three, so... The common opinion was yes there are patients who reach a certain age So there is no shortening of patient's life.

00:17:33: When we actually got into their databases We could derive the information that it shortens life.

00:17:42: It does not because with so few patients whoever reached a certain age that the whole disease is not impacting their lifespan.

00:17:51: So this perception, and this is rare disease.

00:17:53: so in there are lifetime of treating patients.

00:17:56: maybe they saw four patients?

00:17:58: This small sample asymptotics based on anecdotal relationships with these patient

00:18:04: Yeah!

00:18:05: And then you have to survival bias.

00:18:08: Absolutely

00:18:10: Yeah, I love that point.

00:18:11: In terms of these advisory boards completely agrees their complementary to other datasets.

00:18:19: they can inform your research where you want.

00:18:21: look for things like this and maybe sometimes there are faster to get some idea and cheaper than running a landscaping reward evidence practice and then contacting different data sources doing deep dive until study which might take you a year and lots of money.

00:18:42: but that's the very important point.

00:18:45: But they can also mislead you?

00:18:47: Yeah, it definitely misleads me here.

00:18:50: so I'm pretty sure people are clear on what is now happening in terms when to get into phase three and launch if we have all these needs.

00:18:59: from regulatory aspect and JCA is big topic with local HTA topics One of the biggest challenges that I see in that space is that you have this explosion in terms for stakeholders.

00:19:17: When you are on your research early team, yeah... You can probably name all those different people who work with it.

00:19:26: Yeah?

00:19:27: You could easily fit them into a usual office space but as soon as they go to launch moment The number of people explodes.

00:19:39: You have not just the global team, which has also significantly become bigger that you have all these local teams and functions like marketing medical affairs sales HTA market access becomes much much bigger.

00:19:58: So how does the integrated evidence plan helps there?

00:20:04: as you mentioned roadmap and structure to make sure everybody is on the same page.

00:20:10: I think this is where it becomes critical because it's only an anchor that all of these functions, people who are running around doing right things can come back too and then the more core people representing all of these internal stakeholders can decide on next turn or direction.

00:20:37: So this is where it really plays an important role, but when you mention all of those internal stakeholders their involvement needs to happen much earlier Before they start actually doing This.

00:20:50: planning is key especially If you think of evidence of different kinds that cannot be found already in secondary use of data or in the existing clinical trials, if something is missing and needs to be created prospectively.

00:21:07: That needs to blend up front!

00:21:10: If we need a differentiating endpoint like PRO-CRO or any kind of end point... ...that one has to be validated by all external bodies And this process takes several years.

00:21:23: So this is why starting early with right core representatives of all our internal functions, it's really key.

00:21:32: And when these endpoints need to be collected.

00:21:34: for example I just saw an outcome from a German HDA review and they had comparative studies.

00:21:42: that was great!

00:21:43: They have the right endpoint which also good as well as their population.

00:21:48: but there's comparative study reached only up to week sixteen.

00:21:54: And what the German HDA body requires is for chronic disease at least twenty-four weeks, so they missed these eight weeks and there someone either didn't ask or didn't listen yeah?

00:22:10: Didn't know what was the impact of that decision!

00:22:14: That has an impact not just Germany but all over Europe...and it's a disaster.

00:22:21: It reminds me how important a simple definition of an endpoint is and our understanding what end point is, what estimate it's.

00:22:30: What ingredients go into that?

00:22:33: And all components in these ingredients need to be clarified and aligned with the internal stakeholders who are our points-of-contact if you work from industry to external stakeholders bringing that knowledge back to the organization.

00:22:50: And then we make that decision, so choice of an endpoint as sound simple but it isn't at all or choice of a design like do you know how many years patients in Western countries are sometimes waiting for access because We ran a single-hand trial That is accepted by regulators?

00:23:10: It's up to five years.

00:23:12: Five years of time five years of no axis to treatment just because we did not align with the design, with HTA bodies at the same time.

00:23:22: We did it with our regulators.

00:23:24: so this is something that integrated evidence planning is aiming.

00:23:39: once worked on a launch of new drug, and it was a pediatric indication.

00:23:45: And the people who said we need functioning data.

00:23:49: then we got to her yeah We Need Functioning

00:23:51: Data.".

00:23:52: I asked what does that even mean?

00:23:57: Yeah i'm with you!

00:23:59: In comparison at time point What do you want to show us?

00:24:04: Yeah, but it's so easy and big organizations that these kind of buzzwords come up or this broad needs.

00:24:12: I once had we need more real-world evidence data And i was thinking like

00:24:17: what Yeah, I'm so with you and i think at Merck we really have a fantastic team of these clinical measurement scientists And biostatisticians are part of it as well As our team.

00:24:30: As well as quantitative pharmacology precision medicine Our biomarker specialist or data science Specialists in I Think this kind of teams especially when they're integrated together Are really guardians?

00:24:46: Of precision and guardians of scientific rigor, and methodological excellence.

00:24:52: So we are those people who will ask these questions what do you actually mean?

00:24:58: What is the question that we're trying to solve?

00:25:01: And what ingredients need to be there so we can translate it into a design for all quantitative scientists.

00:25:08: We should be proud our knowledge but also courageous because maybe They do not live in the same world.

00:25:18: And maybe we are then helping them articulate exactly what needs to be done, you know for us to be heard by external stakeholders?

00:25:27: Sometimes I ask people and What is the claim or conclusion that they want have after all this exercise?

00:25:35: We create some kind of marketing leaflet.

00:25:38: Yeah!

00:25:38: But it's the end goal.

00:25:39: but there's a story.

00:25:41: Yes...what does the story?

00:25:43: why is this different than anything else?

00:25:46: I mean, we come back to something we discussed before.

00:25:51: If you need explain it your grandma why this medication is better than something else?

00:25:56: This what we are aiming for because if the grandmother or a five-year old nephew understands the story then everybody will.

00:26:04: It leads very good to next and last phase probably post launch phase where one wants grow more patience.

00:26:15: How does the evidence plan there change over time?

00:26:20: It evolves.

00:26:22: It starts including collecting data, I mean delivering data that are needed for post-marketing commitments.

00:26:30: but it evolves from only delivering what is needed to creating more of a story because our clinical trials are just beginning patients on a specific drug.

00:26:45: The real access and the real numbers of patients will happen in real world, in clinical practice where patient's outside-of-inclusion exclusion criteria will be exposed with these differences between patients can actually influence or where the clinical practice adherence, persistence tolerability switching previous treatments anything can actually influence The behavior of that drug and impact on patient.

00:27:14: So this is what we are then trying to untangle On top of a classical safety.

00:27:19: We cannot tell much about safety especially for rare events or rare important event series adverse events with small collections in clinical trials.

00:27:30: It's really interesting said When the rubber hits a road, so to say.

00:27:35: You get into real patients.

00:27:36: what kind of all different topics come up?

00:27:40: Or maybe Topics gets very different weight as you saw before.

00:27:48: I once worked on drugs that had side effects in terms.

00:27:52: it created pain at the injection site And from a regulatory point, all the scientists said that's non-issue for this disease area.

00:28:03: But when it actually came to the market... The patients did care!

00:28:08: So there was demand on.

00:28:11: research shows how you can manage that?

00:28:16: How do you inject?

00:28:17: what kind of temperature use where you inject or decrease your pain?

00:28:23: and so I wouldn't say ignored, but it wasn't sort of being something that was interesting or important.

00:28:31: And then only when it comes to real patients and more patients were completely made up their

00:28:37: mind.".

00:28:38: So this is an interesting one—I once involved in a drug that was an infusion —and it was taking twenty minutes to infuse the treatment... ...and then twenty minutes preparation and twenty minutes cleaning devices?

00:28:54: The company then developed a five-minute inhalator instead, and no one cared except for patients.

00:29:01: For whom an hour twice a day became twice the date of five minutes.

00:29:06: And this is something that you realize only from what's happening in clinical practice.

00:29:13: We are happy to have treatment but is it tailored for patient?

00:29:17: Is there anything we're learning on clinical practice?

00:29:20: Also some benefits of certain treatments that we did not expect, or combination of treatment that you're learning once the drug is on market.

00:29:30: And this was what the magic of label expansion happening as well where we see these additional benefits and not only addition side effects.

00:29:39: Let's come to last point I'll discuss today.

00:29:42: What can go wrong with integrated evidence planning?

00:29:46: So what are your top mistakes that happen in Integrated Evidence Planning?

00:29:52: There were many.

00:29:54: From a strategy, I would say starting too late... Oh!

00:29:59: We're missing an element and that causes us additional delays.

00:30:03: Operationally is really being rooted into our silo's weight of working within function.

00:30:12: so preference to functional excellence over the needs.

00:30:20: If we are talking today to biostatisticians and other quantitative scientists, it doesn't matter what your title is.

00:30:27: It doesn't matters if you're sitting in one group or the other.

00:30:30: work together ask yourself difficult questions.

00:30:33: so this siloed way of thinking my method versus your method Is something that all need to overcome?

00:30:41: Oh yes

00:30:42: So that's when communication, communication, information internally and externally.

00:30:49: You mentioned one of these, lost in translation.

00:30:52: We need functional evidence on functional aspects and no-one knows what these are.

00:30:58: so I would say start early to avoid this.

00:31:01: mistakes.

00:31:02: talk with each other.

00:31:03: don't fall in love your own methodology or function.

00:31:08: you're sitting there because it is important be excellent But your impact is really rooted in collaboration and communication with others, understanding what the problem is.

00:31:20: And so why we do all this?

00:31:23: This is for the patient.

00:31:24: What you want to do is remove all hurdles on our way together.

00:31:29: that would be my three topics amongst others.

00:31:33: I love the rule of three.

00:31:34: If we go into more then there will probably another podcast episode.

00:31:42: Palmer is a fascinating space anyway.

00:31:45: Yeah, and it's evolving.

00:31:47: so thanks so much Gerana for this great discussion where we talk about all kind of different things evidence planning from Where I should start who should be involved?

00:31:58: What kind of data pieces you have?

00:32:00: quantitative qualitative How was the whole thing evolves over time from preclinical research to post launch area?

00:32:09: Who gets involved talked really, really a lot about this.

00:32:13: And I hope people understand that this is a really important thing and shouldn't be underestimated in terms of importance... ...and all the peoples who are listening here should get involved.

00:32:26: If you have one takeaway message from today for the listener what would it be?

00:32:33: Work together and communicate!

00:32:36: This is my collaborate.

00:32:39: If you want to go fast, go alone.

00:32:42: Go together and be proud of your methodological expertise.

00:32:46: Engage Talk about it Ask questions.

00:32:49: Everybody needs you for success For joint success.

00:32:54: Thanks so much.

00:32:55: You're welcome And

00:32:57: thank-you for having me.

00:33:00: This show was created in association with Pierser Thanks to Rainn & her team at VVS who helped us show on the background.

00:33:10: Thank you for listening.

00:33:11: reach your potential, leak right science and serve the patients.

00:33:15: Just be an effective statistician!