00:00:00: You are listening to the Effective Statician podcast.

00:00:04: The weekly podcast with Alexander Schacht and Benjamin Piske designed to help you reach a potential, lead great science & serve patients while having a great work-life balance!

00:00:22: In addition our premium courses on the Effectives Staticians Academy.

00:00:28: we also have lots of free resources for you across all kind of different topics within that academy.

00:00:38: Head over to theeffectivestatistician.com and find The Academy, And much more For You To Become An Effective Statistician.

00:00:48: I'm producing this podcast in association with PSI A community dedicated to leading and promoting user statistics within the healthcare industry for the benefit of patients.

00:01:01: Join PSI today to further develop your statistical capabilities with access about one of my favorite topics and these are external control arms.

00:01:45: But before we dive into this, then maybe you can give a little bit off the background where your coming from?

00:01:52: And how have got interested in to this topic?

00:01:56: Sure happy too!

00:01:57: Thanks so much for having me on your podcast chat today.

00:02:01: So I am a PhD biostatistician by training came from a background and interest in public health.

00:02:11: And so I've always been motivated by statistical questions, problems that are rooted in real applicable issues.

00:02:20: My focus is research has always been on propensity score type methods and ways we can combine trials with observational studies multiple data sources to learn things from disparate source together.

00:02:36: This concept of the external control arm study came about on my mind through that work and as a way to bring in external information, to support how we've learned from trials.

00:02:48: So I worked at Johnson & Johnson currently as a statistical methodologist an internal consultant because they worked up Flatiron Health which is an oncology real world data vendor.

00:02:59: i have had various exposure to ECA's different types designs throughout those experiences.

00:03:07: Very good, awesome!

00:03:08: Yeah it's definitely a very hot topic this kind of interface between real-world data and clinical trial data.

00:03:18: also I don't really the word real world because of course clinical trial is also actual patients but one more experimental setup And the other one is observational setup.

00:03:37: So I like the term observational data actually much more, but real rotators are the buzzwords.

00:03:43: that's all around.

00:03:44: yeah i totally agree with you there.

00:03:47: try to use the lingo and the terminology That The Field Is Using.

00:03:51: But Yeah Certainly Trial Data Are Patience.

00:03:56: It Still Considered Real World.

00:03:58: I Know There'S A Lot Of Debate On These Terms And Weather observational or non-experimental because of course to some extent trials are also observational.

00:04:09: even though they're interventional.

00:04:10: we're also still observing how patients are progressing in their disease and treatment.

00:04:16: So yeah I like to use any, all those terms...I feel i typically describe them as experimental versus non- experimental.

00:04:24: when it comes to the electronic health records that used to talk about real-world data, whether it's retrospective or prospectively generated and collected.

00:04:34: But for all intents and purposes I tend to use real world data just a bit at the times

00:04:44: in.

00:04:57: What are these biases about?

00:05:00: Yeah.

00:05:00: That's a great question.

00:05:02: So when we think about external control arm analyses, one of the most important pieces I think When doing this type of analysis are making sure We have apples to apples comparisons.

00:05:13: so when we identify A comparator arm whether it's from real world data or electronic health records Or even historical trial Data i think typically there is alot Of emphasis on how we make them apples To apples in terms of populations that We want to emulate a study that's randomized and make the two groups comparable in terms of who are recruited and compared.

00:05:38: This research focused on another apples-to-apples comparison, an issue of how endpoints are constructed and measured.

00:05:46: so not only do we have similarity between who is observed in each group but also similarity in when they're assessed for disease.

00:05:59: So this issue, I think mostly comes up when combining a trial with some say retrospective electronic health records based data set where you know.

00:06:11: typically in the trial we have very rigorous standards and control over how assessments are done.

00:06:18: And when they're conducted.

00:06:20: so we have a strict schedule of events where patients come in at regular intervals and clinicians will use specific instruments or tools, whether it's imaging an oncology like solid tumor oncology.

00:06:36: Or validated scales for certain outcomes to be able to assess the patient trajectory and their disease In real world data.

00:06:45: we typically don't have access that wealth information.

00:06:49: We might just a clinical document that is ingested in some EHR system, that says patient disease progressed or patients are doing better.

00:07:02: And how does that compare to the trial standard?

00:07:06: That's really an issue we're tackling with this research trying to understand what those differences between ascertainment of the endpoint and then how can we model them more comfortable for one another so we can make more of an apples-to-apples comparison and be more confident that the benchmarking against this comparator is really due to the difference in treatment, not do it a different than ascertainment of the outcome.

00:07:36: So there's basically two big points.

00:07:38: one is when its collected?

00:07:41: And then the other thing is what is collected and how precise is it collected?

00:07:46: I know.

00:07:46: Of course In a clinical trial you have these regular schedules but still there's some kind of variation around it.

00:07:54: If you say, well we want the patients to come in after twelve weeks most patients will not come exactly after twelve-weeks plus minus something.

00:08:04: and so this variability...of course You may have something similar in the external controller.

00:08:10: How can if okay?

00:08:12: There is some bias for example In health records improvement or worsening on no change.

00:08:24: And in your clinical trials, you have something that is a real scale to kind of give much more information about how much improvement and how much worsening.

00:08:33: How can we now mitigate these differences?

00:08:37: Yeah

00:08:38: it's great question.

00:08:39: I think the short answer depends what data you have, and how well you understand the comparison between what is measured in a trial?

00:08:50: And what's available.

00:08:52: In these whether it's clinical notes or What's available in the comparator arm.

00:08:56: so to that point I think The best way To mitigate those differences Is to have some validation study conducted Or Some data available where You can for each patient directly compare These two different methods of the outcome.

00:09:15: where you can say, okay on this visit there was imaging done.

00:09:20: And per certain criteria it was determined that the patient was responding to treatment and then you also have some version On that same day for that same patient.

00:09:32: That says what sort is documented in that EHR system?

00:09:37: You can compare.

00:09:38: how often do they draw the same conclusion, how often do they not?

00:09:42: And then you can use those metrics of sensitivity or specificity in this validation study and use these metrics to adjust your analyses for future studies.

00:09:57: The best way would be have some data that studies these two phenomena together.

00:10:05: That's obviously easier said than done and in a lot of disease settings that might not be easy or even feasible to design.

00:10:12: So there are ways in which simulation studies can be helpful, partnered with working with clinicians and experts who understand how these disease endpoints are measured and what might cause differences between the two ways of measuring them say, okay well what if this difference is this large?

00:10:36: or what if it's maybe negligible?

00:10:38: and do some sort of quantitative bias analysis.

00:10:42: Or tipping point analysis to see how off are we?

00:10:45: If We assume certain differences but I think even you mentioned What if we assume there Are these biases?

00:10:52: i think that an Even more critical first step Is To stop And Think Are These Issues Going To Manifest As Bias Or Not?

00:11:01: And that's also where simulations or descriptive analyses can be helpful.

00:11:06: What we've seen in our research is, some of these issues matter and they don't.

00:11:13: In particularly tricky cases it might have strong opposing forces at play.

00:11:22: so you may have a strong bias savoring the alchemine one direction due to certain false positives of the outcome, and then you might have strong bias in the other direction to do that.

00:11:36: Other types of errors or due to be different than when patients come into assessment scheduling.

00:11:43: so I think even before we go into mitigation there's a lot important work needs to understand.

00:11:51: in any one case given data what is actually an issue because you might find it's not and then the types of mitigation strategies may not be necessary.

00:12:02: Yep, so as mentioned one important point to kind of the size is important but of course the direction is probably even more important.

00:12:09: yeah will it shrink the current treatment effect or would that increase the current?

00:12:14: that always comes to mind when we talk about combining external control arms with clinical trial data is, When do you actually do these things?

00:12:29: And speaking about claims data or registry data.

00:12:33: These data exist already.

00:12:35: You can have a look into this and better understand what's in there To also inform What you want to collect from your study When for example certain data are available.

00:12:47: Make sure That you're also collecting that in the study and then you have at least one problem less in terms of that.

00:12:53: And if you also need other further data from a regulatory point-of view, You already have some kind of internal validation study in it because you collect both types of data.

00:13:05: So this is super helpful!

00:13:07: That's actually one of my questions.

00:13:09: Would you look into Revolve Data before you run the Study to inform the protocol?

00:13:14: Or would you run that kind of independent or would you work on that?

00:13:18: Yeah, that's a great question.

00:13:20: I think you raised one interesting point and idea of how can the selection of real world data or they use case informed design for single arm study?

00:13:30: One thing that could be advantageous is looking at how endpoints are ascertained in retrospective databases.

00:13:39: if you're going to that's already off the shelf, he licensed from a real world data vendor getting some information on not necessarily the outcomes themselves and not biasing.

00:13:51: The conclusions you might draw.

00:13:52: but understanding the data generation.

00:13:55: are certain vendors using clinical notes and abstracting endpoints based on they see a clinician said patient progressed?

00:14:04: And that's what represented in algorithms or deriving the outcomes using labs in other ways, understanding how it's ascertained can actually be helpful.

00:14:19: And if it is say like a clinician documented response one interesting idea could be to bake that into the trial.

00:14:30: so in addition look at the primary endpoint, there could be ways to as you alluded to bacon a little validation sample or say okay for X percent of patients are for all will also ask the clinician.

00:14:51: To write this note As if in an EHR document Or have some sort of measure based on similar rules Of whatever is collected In real world data set and that can help understand The relationship between the two.

00:15:06: I think there are logistical considerations with that.

00:15:10: Obviously, when planning a trial you don't want to place too much burden on collecting certain values or measures.

00:15:19: but I do think that's one way to improve the single arm design to anticipate some of these questions.

00:15:35: There's strict protocol elements that might be needed, but there is also increasing evidence of introducing pragmatic elements into trials that are beneficial.

00:15:47: So it could be feasible to bake in more flexibility of assessment timing if you think that's more aligned with clinical practice and what.

00:15:59: But I think this is also coupled with the important role of pre-specification and transparency.

00:16:04: You certainly don't want to look too much at the data before you use the data for the actual analysis, so there might be ways in discussions when designing the study and picking which comparator data source you will use without working too much and biasing yourself with the data itself, if that makes sense.

00:16:32: Yes it's an interesting discussion.

00:16:36: in terms of what does pre-specification actually mean here?

00:16:40: I would argue anything you do before your first patient visit is pre specified irrespective to what we have done because we could use these real-world evidence data all the time, and also it's just from a practical point of view.

00:16:59: Nobody can ever control what you have done with this external real world evidence data or one of your key opinion leader that informs the clinical study design has done with their data is... Or maybe they publish it?

00:17:13: So there are some kind of external data I think is not too critical from a pre-specification, if you run all the analysis beforehand.

00:17:25: And it actually includes them also into your article and in to your SAP.

00:17:32: This is how the data will look like... That's basically now the hurdles that we want to jump over!

00:17:38: In a sense its like doing one sided statistical test where set specific hurdle and say, well that's what we want to jump over.

00:17:49: But then of course you can't add stuff later on.

00:17:52: yes it gets a little bit more tricky or you need to pre-specify how exactly will do this.

00:17:59: but thats definitely really interesting thing here.

00:18:02: I don't know whether protocols are up to date in terms for that.

00:18:10: just thinking about templates definitely something for a good discussion to have with your regulators.

00:18:21: And discuss what that means, about kind of the basis and decision rules because in the end we really want clear transparent decision rules when we know exactly where we jumped over.

00:18:39: what?

00:18:39: In terms of that, do you have any experience in terms of documenting these things and pre-specification achieving transparency?

00:18:53: Yeah.

00:18:54: That's a great question!

00:18:55: I think when it comes to this issue One thing that we, or at least I have noticed with the sort of issue of measurement error is it's often recognized as in-the-room with us and possibly an issue but not often addressed in practice.

00:19:17: And so i think one of the goals of the work that we underwent was really laying out these biases due to differences how when patients are assessed and building some tools was an effort to help encourage teams to think more rigorously, and quantitatively about these issues.

00:19:36: So I think doing further R&D on how these problems manifest in different disease settings can help bake in some of these concepts when pre-specifying external control arm analyses.

00:19:52: so I think, at least from what i've seen and this may not be the case completely but it seems as problem of endpoint misalignment is listed as a limitation or possible source.

00:20:04: Of an issue that is left unaddressed in our hope Is That This Can Be Incorporated Further When Designing A Study Or Writing Up An SAP To Really Make Space In The Protocol To Say when addition to these primary analyses we will evaluate how the endpoint definition affects our results to really proactively address some of these questions.

00:20:31: And there has been at least one, if not more regulatory responses.

00:20:36: I've seen where These question of endpoints have come up in feedback to submissions.

00:20:41: Where it's been documented FTA has said Oh but this endpoint might be measured differently or patients assessed at different time intervals than your trial when using this retrospective data.

00:20:54: So I do think moving towards prespecifying how those questions will be addressed for the feedback would be really critical.

00:21:03: and that's where, at least i hope as a field with ECA is we're moving toward so being more proactive rather then reactive?

00:21:11: And whether The timing of our assessments will have negligible impact on bias, and here's why.

00:21:23: Or we think the real world endpoint might be biased towards later times based on how the end point is ascertained in.

00:21:31: Here Is Why even those pieces of information or simulation can be helpful when writing a protocol?

00:21:38: I think it's really important to understand this.

00:21:41: yeah In general we think measurement error if its at randoms and we just need to have more patience or it's in a way conservative.

00:21:50: But this kind of term, conservative I'm not super happy with because these can also be safety endpoints.

00:21:57: if its not clearly documented We may actually overlook certain safety signals.

00:22:04: It is like always better data Yeah!

00:22:08: And when it comes One sort of big assumption in the framing that I make here is that, The error is not random by design.

00:22:18: If we think of...the way things are measured on a trial Are Not Perfect Themselves.

00:22:23: But if We treat the Trial as the gold standard Or..The Way Of Assetting the Endpoint That We Want Our Comparitor To Align to then any difference from how the endpoint is ascertained or when patients come in relative to what happens on trial, is error not at random because it's quite systematic.

00:22:46: There an issue of these differences and yes there are certain types issues that may land towards biasing toward better-or worse treatment benefit like for example if you think that comparator way measuring end point is always going to undercount or delay when they detect it, then you might think that your comparator will be biased towards later times and not make your treatment effect look diluted.

00:23:19: So there's some sort of intuition between what types of errors may lead into what type of biases but again to what extent?

00:23:30: until you dig into the problems.

00:23:34: Thanks so much!

00:23:35: Now if the listener wants to learn a little bit more about this and dig deeper, What kind of resources would you recommend or what kinds of areas?

00:23:46: Would you guide listeners too?

00:23:48: Yeah I think first and foremost The regulatory guidance documents from agencies like FDA EMA are great places To get some sort high-level understanding of these issues and how agencies are thinking about them.

00:24:06: I know EMA is, i think working on draft guidance on eca's.

00:24:11: they just had this workshop to solicit input and generate ideas in november of last year which was where i presented some of this work.

00:24:21: so i think looking at regulatory guidance as well as available how agencies are responding about these problems, our good places to learn about the regulatory perspective.

00:24:39: I think also there's a lot of rich literature in epidemiology journals, pharmacopedemiology journals that talk about measurement error or misclassification.

00:24:52: maybe not in the context of external control arms but they're highly relevant.

00:24:57: and then We have a few papers that have been published through this research.

00:25:03: Our third and final paper is actually about to be published in the American Journal of Epidemiology any day now, where we walkthrough into context of external control arms.

00:25:28: adjust survival curves when we suspect the outcomes are misaligned.

00:25:33: And then this last paper that's being published in AJE is more focused on quantitative bias analysis techniques and how we can address these problems, even if we don't have the data to do so and want to contextualize our results in the presence of measurement error?

00:25:50: So I'm happy to share links to those after we speak.

00:25:57: I think we're really just at the tip of the iceberg with trying to uncover these issues and developing tools in a context of ECA is too, to address them.

00:26:08: I completely agree this ECA topic as huge topic.

00:26:11: that's not so first episode on where i'm covering says And today We talked Really about more The end point problems, not so much the population problems and comparison problems but really understanding biases around CN points as well as touching on pre-specification under a couple of other more practical topics.

00:26:34: And we'll definitely put all the links into the show notes.

00:26:37: here you can just head over to the effective statistician dot com search for Ben Ackerman and then I

00:26:49: get one just last quick point.

00:26:51: Yesterday we focused on these endpoint alignment issues, it's also important to see that all of the biases come up in external control arms whether they're due to population or endpoints are happening simultaneously.

00:27:08: so these analyses have this constellation and more important to understand how these pieces work in isolation, but also how they interplay together.

00:27:24: So bias due to measurement difference might also counteract bias due population differences and so we might imagine a case where we do an ECA.

00:27:38: everything looks balanced and unbiased But under the hood there are all of these strong opposing forces.

00:27:46: contextualizing and understanding how these pieces function on their own, helps us put the puzzle pieces together.

00:27:54: And do more rigorous

00:27:55: research.".

00:27:56: Thanks so much.

00:27:57: It's a very good final sentence!

00:28:00: I have some feeling we'll talk again...

00:28:02: Yes definitely and thanks for having me and looking forward to seeing where field continues to move with this type of studies in design.

00:28:15: This show was created in association with PSI.

00:28:19: Thanks to Reign and her team at VVS, who help with the show in the background.

00:28:23: And thank you for listening!

00:28:25: Reach your potential, leak great silence & serve patients.

00:28:29: Just be an effective statistician.