00:00:00: You are listening to the Effective Statician Podcast.

00:00:04: The weekly podcast with Alexander Schacht and Benjamin Piske designed help you reach a potential, lead great science & serve patients while having a great work-life balance!

00:00:22: In addition our premium courses on the Effectives Staticians Academy.

00:00:28: we also have lots of free resources for you across all kind of different topics within Z-Academy.

00:00:38: Head over to theeffectivestatistician.com and find The Academy, And much more For You To Become An Effective Statistician.

00:00:49: I'm producing this podcast in association with PSI A community dedicated to leading and promoting user statistics within the healthcare industry for the benefit of patients.

00:01:01: Join PSI today to further develop your statistical capabilities with access to the ever-growing video on demand content library, free registration for all PSI webinars and much more.

00:01:13: Head over to the PSI website at psiweb.org To learn more about PSI activities and become a PSI member to pay!

00:01:30: Welcome to another episode of The Effective Statistician And Today I'm super happy to talk about a topic that was only radar for quite some time, but I never found the right person to actually talk about it.

00:01:46: And therefore i'm super happy that Ning is today on the show and we can speak about our consortium.

00:01:55: so welcome to the episode and see this podcasting.

00:01:58: maybe you can start by introducing yourself.

00:02:02: That's great.

00:02:03: thankyou so much for having me.

00:02:05: So my name is Ning, and right now I'm at Wee.

00:02:08: I recently joined FB about three months ago And before that i spent ten years as Roche Genotech .I was trained as a statistician so I graduated from University of Wisconsin Madison with the focus on computational genomics.

00:02:22: After that stay in academia little bit At Roche ,i started as an statistician for the first five six-years and then kind of like on the side because i don't really know sass coming to industry.

00:02:34: So myself and also several friends, Waving Roche.

00:02:37: we actually started a R-user community at that time.

00:02:41: Head of Kitchen People how to like write our packages How To Write Shiny Apps And How To Use Git.

00:02:47: And then during my six or six years learning at Roche Gen Tech basically Our leadership team decided to have this strategy Of adopting open source languages and modernized technology.

00:02:58: so I got moved into an innovation team to drive adoption of R, to drive the cloud migration and also get adoption in shiny adoption etc.

00:03:13: So that was kind how I got involved into this whole very exciting R transformation within the cross-industry world as well with Roche Genetag at that time.

00:03:23: I think there's a couple companies who invested quite a lot into R. Roche was definitely one of them Early movers, many more companies that go into that direction.

00:03:34: But I think none of the company's have as radically transformed this rush date at least looking from outside to see our consortium.

00:03:44: did you personally got involved in that?

00:03:47: How would it happen?

00:03:49: so basically it kind was triggered by some internal discussion about why are we not using R for regulatory submission Actually adoption of R, I think within Roche we have been discussing that for more than ten years.

00:04:02: What i heard was even in twenty like ten-years time people actually started to talk about of R and etc.

00:04:11: as a group We try to figure out why the industry is not using our for regulatory submission.

00:04:17: In the beginning People were like oh because FDA doesn't accept that.

00:04:20: But if you do a little bit more research, You realize since twenty fifteen I think FDA issued A statement saying that FDA doesn't require any specific programming language.

00:04:30: And we just need to make sure That your program is traceable yet have high quality and they will be repeating the same message during many conferences over there.

00:04:39: So basically FDA does assess R in also for the kind of younger generation reviewers.

00:04:45: We learned that They come out of school with our skill pie cell skills and they already use our in their day-to-day job anyways.

00:04:52: And we realized the bottleneck is really that there's no good example showing people how to us for regulatory submission, like put your materials into a certain format through the ECTD submission portal so that the FDA reviewers can retrieve information needed during review.

00:05:13: So with that We kind of decided to have A working group working together with FDA to do some kind of mock submissions, to FDA and put everything in the public so that for companies who want to do our submission they can use those mock submissions as a template.

00:05:31: To show them how to do certain things.

00:05:33: This is so cool.

00:05:34: When I remember when i left university In two thousand and two years ago I was already using R&D Although I was more of a sauce-trained person, but all the new younger people at that time were already using R and also packages for ready done in R. And it took quite some time to get here!

00:05:59: But i think the R consortium actually played very critical role inside.

00:06:05: what's the role of our consortium in whole R space?

00:06:09: That's a very good question.

00:06:10: So I think our consortium as an organization, basically it is not pharma specific so it is non-profit organisation trying to promote the good practices and usage of R across different industries.

00:06:23: During recent years i see pharma definitely being one of most active space with the most number working groups for our consortia because our research adoption of R in open source language Under our consortium, there are several working groups.

00:06:39: I think the most active two was we were like that submission-working group where we work together with FDA on a number of pilot submissions trying to showcase how do submit certain components to FDA and also in our validation hub when people found materials really useful they try come up with white papers to document.

00:07:04: what does My addition means for open source language and also the open source some of tools over there in terms of quality metrics measurement, etc.

00:07:14: And I know several conferences are ready to conference that our consortium is supporting.

00:07:20: so i definitely feel like a consortium as a non-profit organization provided this very nice platform for us collaborate across companies with FDA.

00:07:32: So basically the R-consortium is an organizational backbone of our whole community across different industries and it's great to hear that the farm industry got on board, now very active.

00:07:46: And I'm pretty sure many other industries can benefit from this.

00:07:53: You already mentioned submission case studies.

00:07:57: so what do you see?

00:07:59: How are these leveraged by the different companies, assist us?

00:08:05: Embraced or is it still some kind of outlier here and there?

00:08:10: on what's the status off our base submissions.

00:08:13: Right yeah maybe I could first give an overview about a pilot that we're running in.

00:08:19: then I can also share like how to good practice evolve over time based on our learnings etc.

00:08:26: So for Archive Socialim submission working group, right now we are at pilots six and seven.

00:08:32: We already finished four pilots.

00:08:34: there were four pilots that already have FDA response letter.

00:08:38: recently the number-four was completed.

00:08:40: if people interested The latter is on our website Right Now.

00:08:44: so the first one was very simple pilot.

00:08:46: one Was Very Simple basically?

00:08:48: We try to submit Four TLGs using the city's pilot with the similar data.

00:08:54: To submit a four TLG We had the ECD portal and then to make sure that FDA reviewers were able use the exact same package, same environment.

00:09:04: To reproduce this for TLGs.

00:09:07: For panel number two we add a SHINee component.

00:09:11: so basically we believe right now with trials being more complicated And when more endpoints one more complex measurements etc.

00:09:21: like SHINEE could be really nice supplementary tool to accelerate review.

00:09:25: So for panel two, we success a Shiny app to FDA.

00:09:29: In panel three in addition to TLF code.

00:09:33: We also submitted Adam code so that it will enable the FDA reviewers To reproduce codes and generate Adam as well.

00:09:41: In Panel four The recent one was very bold one.

00:09:45: Basically we submit the Shiny app through container in WebAssembly.

00:09:50: So the FDA reviewer was able to reproduce an environment using Docker and WebAssembly, but there were also some IT challenges.

00:09:58: that's kind of outlighted in the response letter.

00:10:02: In part five we are trying to submit a dataset in JSON format instead of XPT format so people don't need to do this artificial conversion from like a JSON data file to XPT data file over here In then, PaloSix and Seven are kind of like our playground right now to try out different AI solutions.

00:10:23: Palo Six is trying to use AI to generate the admin TLF And Palo Seven is trying create benchmark data set in benchmark test cases for AI tools and also a traditional open source tool.

00:10:38: So that's where we're at.

00:10:40: Back to your question about how does this impact the company's real submission.

00:10:46: What I heard was definitely that, from multiple companies like Novel and D&J That when they did their first R-submission They closely studied our pilot submission cases And tried to mimic what we did.

00:11:00: Tryed avoid the bottleneck or the caveats we encountered at that time Roached a saying We try follow open pilots' learnings over there.

00:11:10: And the other thing I want to call out is that also because of those pilot submissions, we were able to influence FDA To update some of their guidance or documents.

00:11:21: For example like in the past In there file format guidance They said like no Z file and they didn't specify too much about Like the files with .r like suffix.

00:11:33: so Not sure whether we can submit a perpetual R package because often types that come with the zip file, the suffix and also wasn't sure how to submit it.rfile.

00:11:46: so in the very beginning even artificially converted .rfiles to .txt files.

00:11:51: Last year actually FDA was able update their file format guidance specifically saying all of us allow dozen zip files for our packages submission We allowed for like .R etc.

00:12:04: Ibrisapixx is around our ecosystem.

00:12:06: So with that actually does streamline a lot of the operational side of submission as well.

00:12:12: That's so cool, this overall collaboration and influence it has on various companies but also on the FDA to choose how people if they have common goal can work very effectively together.

00:12:27: Also think that will help quite alot in making process better, like the Chinese thing.

00:12:35: Have you done anything that is more HTML based?

00:12:39: So some kind of markdown.

00:12:41: so it's at your.

00:12:42: don't need a container or something like this but just an html file to have some interactive graphics.

00:12:49: We haven't submitted any HTML But I feel like the web assembly portion of the pilot four is toward that direction.

00:12:57: Basically for our web assembly approach reviewer will be able opened the shiny and been their own browser without too much of environment setup.

00:13:08: So that one actually was through quite smoothly, I would say

00:13:12: because they think is definitely part of the future.

00:13:15: said we don't submit documents where's he only difference to see pre electronic submission?

00:13:23: Is it now a PDF or document instead off an actual binder physical binder and the biggest advantages having hyperlinks, but now actually embracing what's possible with electronic submissions.

00:13:40: Having interactive things, having a more kind of traceability all these kinds of different things is so much opportunity in the future.

00:13:49: And I'm really curious What will happen?

00:13:52: In terms of such generative AI... ...I am looking forward to that because i would personally have lots questions around it.

00:13:59: Because or a tested AI for producing text around tables and things like this.

00:14:07: And it's not as I would like to be still, but what is your perception of to know?

00:14:14: Yeah that exactly kind our motivation of moving towards pilot seven where i mentioned we started thinking about what does Benchmark mean For the clinical trial reporting also for trial design basically the biostat and programming related test.

00:14:31: So for panel seven, actually what we're doing right now is that We ask the community to contribute their AI skills.

00:14:39: As you probably know That basically skill Is a very powerful approach To provide SOP-II AI algorithm To tell them when to use certain packages When to use some tools And how QC before return results to me etc.

00:14:58: So we asked the community to open-source several skills.

00:15:03: they're writing.

00:15:04: Right now, there are three sites of skills one is about sequential design One is about trial simulation and one it's about item generation.

00:15:14: After open sourcing those skills right now We also ask the communities to contribute benchmark test cases so that people can provide us with the craziest case or most weird corner case they encountered, and we try to improve those AI tools based on those corner cases.

00:15:34: We're still at a very early stage I think which only started probably like two months or more... A little bit over a month ago.

00:15:42: but there is a very strong contribution over there.

00:15:45: I saw that there are like over fifty stars in the repulse free and we have over hundred issues now.

00:15:51: so hopefully with this approach will come up some high quality skills.

00:15:56: it can help us to manage the kind of like uncertainty around generative AI a little bit better and then too, kinda really unleash that in clinical trial setting.

00:16:07: I have some feelings there is another episode inside.

00:16:10: Thanks so much Ning for this awesome discussion And getting our information about the role of the R Consortium What's overall handling going on In the community how much that is impacting our industry, our collaboration with the FDA and also to have a look out what further things will come in the future.

00:16:37: Thanks so much Ning!

00:16:39: One final thing if someone is more new in the R consortium

00:16:46: area...

00:16:47: What would be good things to check-out & learn about?

00:16:52: Yeah, that's a good question.

00:16:53: So for our consortium one thing I'm really proud about is everything is public.

00:16:59: so For anyone who is interested in our working group or any of the projects all The meeting minutes are on our website and other recordings.

00:17:07: You can also find out our website again.

00:17:10: Also everybody welcome to join our slack channel?

00:17:18: and identify which specific area they want to get

00:17:28: involved in.

00:17:48: Reach your potential, leak-rate silence and surf patient.

00:17:52: Just be an effective statistician!

00:17:54: This show was created in association with Pearson.

00:18:03: Thanks to Rainn & ThirteenFUVS for help with the show's background and thank you all for listening.

00:18:09: Reach Your Potential, Leak-Rate Silence And Surfed Patients.

00:18:13: Just Be An Effective Statistician.