00:00:00: You are listening to the Effective Statician Podcast.

00:00:04: The weekly podcast with Alexander Schacht and Benjamin Piske designed help you reach a potential, lead great science & serve patients while having a great work-life balance!

00:00:22: In addition our premium courses on the Effectives Staticians Academy.

00:00:28: we also have lots of free resources for you across all kind of different topics within that academy.

00:00:38: Head over to theeffectivestatistician.com and find The Academy, And much more For You To Become An Effective Statistician.

00:00:49: I'm producing this podcast in association with PSI A community dedicated to leading and promoting user statistics within the healthcare industry to benefit patients.

00:01:01: Join PSI today to further develop your statistical capabilities with access.

00:01:30: Welcome to another episode of The Effective Statistation.

00:01:33: It's in the middle of summer here and it is going to be really, really hot!

00:01:38: And we are talking about two topics that are really hot as well... ...and combined they're simply even hotter.

00:01:48: R has been quite a big topic for long time in this industry.. ..and continues to be one.

00:01:56: Of course other topic is HTA which all the changes in terms of European JCA process, there is so many things going up.

00:02:06: So we'll talk about exactly that and for this I am very happy to have Robert Smith here on the line.

00:02:16: he has his own company you can tell a little bit more but he's deeply into this topic.

00:02:25: So, Robert before we go into our end HTA maybe you can introduce yourself a little bit and how you came to get connected with this topic.

00:02:35: Yeah sure thanks for having me.

00:02:37: it's great to be on the podcast.

00:02:38: so I first got involved in the application of our health economics during PhD at Shara University of Sheffield.

00:02:46: i'd previously worked in Health Consulting in universities.

00:02:56: They're kind of renowned for alcohol, smoking physical activity modeling.

00:03:02: And because the nature of these models they tend to be substantially more complex than a lot of the pharmaceutical based models that we see for submissions to HTA agencies and so the modelling structures tend to Be slightly more complex with capturing things like human behavior?

00:03:19: There's a lot Of feedback loops an individual level effects which means that the models end up being substantially, more computationally burdensome and so tend to be built in programming languages.

00:03:29: So during my PhD I was specialising at building out these types of models and realised that there were a lot other benefits rather than just computational time That could be applied for models for HTA and health economics More specifically And got very involved with group called the Arthur HTA consortium.

00:03:48: The work they're doing is trying make methods more open transparent and widely used and found that group to be incredibly welcoming, a great sort of industry group.

00:04:00: That's how I've come into being involved in that.

00:04:03: And the interesting piece is when i think about health economics models... ...I primarily think about Excel yeah?

00:04:09: When all the models that are seen were usually excel based why are instead of Excel ?

00:04:16: So what has happened over time?

00:04:19: we've moved from a system or situation where a lot of the models were very simple.

00:04:24: So if you think back ten, twenty years there was some very simple calculations being made with some assumptions around quality life and longevity from these various interventions.

00:04:36: And these are originally almost borderline back-of-an-envelope calculations.

00:04:40: that then became one or two tabs for an Excel spreadsheet some calculations and fits to curves.

00:04:47: And then they gradually introduced within the models things like one-way sensitivity analysis, and various other components which meant that you suddenly had lots of different tabs in subanalysis running within a model .

00:04:57: Then we introduce probabilistic sensitivity analysis ,and then expective value information.

00:05:01: We've had this kind of gradual shift from what was very simple set of calculations towards now for submissions one-way sensitivity analysis on the probabilistic model, which changed kind of level of computational burden and complexity.

00:05:20: Of the models from being a simple calculation And we've slept walked with same piece of software.

00:05:25: that isn't really quite fit for purpose.

00:05:27: Some people are having to hack solutions To get models run in Excel But thats not just say excel doesn't have place.

00:05:34: I think we tried be open into group.

00:05:37: That there's lots times where Excel is absolutely fine.

00:05:41: It's just that as you start moving towards models which are trying to capture individual level effects or interactions between groups of people, any other level of complexity it becomes very frustrating.

00:05:57: To work with a tool is absolutely fantastic for analysis and financial data or spreadsheet software for accountancy but its not really designed a micro simulation model.

00:06:10: It's not really what Excel is built for?

00:06:12: Yeah, honestly when I think about that, Excel has so many different features that make me go crazy.

00:06:20: if you have just the simplest thing moving from US setting to German setting and all kinds of things break down because of the dot in the comma or maybe accidentally change something somewhere as cell And it's impossible to debug that or to put things somehow in the development and production form.

00:06:43: That's also not possible, so I always... In terms of Excel i'm super anxious about these kind of thing

00:06:52: Yeah!

00:06:52: And when you end up working now or inherit a model that is built on Excel from somebody else.

00:06:57: So your doing country adaptation Or developing some element of this model You inherit it which is just huge number of tabs The share volume of cells most of which do exactly the same thing as each other with a drag dump, but hundreds or thousands cells that nobody could possibly go through and review.

00:07:19: It takes absolutely forever!

00:07:21: Each cell often will contain quite large equations – almost impossible to decipher what it does – so there's huge amount time spent by health economists gradually working through cells in Excel.

00:07:37: other industries work.

00:07:38: So it's an incredibly manual process and it is not like.

00:07:41: Uber doesn't send out a million different Excel files for the location of its drivers or their routes to certain places, just no modern approach at this problem.

00:07:49: so really quite often people are using Excel because it's the status quo as opposed.

00:07:58: Where we really see the shift is where suddenly it completely falls flat.

00:08:02: So I think people are very willing to go through the pain because It's The Status Quo and they know that other People Are Used To Receiving Models In That Way, And so They'll Just Continue Along Because We're All Very Averse To Change Unless You Absolutely Have To But While We'Re Seeing The Switches Initially Is For Things Like Discrete Event Simulations Micro-Simulations Anything Where you Need To Do Value Information Analysis infectious disease modeling is, again anywhere where there's interactions between individuals or groups of individuals.

00:08:33: Or resource sharing has shifted over to R and I'm sure we'll touch on gen AI because it's the big topic at the moment.

00:08:39: but that also really accelerated the shift in the last probably two years.

00:08:43: three years would have really noticed a ramp up engagement with script based programming.

00:08:48: Let's talk about why did we go for R?

00:08:52: And not another open source language?

00:08:55: That's a really good question, and one we get a lot in the group.

00:08:58: So it is why not Python?

00:08:59: And the first thing to say I think that after HGA Group John Luka might be angry at me for saying...I think there was mistake when it was first named.

00:09:09: Ideally would open source modeling group with idea being should shift from Excel, from spreadsheets to some script-based approach.

00:09:18: And I don't think it really matters too much what the language is.

00:09:21: so sometimes people email saying can i submit this talk?

00:09:24: It's in Python!

00:09:25: We're like yeah go ahead let's look.

00:09:26: we're all very interested because really TheJomp Is Really From Excel From Spreadsheets To A Script Based Approach and anyone who's very well versed and can code an advanced user of R or Python We'll find it quite easy to review the other so we could pick up a model in Python and be able to work through very quickly.

00:09:46: So why are, when the original kind of choice is made?

00:09:49: Why does industry gravitation towards R not Python?

00:09:52: I think that's because historically there were software packages being built unvalidated for survival analysis the clinical side through to health economics.

00:10:05: A lot of that, survival analysis and analyses done on trial data.

00:10:10: there's some fantastic packages FlexServe.

00:10:13: others know kind of our ecosystem will know.

00:10:17: those are widely accepted And if you use these R-packages no one really questions them their quality off the software.

00:10:23: So because of that There has been a natural tendency If your doing your survival analysis in R it kinda makes sense.

00:10:29: just continue though to build model as well.

00:10:32: And so that, I think it is the reason.

00:10:34: and i think R will continue to dominate just because of that legacy in a same way that Excel has remained around.

00:10:40: So suppose there are kind of groupful benefit from that In much the same ways as the excel camp have benefited From their legacy effect.

00:10:48: We're seeing certainly on the clinical space With things like Adam Datasets The FDA approving submissions in R Which we were seeing earlier On in the clinical pipeline.

00:10:58: It'll be interesting To see how this flows through because if there's a movement from SAS to R, then perhaps that benefits further down the pipeline.

00:11:06: A movement for health economic components as well?

00:11:10: I think it is also something to do with how many people in our industry already use R. having validated environments for all these kinds of different things makes things easier In terms of this whole group R for HTA.

00:11:26: maybe you can talk about two words What's happening in this group and especially also to what the goal of this group?

00:11:34: Yes, I think they overall go off.

00:11:36: The group is to make or Propagate use of our for HDA.

00:11:41: so Ranging from kind of survival analysis To model builds two submissions on the practical implications of submissions in art a hd bodies And it's primarily an academic endeavor with many of the key individuals being based at universities either as academic positions, but also in ERGs.

00:12:00: And there's involvement from key individuals in HTA bodies and across industry.

00:12:05: so it was quite a nice mix.

00:12:06: I think we did a poll at the last workshop that we ran an annual workshop We were operating around third being kind of academic ERG-HTA space a third consultancies and a third industry.

00:12:22: So its actually really nice mixed between different stakeholders.

00:12:26: The group really is coming together to try and push the use of open source programming languages in health economics.

00:12:34: And it's gone a long way into providing a kind platform for people to publicize any work that they're doing, any open-source work they are doing or learning what we have around how they've navigated working with industry within HGA bodies or ERGs.

00:12:51: And I think the thing i've been most impressed with is how helpful The group have been to each other.

00:13:00: So, their willingness To publish things open source Their willingness to provide tutorial papers to others To really push these methods along Is very impressive relative to Other industries.

00:13:13: so I Think there's a really good spirit of that Open Source Spirit Of being transparent about what you're doing in the methods and they are the type people are interested in that.

00:13:21: and so then they embody that as well, what we're doing.

00:13:24: What have you been concentrated on?

00:13:27: If I understand it most of your aspects has around economic models these kind things

00:13:34: a real mix.

00:13:35: there's presentations really encourage people to go look at the website after HDA website i'm sure link is in the comments originally set up by John Luca Beo from UCL Bristol and now has a wide pool of the scientific committee who will have their own expertise.

00:13:53: So ranging from NMAs, what was the stats analysis through to modeling?

00:13:58: Through to value information.

00:14:00: so everybody I suppose comes at it with a slightly different background and expertise.

00:14:04: i'd say that area which has the most focus has probably been the modelling side.

00:14:10: there's a lot of talks this year around discrete event simulation micro-simulation and also a lot around kind of statistical analysis makes, NMAs.

00:14:21: And then some are on value information.

00:14:23: but I think the area that A LOT people are particularly interested in is the modelling side.

00:14:28: i think it goes back to The question where Excel is just not very well suited for running discrete event simulations and individual level simulations.

00:14:36: as mentioned before often People will build a model and its quite simple.

00:14:40: Then they want that model to answer a question That's slightly more complex than maybe incorporates individual treatment adherence also thing.

00:14:48: And then there's this shift towards a individual level model and micro simulation or indiscriminate simulation, I think that's where we've seen huge amount of interest in the presentation to see what presentations have an interesting kind of our package to do DS.

00:15:01: so The Warden Package was presented on them at number different talks.

00:15:05: when people presented models they built

00:15:08: This is so cool and as you say, network meta-analysis matching in direct comparisons.

00:15:13: These kind of things are not just helpful for an HTA submission they're also helpful to all kinds other areas.

00:15:21: if you want a design study or do submissions to FDA EMA there's so many aspects.

00:15:29: Have ever worked on something related more than German submissions?

00:15:34: I've not, no.

00:15:35: Unfortunately i don't have that experience... ...I know we had some interaction with a quick on developing our skills and so it's something they're interested in but.. ..i dont personally work at the German market for now!

00:15:49: Hmm, that

00:15:55: sounds fancy.

00:15:57: And my company would really love to get engaged or maybe I personally want to Get engaged in that work?

00:16:03: How does it work?

00:16:04: Yeah so think if you login and look at the RFA-HGA website is probably the best protocol and on there's lots of training materials, so there are lots of links to my upcoming open source Open Access academic papers.

00:16:17: There is the annual workshop that we invite people too!

00:16:19: And generally another hackathon session as well in a year at some point... That was really fun.

00:16:25: I enjoyed it.

00:16:25: People who come to that find engaging so they split off into small groups trying to solve an interesting problem and learn from each other bespoke training sessions that are run for groups from often, from HDA bodies or ERGs.

00:16:41: So there's the RFAHTA kind of consortium group and then I also run a consulting company Dark Peak Analytics that specializes in these methods.

00:16:49: so we do a lot of consulting work into industry and governments on this method as well.

00:16:55: We have our own training courses some which are linked to the RFHA site.

00:17:00: Definitely worth checking those out, but it's obviously great to promote my own stuff where I can.

00:17:05: There is also just a lot of material there.

00:17:07: so we always signposts for free resources and materials that generally suggest people look at them first see whether the courses online are worth it because often they might not be worth it or you read open access papers then get even more interested in coming on course.

00:17:28: This is so cool.

00:17:29: Whenever you want to build a model, or if you wanna have some kind of R-based things built for you it's great way to reach out to roller dentists colleagues at Dark Peaks and Analytics To help you with these kinds of thing And If You're not in the HR area recommend them to your friends In that space Especially the economic people.

00:17:53: I'm sure there are lot more things you can benefit from The nice thing.

00:17:57: You only pay once, and then it's public.

00:18:00: And so you can reuse it again and again?

00:18:03: Yeah exactly there is a lot of interesting work out there.

00:18:06: I think what i've been really impressed by the willingness of industry to sponsor a lot off work that gets made open source... ...and I think some people are often surprised how much open-source work is sponsored from industry.

00:18:20: sometimes The industry are keeping everything closed and hidden.

00:18:25: And actually often a lot of the methodological work is in their industries interest to make that open source, to improve the quality over it an improve the robustness.

00:18:34: because they're not trying to protect IP of statistical analysis They try into protect the IP off there product on.

00:18:39: industries often actually really pro making some things open source which is great?

00:18:44: In the end if you want I have something set is understood and believed by your stakeholders so by the payers and the decision-makers on the HDA side.

00:18:55: And having something that is open source, which can be critiqued by everybody... ...is much easier than something with a black box

00:19:03: somewhere.".

00:19:04: Yeah I couldn't agree more!

00:19:06: This was so important.

00:19:07: it's really nice to see this happening gradually.

00:19:10: From my understanding from an industry perspective there are least amount of friction going through the process?

00:19:15: The

00:19:15: better?!

00:19:16: And I think the open source and transparency on the methods reduces that friction.

00:19:22: It just makes our process smoother, so in everybody's interest to make it smoother... The IP on the modelling side probably sits more with the consultancies but industries may be slightly less concerned about that.

00:19:37: Yeah for consultancies is really kind of a mind-shift thing?

00:19:42: Certain consultancies are completely R based And then there's others that are more coming from the SAS experience.

00:19:50: So it's a completely different business model.

00:19:52: Yeah, definitely and we're definitely seeing shift in consultancies getting ready for this move.

00:19:57: so yeah We've seen ERGs.

00:20:00: obviously The universities have quite well set up on some of them teaching A lot of our unmasters programs.

00:20:04: so they were quite quick to Move.

00:20:06: and now you know?

00:20:07: You've differently seen Consultancy has taken interest because I want to make sure They're ready incase There is no demand from industry For these types of models.

00:20:14: So we've seen a lot around, in particular Shiny apps because they look very nice and their fancy.

00:20:18: This is kind of development of Shiny apps within consultancy for industry And a lot of starting to build out internal open source packaging.

00:20:27: so We do a lot review work through other consultancies So reviewing some of these internal packages that have been built which it's really good To see.

00:20:33: yeah there definitely progress There but its very mixed as you say Some consultancies are just not engaging with at all and others Are starting jump-in Because it has an additional service.

00:20:42: They can offer any value add especially around some of the sort of bolt-on stuff, around automating our reports and shiny apps.

00:20:48: And now with use of AI I think they're seeing that development time on a model build is much lower in R than Excel.

00:20:56: just because it's way that Genitivere works you can write code very efficiently.

00:21:00: whereas at Genitavere tool understanding their relationship between a cell and then header above or to side... It was complicated actually surprisingly tricky for a gen-to-vei tool to really understand what the value in this cell actually corresponds too.

00:21:16: Whereas as human, you look at it and go oh yeah that's cost four of these.

00:21:19: but often if there is missing those gaps between cells or other text included then I need to ignore them because they don't relate to us whereas obviously with Gen-To-Vare Tool its much easier to misinterpret something not have full context than just going implicitly.

00:21:34: Yeah, Gen AI tools were written by people who are really good on programming.

00:21:39: That's

00:21:39: true as well!

00:21:40: You'll never touch it so...

00:21:42: They're trained on essentially GitHub which is basically a load of open source coding for tasks like building out websites or doing analyses and visualizations from data.

00:21:53: they are incredibly good because that what their trained on there not trained on.

00:21:57: There aren't many health economic model Excel models in the public literature that they're being trained on, like.

00:22:04: it's just not something we use for.

00:22:06: And probably the developers have said why would I code something in Excel?

00:22:11: That is what...

00:22:13: Yeah exactly!

00:22:15: But yeah.

00:22:15: so we've really seen and noticed over the last couple of years.

00:22:19: So we run these courses.

00:22:19: as i mentioned We have a large number of delegates going through those courses every year.

00:22:25: When we first started running them A lot of very simple questions Like..I'm learning to code I can't get this line of code to run.

00:22:32: And the response was, ah you're using the wrong type of bracket or you've missed a comma or something.

00:22:37: these basic errors we getting none of those questions now because The first thing You do is just ask chat gbt Or people have now clawed built into their IDs for coding and so nobody makes any Of Those mistakes.

00:22:48: the time for Questions.

00:22:49: Now We Have Our Very Specific Nuance Questions For Health Economics For Which These LLMs Are Not.

00:22:55: They're not Trained To Be Specifically Health Economics Modelling Experts.

00:22:59: So The

00:23:00: whole thing moves much more to the specific knowledge, domain knowledge understanding communication and it's not so much kind of in the pure execution or programming anymore I think.

00:23:16: Yeah exactly!

00:23:17: So if you think about the steps for conceptualizing your problem with a client... ...and then understanding how do take that conceptualization like conceptual model?

00:23:26: move it into a mathematical model, and then do that in to piece of code that replicates the model or builds our model out into programming language.

00:23:33: That step can now be done very well by an alarm with supervision.

00:23:38: but really this scale is how you take understanding from problem and convey them as such way computer and alarm build up.

00:23:47: I think there's people trying make jump sort of the clients ideas and map them directly into a model.

00:23:55: And that's a big ask, whereas I think having a human be at the interaction between The need for the model in the conceptualization or the model?

00:24:03: And then work with an LLM to code it.

00:24:05: i think That's very possible and that's what we're seeing.

00:24:08: development time on models fall quite rapidly.

00:24:11: But you still need that human and my concern is a Very high-end skill ,and that's the high end skills generally done by more senior members of team who are running projects And so maybe some of the day-to-day coding that was done by juniors is now being automated away, but we need those junior team members to develop their skills and become seniors.

00:24:31: So it's a real risk for us not necessarily training our next generation or generations very much reduced because we're automating out part of work they would have otherwise done while building up skills.

00:24:42: Thanks so much!

00:24:43: We had really nice ride here through all kinds different topics open source HDA, what's happening in that space is a whole kind of group of R and an HDA.

00:24:56: What's happening on that side was the trainings that are developed to regular yearly conferences, catch-ups... And also how overall field is developing which I think it really interesting upto even how Generative AI is now included as our overall workspace and where most of the value is probably it.

00:25:20: If you want to leave the listener with one key lesson, what would be it?

00:25:26: I think it depends on their background but i think being willing to get involved in some of the open source work that's going on coming to say hi just saying... What could i get involved as a small side product goes along way moving into a new interesting field.

00:25:44: And people are incredibly friendly and generous with their time in our group, so I think being willing to put your neck out there and say hi... What can i get involved?

00:25:52: This looks really cool!

00:25:53: That's appreciated.

00:25:55: People have been incredible at doing that.

00:25:57: please feel free.

00:25:58: Awesome very good thanks.

00:26:00: so much Robert.

00:26:01: you'll find all the links to what we talked about and his company, so on.

00:26:11: So just check out the effective statistician.com.