Systematic literature reviews (SLRs) are instrumental in supporting healthcare decision-making and market access. Traditionally, literature reviews require a resource-intensive process, thus highlighting the need for more efficient methodologies without compromising rigor or integrity. Over the past few years, artificial intelligence (AI) has emerged as a powerful tool that can significantly enhance efficiency in conducting reviews. AI, particularly machine learning and large language models (LLMs), has appeared as a promising tool to improve the conduct of reviews.
At EVERSANA, we have been strategizing on the use of AI at the various stages of the systematic review process:
- Search strategies: LLMs can be used to design search strategies but can pose challenges due to the inclusion of concepts of controlled vocabulary terms (i.e., MeSH and Emtree), use of Boolean operators and truncation terms.
- Deduplication: Reviews include identifying evidence from various sources and may identify duplicate citations. LLMs can be a powerful tool to identify duplicates by comparing titles, abstracts and publication details.
- Screening: Screening is a very time-consuming step of literature reviews. A researcher is typically expected to screen 400 to 1,000 citations per day to determine if studies meet inclusion criteria. At EVERSANA, we have piloted the use of LLMs on a dataset to extract PICOS criteria from 1,000 citations in 33 minutes.
- Data extraction: Extracting data using AI is challenging given the variability of reported outcomes across the studies. The success of LLM application is dependent on the use of platform, prompts, approach to parsing larger texts and the specific type of model used. It also requires a person-specific skill set (content expertise and therapeutic area knowledge) to work iteratively to achieve the desired accuracy. Based on EVERSANA’s experience, these components can have a significant effect on the accuracy and quality of the final output/product.
- Feasibility assessment: Using LLMs, at EVERSANA, we have piloted the conduct of feasibility assessments in small and large systematic reviews across different therapeutic areas with a single-human-reviewer level of accuracy of 80% to 90%.
- Meta-analysis and synthesis: EVERSANA provides world-class expertise in the conduct of indirect treatment comparisons (ITCs). In the U.S., ICER is using EVERSANA’s proprietary technology as the preferred platform for conducting ITCs to obtain comparative effectiveness data in the absence of head-to-head clinical trials.1 AI can also be leveraged to conduct analysis, automate data input, identify heterogeneity and generate forest plots. LLMs can also be used to create a report containing the results from the analysis along with an interpretation of those results.
- Reporting: AI can also be used to generate PRISMA diagrams and tables and figures. LLMs can be used to summarize evidence, create abstracts and executive summaries for a specific audience, and translate to the end user’s desired language.
One of the key aspects of conducting systematic reviews is the transparency of process. However, documenting the rationale behind AI selection can be challenging due to complex algorithms. Therefore, it is important to have transparency in:
- Reporting the use of AI
- How data was parsed
- What prompts were used
- How the process was developed
- What QC process was implemented
- How hallucinations were identified
Issues preventing the full automation of SLRs include the variability in study outputs, the contextual subtleties of textual data and the need for nuanced understanding and strategic interpretation. However, while full automation is beyond our current grasp, AI’s strength in textual data analysis and management can be utilized to accelerate components of the systematic review process.
The implementation strategy at EVERSANA involves a blend of cross-disciplinary expertise in clinical content, HEOR practice and advanced technical knowledge of AI. This multidisciplinary approach ensures that while the AI does the heavy lifting of data processing, the essential judgments and strategy development that require human expertise are meticulously managed. Our workflow integrates strict quality control measures and transparency in every step, thus maintaining the trust and integrity essential in systematic reviews.
While the total automation of SLRs remains a vision for the future, the adoption of AI as a tool for expediting and enhancing aspects of the SLR process is already transforming the field. These advances will create time for more strategic insights and interpretation of the data generated. EVERSANA will continue to progress in this transformative integration, combining deep domain expertise with AI to deliver superior, efficient and accurate systematic reviews.
For more details on the application of AI in SLRs and ITCs, please contact the EVERSANA team.
Author
As an accomplished professional in the healthcare industry, Deepika Thakur brings a wealth of experience and expertise to the table. With over a decade of involvement, spanning roles in both industry and healthcare economic…
Tim is an experienced analyst and emerging thought leader in the application of Bayesian methods to complex problems. His Vanier Canada Graduate Funded dissertation research focused on the incorporation of multivariate evidence synthesis in…
As Senior Director of the Value & Evidence team at EVERSANA, Imtiaz leads evidence synthesis projects that support global HEOR initiatives involving systematic literature reviews, indirect treatment comparisons, and health economic modeling, to support reimbursement…