CONCEPT

Systematic reviews (SR) and meta-analyses (MA) are tools to synthesize evidence and provide decision makers with estimates of effect that are more precise than those provided by individual studies. SRMAs inform decision makers about the certainty in these estimates to allow tradeoffs and inform shared decision making. SRMAs also support clinical practice guidelines (CPGs) and Institute of Medicine (IOM) and Guideline International Network (G-I-N) require an updated systematic review as one of the criteria for trustworthy clinical practice guidelines (CPGs).

There are two major limitations in the current approach to evidence synthesis. The process of creating SRMAs is cumbersome and slow and the presentation of results using conventional static tables and figures limits the depth of information that can be informative to clinicians and evidence users. Consequence is an epidemic of redundant, conflicted systematic reviews and meta-analyses (SRMAs). In areas with rapidly moving evidence, many SRMAs are outdated as soon as they are published. Usually, there is little incentive for original team to undertake the laborious updating process and hence a completely new time will try to create (update) an SRMA from the scratch. Often, there are just too many of redundant, overlapping systematic reviews.

Some examples are atrocious, such as patent foramen ovale (PFO) closure management example; where there are dozens meta-analyses synthesizing the information from four published randomized clinical trials. This duplication of effort is not only wasteful but often results in conflicted findings due to subtle differences in design or analysis strategy.

Living Systematic Reviews—which are continually updated, incorporating relevant new evidence as it becomes available—has been suggested as a solution to address the challenge of synthesizing evidence in fields with rapidly moving evidence. Leading journals such as Annals and BMJ have welcomed this approach where authors commit to frequent updates on accepted systematic reviews. However, without a framework supported by advanced programming and Artificial intelligence, the approach of living systematic reviews is not “truly” living and merely represents an effort of undertaking a conventional systematic review with a commitment to frequent dates. While the latter is a step in the right direction but only reflects a part of the solution. Finally, SRMAs often have multiple tables, figures and analyses hidden in the supplemental materials with no user-friendly access. This alarming increase in wasteful efforts for SRMAs of minimal value warrants reconsideration of methods, production, and reporting of SRMAs. Hence, we propose a Living Interactive Evidence Synthesis (LIvE) framework as an approach to create Living interactive Systematic Reviews (LISRs).

Living Interactive Evidence Synthesis Workflow


LIvE FRAMEWORK

To create living, interactive systematic reviews (LISRs), we built a living interactive evidence synthesis (LIvE) platform, which is undergoing constant updates to fully implement LIvE framework. Each LISR created using LIvE platform is linked to an independent webpage which is automatically updated as new data is added or new changes are implemented. As shown in the figure, LIvE framework consists of five major components to cover the entire process of living systematic reviews. To complete a LISR, the five components (i.e.., the automated search, the scanner, the extractor, the analyzer, and the tabulator) enable the raw data collection, screening, information extraction, data analysis, and interactive visualization of the analysis results, respectively. Moreover, LIvE framework has three pathways to further improve the flexibility and efficiency: conventional pathway, semi-automated human-in-the-loop pathway, and artificial intelligence (AI) powered pathway. Each pathway shares the same data structure to be compatible with different technical implementations and interchangeability, while has its own characteristics. The data can enter or leave the pipeline at each component and move across pathways allowing the flexible use of platform for specific tasks such as data-analysis or creating summary of findings (SoF) tables. The conventional pathway allows the data collection outside the LIvE platform and subsequently the structured data can be uploaded to the platform to maintain the LISR. The human-in-the-loop pathway allows the process to be fully completed in LIvE platform and facilitates the process by automated execution of a defined search strategy, web-assisted rule-based screening of new citations and data extraction on an interactive graphical user interface and automated data analysis. The AI-powered pathway, which is currently under development, will allow near-automation of this process including screening, data extraction and analysis. A key strength of LIvE platform is the interactive features (e.g., interactive PRISMA, interactive table of results, dynamic pairwise and network meta-analysis output, and SoF tables) regardless of the pathway used to maintain LISR in LIvE platform.

Human-in-the-loop
pathway
Human-in-the-loop...
Human-in-the-loop
pathway
Human-in-the-loop...
AI-powered
pathway
AI-powered...
THE WATCHER
THE WATCHER
THE EXTRACTOR
THE EXTRACTOR
NLP-based
term extraction
NLP-based...
AI-powered
pathway
AI-powered...
Manually extracted result upload
Manually extracte...
THE ANALYZER
THE ANALYZER
Automated network meta-analysis
Automated networ...
Automated pairwise
meta-analysis
Automated pairwi...
THE TABULATOR
THE TABULATOR
Interactive NMA summary of findings
table
Interactive NMA sum...
Interactive PWMA summary of findings
table
Interactive PWMA su...
Study receiver
Study receiver
Search engines
Ovid
Search engines...
Central repository
Central reposit...
THE SCANNER
THE SCANNER
System Infrastructure
System Infrastructure
Output
Output
Living Interactive PRISMA
Living Interactive PRISMA
Interactive table
Interactive table
Interactive plots and figures
Interactive plots and fi...
Interactive evidence profiles
Interactive evidence p...
Living search
Living search
Study collector
Study collector
Search engines
PubMed
Search engines...
Auto Push
Auto Push
Auto Pull
Auto Pull
Categorize and merge
Categorize and merge
Search strategy
Search st...
Summary PRISMA result upload
Summary PRISMA re...
ML-based
RCT identification
ML-based...
Rule-based screening
Rule-based screen...
Interactive graphical user interface
Interactive graph...
ML-based
study classification
ML-based...
Interactive graphical user interface
Interactive graph...
Rule-based extraction
Rule-based extrac...
Conventional
pathway
Conventional...
Conventional
pathway
Conventional...
scikit-learn
scikit-learn
netmeta
netmeta
requests
requests
BUGSnet
BUGSnet
meta
meta
MedTagger
MedTagger
R meta-analysis packages
R meta-analysis packages
NLP software and platforms
NLP software and platforms
MetaMap
MetaMap
Python data analysis and
machine learning packages
Python data analysis and...
tensorflow
tensorflow
pandas
pandas
NumPy
NumPy
Python network and
data collection packages
Python network and...
scrapy
scrapy
JavaScript visualization packages
JavaScript visualization packages
Plotly
Plotly
D3.js
D3.js
Vue.js
Vue.js
Data storage
Data storage
MySQL database
MySQL database
SciPy
SciPy
Matplotlib
Matplotlib
NLTK
NLTK
spaCy
spaCy
Python NLP packages
Python NLP packages
dmetar
dmetar
geMTC
geMTC
THE DECISION AID
THE DECISION AID
Interactive decision guidance
Interactive decision g...
JavaScript visualization packages
JavaScript visualization packages
Plotly
Plotly
D3.js
D3.js
Vue.js
Vue.js
Mixed comparative decision aids
Mixed comparative d...
Direct comparative decision aids
Direct comparative...
Text is not SVG - cannot display

SYSTEM ARCHITECTURE

To implement our proposed LIvE framework, we built the LIvE platform in a five-layer architecture, which includes: application layer, shared module layer, core service layer, middleware layer, and storage layer respectively. The modules in each layer decomposes the requests from upper layer and aggregates the responses from lower layer to complete specific tasks. In the application layer, there are 6 applications to provide the graphical user interface (GUI) to different users. Each application is designed for a specific task, such as monitoring the latest updates in studies (the watcher), screening studies for inclusion or exclusion and extracting data on or further analysis using web-assisted rule-based annotations implement through interactive graphical user interface (the scanner and the extractor), conducting the meta-analysis (pairwise or network meta-analysis) on selected studies (the analyzer). To support these functions, applications are built upon customized GUI modules and third-party frontend packages, which are listed in the shared module layer; In the core service layer, the meta-analysis application programming interface (API) receives the user operations from the modules in each application. Those user operations are forwarded to specific service modules, which are used to execute commands or packages in the middleware layer. For example, the outcome analysis service in the project management converts the requests from the modules in data analyzer to R script by Python network packages, then this service gets the results of the R script and sends the results back to users; In the storage layer, all the data used in the system, including the structured meta data of projects and studies, semi-structured user annotation data, and free-text data of study PDF files, are formatted as defined in our meta-analysis data standard and saved in different place according to its characteristics.

Public Website
Public Website
User Portal
User Portal
Watcher
Watcher
Scanner
Scanner
Extractor
Extractor
Analyzer
Analyzer
Tabulator
Tabulator
Decision Aid
Decision Aid
Living Evidence Synthesis Core Applications
Living Evidence Synthesis Core Applications
Research team
Research team
Public users (clinicians & patients)
Public users (clinic...
Application Layer
Application Layer
Summary Table
Summary Table
PRISMA
PRISMA
PWMA Forest Plots
PWMA Forest P...
NMA Forest Plots
NMA Forest Pl...
NMA Rank Plots
NMA Rank P...
League
Table
League...
SoF
Table
SoF...
Evidence Map
Evi...
Decision
Aid
Decision...
Project Visualization Modules
Project Visualization Modules
Third Party Frontend Packages
Third Party Frontend Packages
Core Service Layer
Core Service Layer
Core Service Application Programming Interface
Core Service Application Programming Interface
Eligibility Criteria Management
Eligibility Criteria Management
Tags Management
Tags Management
Project Settings Services
Project Settings Ser...
Study Meta Data Services
Study Meta Data Services
EndNote/OVID/PubMed XML Parsing
EndNote/OVID/PubMed XML P...
Study Meta Data Fixing
Study Meta Data Fixing
PubMed E-Utilities API
PubMed E-Utilities API
Study Batch Import
Study Batch Import
XML/XLS/CSV Import
XML/XLS/CSV Import
Clinical Trial Data Update
Clinical Trial Data Update
Middleware Layer
Middleware Layer
Web Programming
Web Programming
SQLAlchemy
SQLAlchemy
Flask
Flask
Werkzeug
Werkzeug
Requests
Requests
Python Packages
Python Packages
Frequentist PWMA
Frequentist PWMA
meta
meta
metafor
metafor
R Packages
R Packages
bayesmeta
bayesmeta
Bayesian PWMA
Bayesian PWMA
SciPy
SciPy
rpy2
rpy2
NumPy
NumPy
Pandas
Pandas
Frequentist NMA
Frequentist NMA
netmeta
netmeta
gemtc
gemtc
BUGsnet
BUGsnet
Bayesian NMA
Bayesian NMA
dmetar
dmetar
Third-Party Software
Third-Party Software
Nginx
Nginx
Storage Layer
Storage Layer
Project & Study Metadata
Project & Study Meta...
Scanner
Annotation Data
Scanner...
Extractor Annotation Data
Extractor Annotation...
PDF Files and
Raw Text Data
PDF Files and...
System/User-generated Data
System/User-generate...
PRISMA
PRISMA
SoF Table
SoF Table
Summary Table
Summary Table
Evidence Map
Evidence Map
PWMA & NMA Results
PWMA & NMA Results
Decision Aid
Decision Aid
Project Management
Project Managem...
Projects List
Projects List
Project Editor
Project Editor
OVID Email Alert Watcher
OVID Email Alert W...
PubMed Search Watcher
PubMed Search Watc...
Study Screening Decision
Study Screening De...
Study Labelling
Study Labelling
Study Import/Export
Study Import/Export
Outcome Meta Data Editor
Outcome Meta Data...
Extraction Interface
Extraction Interfa...
Extraction Statistics
Extraction Statist...
Primary, Sensitivity & Subgroup
Primary, Sensitivity...
Freq. & Bayes Network MA
Freq. & Bayes Netw...
Pairwise MA
Pairwise MA
Evidence Map
Evidence Map
PWMA & NMA SoF Table
PWMA & NMA SoF Tab...
Absolute Effects
Absolute Effects
Mixed Comparative Evidence
Mixed Comparative...
Direct Comparative Evidence
Direct Comparative...
Project Interactive Modules
Project Interactive Modules
Dynamic Filters
Dynamic Filters
Categorized Outcomes Tree List
Categorized Outcomes...
Keywords Highlight
Keywords Hig...
Vue.js
Vue.js
D3.js
D3.js
ECharts
ECharts
jQuery
jQuery
AlaSQL
AlaSQL
Bootstrap
Bootstrap
datatables.js
datatables.js
PDF.js
PDF.js
Vuetify.js
Vuetify.js
Data Extraction Auto Hint
Data Extraction...
Frontend Indexing and Searching
Frontend Indexing an...
Dynamic Relative Effects Calculation
Dynamic Relative Eff...
Core Service Agents
Core Service Age...
Dynamic Context Menu
Dynamic Context...
Metadata Management
Metadata Management
Decision Management
Decision Management
Conditional Study Selection
Conditional Study Select...
Screening Service
Screening Se...
Study Stage Statistics
Study Stage Statistics
Meta Data Extraction Management
Meta Data Extraction Man...
Study Extraction Mapping & Binding
Study Extraction Mapping...
Extraction Services
Extraction S...
Outcome Meta Data Management
Outcome Meta Data Manage...
PDFs Upload and Download
PDFs Upload and Download
PDF Meta Data Management
PDF Meta Data Management
PDF File Services
PDF File Ser...
PDFs Keywords Highlight and Search
PDFs Keywords Highlight...
ML & NLP Services
ML & NLP Services
Medical Term Detection
Medical Term Detection
Text Classification
Text Classification
RCT Detection
RCT Detection
NCT Number Extraction
NCT Number Extraction
Publication Date Detection
Publication Date Detection
Backend Administrator CLI
Backend Administrator CLI
Static Website Management
Static Website Management
Backend System Services
Backend System Servi...
Meta-Analysis Services
Meta-Analysis Services
Cumulative/Subgroup Analysis
Cumulative/Subgroup Analys...
Frequentist Network
Meta-Analysis
Frequentist Network...
Sensitivity Analysis
Sensitivity Analysis
Bayesian Network
Meta-Analysis
Bayesian Network...
R-script Generation
R-script Generation
Results Data Format Conversion
Results Data Format Conver...
Pairwise Meta-Analysis
Pairwise Meta-Analysis
Network Meta-Analysis
Network Meta-Analysis
Adapters
Adapters
Data Processing
Data Processing
TensorFlow
TensorFlow
PyTorch
PyTorch
scikit-learn
scikit-learn
Transformers
Transformers
Machine Learning
Machine Learning
Gunicorn
Gunicorn
Unified Meta-Analysis
Data Model
Unified Meta-Analy...
Docker
Docker
MySQL
MySQL
Named Entity
Recognition
Named Entity...
Email Update Management
Email Update Management
Text is not SVG - cannot display