Last month, CLOSER held a workshop at the University of Manchester that brought together speakers and participants from diverse disciplinary backgrounds to share and discuss their experiences of undertaking cross-study research. The event offered us an opportunity to identify key considerations of relevance to all researchers planning, undertaking and disseminating outputs from cross-study investigations.
Different disciplines, common challenges
As a consortium, CLOSER includes studies that cover both the social science and biomedical spectrums of research, and the invited talks at this workshop reflected that broad scope. These different disciplines may have distinctive characteristics, but when it comes to performing cross-study research, there are also common challenges that transcend disciplinary boundaries. It might seem that data harmonisation is more straightforward for some topics, especially in the biomedical area, but this over-simplifies the considerations required in ensuring data collected by different clinical instruments or in different settings can be meaningfully compared and consolidated.
Two different MRI scanners for example cannot be assumed to provide identical measurements, just as two measures of socio-economic status may differ in how they capture information about respondents. Similarly, divergences may arise within the same instruments over time due to ageing parts or changing assessment contexts, and this is especially pertinent to longitudinal data. Standardisation and calibration were identified as issues of cross-disciplinary importance in the workshop discussions. By working collaboratively, consortiums such as CLOSER are helping to identify and promote solutions to such shared challenges.
Strategising for success
For any topic area, cross-study research is a difficult endeavour, and researchers may encounter pitfalls at each step along the way. During practical sessions in the workshop led by Prof Rebecca Hardy, we explored how the data harmonisation stage of cross-study work can be particularly difficult. As we seek to make the different data sources more equivalent, we run the risk of losing informative detail, given that we are likely restricting our focus to the points of cross-study overlap, i.e. the ‘lowest common denominator’.
There is also a need to take into account issues around missing data and to consider the importance of sensitivity analyses and validation exercises. Strategic approaches such as that advocated in the Maelstrom Research Guidelines can help researchers best prepare for the hurdles they can face when bringing together data from different studies.
Importance of documentation
Access to accurate and comprehensive documentation about study data is a necessary precursor to effective cross-study research. It enables us to establish what data are available and amenable to inclusion in our work. Understanding the way in which the data were originally collected is also important as it ensures we use it appropriately. During Dr Vasiliki Bountziouka’s talk at the workshop, she recounted the mixture of experiences she had in sourcing metadata information from the different studies she was utilising in her work. Recognising that this is not an uncommon issue, CLOSER has developed a metadata repository, CLOSER Discovery, to help researchers locate and explore detailed descriptions of the data available across a number of UK longitudinal studies.
Documentation is also relevant when it comes to our own processing of data. For example, when we harmonise data, we are making a series of decisions about the comparability of particular variables and values, about which cases can be included, and about the way in which the data should be prepared and processed. Knowledge about these decisions is essential to ensuring the harmonised data outputs we then go on to create are appropriately used, particularly where these datasets are made available to other researchers. This is why CLOSER is providing detailed user guides and code files alongside its harmonised data outputs, to ensure the data are properly handled and interpreted in later research usage. These are made available via the CLOSER series page on the UK Data Service. New cross-study datasets will be added over the coming months.
No ‘one size fits all’ solution
A further important point highlighted in the workshop was how there is no catch-all solution to resolving measurement differences, either between studies or over time. The data selected, the harmonisation method used and the format adopted for data outputs should instead each be adaptable to the scientific question of interest and the types of original data available. Similarly, we cannot assume a measure harmonised for one purpose is always suited to addressing another research question we might have. Reusing any data appropriately requires that we must first familiarise ourselves with how the data have been originally collected and any subsequent processing applied, again highlighting the key role played by data documentation.
In their talks at the workshop, Dr Natasha Wood and Dr Graciela Muniz-Terrera both discussed examples where it had not been possible to make data from different studies exactly equivalent, either due to measurement differences or data access restrictions. Both speakers illustrated why it is useful to distinguish between cross-study similarity in the measurement approach used and in the underlying concepts being assessed.
Cross-study investigations can be complex undertakings, and there is no simple solution to the challenges researchers can encounter in such work. However, through shared learning and discussion, we can continue to build a better understanding of these hurdles and develop improved tools and processes to facilitate and encourage these important and fruitful avenues of inquiry.
Further information
The slides from each of the presentations at the workshop are available on the event page. This workshop is part of a wider engagement and training programme at CLOSER, and new events are regularly advertised on our events page.
You can also find out more about CLOSER’s harmonisation work.
Dr Dara O’Neill is a Research Associate for CLOSER, the home of longitudinal research.
Suggested citation:
O’Neill D (2018) “Cross-study research: Overcoming obstacles, uncovering opportunity”, CLOSER blog.
