Literature searches performed for use in a systematic review should be peer- reviewed by someone outside the research group. An initial scrutiny may be carried out with the support of checklists. Important for assessment is:
A more detailed run-through of this process can be found in the article PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement.
A phenomenon that may influence search results is the cognitive bias of the searcher. By paying attention to this issue, results will be improved and the risk for skewed, slanted or inaccurate results is reduced.
Some examples of bias to be aware of:
Confirmation bias - mostly using information already known, thereby confirming and strengthening knowledge and opinions already held,
Availability - mostly using tools and resources that are easily available and most well-known,
Focusing – sometimes, focusing on one aspect or minor part of the research question may be relevant, but all parts of a question must get a fair weight in the search process,
Anchoring - paying too much attention to early results, hindering the development of new thoughts during the search process,
Substitution - understanding of a complex question only in general terms may be a cause of simplifying and incorrect reasoning which will not catch the full extent of the research problem,
Too much trust in the tools - putting too much uncritical trust in functions and techniques of databases may be risky, e.g. in the use of limitation alternatives provided by the interface of a database,
Search satisficing – belief that everything has been retrieved and explaining the search complete before this is so,.
Over-confidence in the capabilities of others - the detailed knowledge of experts may be difficult to communicate to colleagues within other disciplines, risking misunderstandings,
Over-confidence in one's own capabilities - an all-too-strong trust in the capacity of the own group may cause reluctance to take in opinions and new knowledge from others
Strategies to reduce the risks mentioned above:
- Be aware that every search process is unique,
- Reflect upon your own thinking, be self-critical and do not rush through the process,
- Simplify without losing the essence of the subject,
- Think broadly in looking for alternative search terms, tools and methods,
- Keep notes, do not trust your memory alone,
- Train and get help from others,
- Stay up-dated, tools appear and tools change,
- Look for feedback and peer review, at first within the group and later outside
(Based on Jeannine Cyr Gluck, "How searches fail: Cognitive bias in Literature Searching", https://doi.org/10.1080/15323269.2020.1702839)
Getting inspiration from the searches of others is helpful when designing your own searches but be aware of common mistakes.
A study of systematic reviews and meta-analyses in medicine showed that 97,2% had faults in their PubMed searches and in 78,1% of the cases this had affected the search results in a negative way (Salvador-Oliván, J. A., Marco-Cuenca, G., & Arquero-Avilés, R. 2019). The most common mistakes were missing terms, missing variations such as spelling and suffixes and wantage of controlled vocabulary or faulty use of them. See also: Sampson M. & McGowan J. (2006).
Even if a database may seem to be a neutral, reliable tool, it does not, of course, deliver the complete truth. Its contents and construction may influence the search results in different ways, with built in systematic imbalances, a.k.a. bias.
Varying coverage has already been mentioned. It is also relevant to note how database records are indexed, i.e. how the subjects of articles are described. Some databases have editors who actively work with key words in accordance with controlled vocabulary where a certain phenomenon is always described using the same terms. More commonly however, is that key words are picked up from the article and have been set by either the author or the journal editors. Key words are then not chosen from a set vocabulary but are freely formulated.
Database records often include information like subject area or discipline, used for filtering a search. But these categories do not usually identify the contents of the article at hand, but that of the journal in which the article is published. Some databases offer the possibility to filter on peer review. In those cases, you should remember that the principles for such scrutiny differ - to categorize journals as peer reviewed or not is not always easy. It is the categorizing and definitions set by each database provider that determine the coding.
LiU researchers and PhD students can consult a librarian, for example when doing a literature review for a dissertation or a systematic review. Together we will look at your specific research questions and discuss topics such as:
Make an appointment by completing this form: