skip to main content
10.1145/2601248.2601268acmotherconferencesArticle/Chapter ViewAbstractPublication PageseaseConference Proceedingsconference-collections
research-article

Guidelines for snowballing in systematic literature studies and a replication in software engineering

Published: 13 May 2014 Publication History

Abstract

Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Objective: This paper presents guidelines for conducting literature reviews using a snowballing approach, and they are illustrated and evaluated by replicating a published systematic literature review.
Method: The guidelines are based on the experience from conducting several systematic literature reviews and experimenting with different approaches.
Results: The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.
Conclusions: It is concluded that using snowballing, as a first search strategy, may very well be a good alternative to the use of database searches.

References

[1]
Kitchenham, B. A., Dybå, T. and Jørgensen, M. 2004. Evidence-based software engineering. In Proceedings of 27th IEEE International Software Engineering Conference, 273--281, IEEE Computer Society, 2004.
[2]
Webster, J. and Watson, R. T. 2002. Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly 26, 2, xiii--xxiii.
[3]
Pickard, L., Kitchenham B. A. and Jones P. 1998. Combining empirical results in software engineering. Information & Software Technology 40, 14, 811--821.
[4]
Miller, J. 1999. Can results from software engineering experiments be safely combined?. In Proceedings IEEE 6th International Symposium on Software Metrics, 152--158.
[5]
Hayes, W. 1999. Research synthesis in software engineering: A case for meta-analysis. In Proceedings 6th IEEE International Software Metrics Symposium, 143--151.
[6]
Basili V. R., Shull F. and Lanubile F. 1999. Building knowledge through families of experiments. IEEE Transactions on Software Engineering 25, 4, 456--473.
[7]
Kitchenham B. A. and Charters S. 2007. Guidelines for performing systematic literature reviews in software engineering. Version 2.3, EBSE Technical Report, EBSE-2007-01, Keele University.
[8]
Petersen K., Feldt R., Mujtaba S. and Mattsson M. 2008. Systematic mapping studies in software engineering. In Proceedings 12th International Conference on Evaluation and Assessment in Software Engineering.
[9]
Kitchenham B. A., Budgen, D. and Brereton, O. P. 2011. Using mapping studies as the basis for further research -- a participant-observer case study. Information and Software Technology 53, 6, 638--651.
[10]
Wohlin C. and Prikladnicki R. 2013. Systematic literature reviews in software engineering. Information and Software Technology 55, 6, 919--920.
[11]
MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676--687.
[12]
Wohlin, C., Runeson, P., da Mota Silveira Neto, P. A., Engström, E., do Carmo Machado, I. and de Almeida, E. S. 2013. On the reliability of mapping studies in software engineering. Journal of Systems and Software 86, 10, 2594--2610.
[13]
Kitchenham, B. A., Brereton, O. P., Budgen, D., Turner, M., Bailey, J. and Linkman, S. 2009. Systematic literature reviews in software engineering: A systematic literature review. Information and Software Technology 51, 1, 7--15.
[14]
Kitchenham, B. A., Pretorius, R., Budgen, D., Brereton, O. P., Turner, M., Niazi, M. and Linkman, S. 2010. Systematic literature reviews in software engineering: A tertiary study. Information and Software Technology 52, 8, 792--805.
[15]
Greenhalgh T. and Peacock, R. 2005. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: Audit of primary sources. BMJ 331, 7524, 1064--1065.
[16]
Skoglund, M. and Runeson, P. 2009. Reference-based search strategies in systematic reviews. In Proceedings 13th Evaluation and Assessment in Software Engineering, 31-40.
[17]
Kitchenham, B. A., Brereton, O. P., Li, Z., Budgen, D. and Burn, A. 2011. Repeatability of systematic literature reviews. In Proceedings of the 15th International Conference on Evaluation and Assessment in Software Engineering, 46--55.
[18]
Zhang, H., Babar, M. A. and Tell, P. 2011. Identifying relevant studies in software engineering. Information and Software Technology 53, 6, 625--637.
[19]
Jalali, S. and Wohlin, C. 2012. Systematic literature studies: Database searches vs. backward snowballing. In Proceedings 6th International Symposium on Empirical Software Engineering and Measurement, 29--38.
[20]
Jalali, S. and Wohlin, C. 2012. Global software engineering and agile practices: A systematic review. Journal of Software: Evolution and Process 24, 6, 643--659.
[21]
Smite, D., Wohlin, C., Galvina, Z. and Prikladnicki, R. 2012. An empirically based terminology and taxonomy for global software engineering. Empirical Software Engineering: An International Journal 19, 1, 105--153.

Cited By

View all
  • (2025)A Systematic Review of Factors Contributing to Ineffective Cultural Heritage ManagementSustainability10.3390/su1701036617:1(366)Online publication date: 6-Jan-2025
  • (2025)Efficacy of intermittent fasting on improving liver function in individuals with metabolic disorders: a systematic review and meta-analysisNutrition & Metabolism10.1186/s12986-024-00885-x22:1Online publication date: 6-Jan-2025
  • (2025)A Survey on Online Aggression: Content Detection and Behavioural Analysis on Social Media PlatformsACM Computing Surveys10.1145/3711125Online publication date: 4-Jan-2025
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
May 2014
486 pages
ISBN:9781450324762
DOI:10.1145/2601248
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

  • Brunel University: Brunel University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2014

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. replication
  2. snowball search
  3. snowballing
  4. systematic literature review
  5. systematic mapping studies

Qualifiers

  • Research-article

Funding Sources

Conference

EASE '14
Sponsor:
  • Brunel University

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)2,626
  • Downloads (Last 6 weeks)312
Reflects downloads up to 09 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2025)A Systematic Review of Factors Contributing to Ineffective Cultural Heritage ManagementSustainability10.3390/su1701036617:1(366)Online publication date: 6-Jan-2025
  • (2025)Efficacy of intermittent fasting on improving liver function in individuals with metabolic disorders: a systematic review and meta-analysisNutrition & Metabolism10.1186/s12986-024-00885-x22:1Online publication date: 6-Jan-2025
  • (2025)A Survey on Online Aggression: Content Detection and Behavioural Analysis on Social Media PlatformsACM Computing Surveys10.1145/3711125Online publication date: 4-Jan-2025
  • (2025)A Systematic Literature Review of Enterprise Architecture Evaluation MethodsACM Computing Surveys10.1145/370658257:5(1-36)Online publication date: 9-Jan-2025
  • (2025)Survey and Benchmark of Anomaly Detection in Business ProcessesIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.348415937:1(493-512)Online publication date: Jan-2025
  • (2025)Assessing the potential impacts of floating Offshore Wind Farms on policy-relevant species: A case study in the Gulf of Roses, NW MediterraneanMarine Policy10.1016/j.marpol.2024.106518172(106518)Online publication date: Feb-2025
  • (2025)A structural taxonomy for lifted software product line analysesJournal of Systems and Software10.1016/j.jss.2024.112280222(112280)Online publication date: Apr-2025
  • (2025)Software architecture-based self-adaptation in roboticsJournal of Systems and Software10.1016/j.jss.2024.112258219(112258)Online publication date: Jan-2025
  • (2025)Survey of federated learning in intrusion detectionJournal of Parallel and Distributed Computing10.1016/j.jpdc.2024.104976195:COnline publication date: 1-Jan-2025
  • (2025)Systematic mapping study on requirements engineering for regulatory compliance of software systemsInformation and Software Technology10.1016/j.infsof.2024.107622178(107622)Online publication date: Feb-2025
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media