Skip to main content
medRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search

A pen mark is all you need - Incidental prompt injection attacks on Vision Language Models in real-life histopathology

View ORCID ProfileJan Clusmann, Stefan J. K. Schulz, Dyke Ferber, Isabella C. Wiest, Aurélie Fernandez, Markus Eckstein, Fabienne Lange, Nic G. Reitsam, Franziska Kellers, Maxime Schmitt, Peter Neidlinger, Paul-Henry Koop, Carolin V. Schneider, Daniel Truhn, Wilfried Roth, Moritz Jesinghaus, Jakob N. Kather, Sebastian Foersch
doi: https://doi.org/10.1101/2024.12.11.24318840
Jan Clusmann
1Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
2Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Jan Clusmann
Stefan J. K. Schulz
3Institute of Pathology, University Medical Center Mainz, Mainz, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Dyke Ferber
1Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
4Department of Medical Oncology, National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Isabella C. Wiest
1Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
5Department of Medicine II, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Aurélie Fernandez
3Institute of Pathology, University Medical Center Mainz, Mainz, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Markus Eckstein
6Universitätsklinikum Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Erlangen, Germany & Comprehensive Cancer Center Erlangen-EMN (CCC ER-EMN), Erlangen, Germany & Comprehensive Cancer Center Alliance WERA (CCC WERA)
7Bavarian Cancer Research Center (BZKF)
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Fabienne Lange
6Universitätsklinikum Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Erlangen, Germany & Comprehensive Cancer Center Erlangen-EMN (CCC ER-EMN), Erlangen, Germany & Comprehensive Cancer Center Alliance WERA (CCC WERA)
7Bavarian Cancer Research Center (BZKF)
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Nic G. Reitsam
1Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
7Bavarian Cancer Research Center (BZKF)
8Pathology, Faculty of Medicine, University of Augsburg, Augsburg, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Franziska Kellers
9Department of Pathology, University Hospital Schleswig-Holstein, Campus Kiel, Kiel, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Maxime Schmitt
10Institute of Pathology, Philipps-University Marburg und University Hospital Marburg, Marburg, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Peter Neidlinger
1Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Paul-Henry Koop
1Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
2Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Carolin V. Schneider
2Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Daniel Truhn
11Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Wilfried Roth
3Institute of Pathology, University Medical Center Mainz, Mainz, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Moritz Jesinghaus
10Institute of Pathology, Philipps-University Marburg und University Hospital Marburg, Marburg, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jakob N. Kather
1Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
12Department of Medicine I, University Hospital Dresden, Dresden, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Sebastian Foersch
3Institute of Pathology, University Medical Center Mainz, Mainz, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: Sebastian.foersch@unimedizin-mainz.de
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Data/Code
  • Preview PDF
Loading

Data availability

The original data (images, prompts, model outputs, ratings, summary statistics) are available in the supplementary data and supplementary information. The following publicly accessible data was used for this study: COAD-READ, the THCA, the OV, and the PRAD cohort of the TCGA, can be found at https://portal.gdc.cancer.gov/. Slides for the lymph node metastasis scenario can be found at https://www.cancerimagingarchive.net/collection/sln-breast/.

Back to top
PreviousNext
Posted December 12, 2024.
Download PDF
Data/Code
Email

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
A pen mark is all you need - Incidental prompt injection attacks on Vision Language Models in real-life histopathology
(Your Name) has forwarded a page to you from medRxiv
(Your Name) thought you would like to see this page from the medRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
A pen mark is all you need - Incidental prompt injection attacks on Vision Language Models in real-life histopathology
Jan Clusmann, Stefan J. K. Schulz, Dyke Ferber, Isabella C. Wiest, Aurélie Fernandez, Markus Eckstein, Fabienne Lange, Nic G. Reitsam, Franziska Kellers, Maxime Schmitt, Peter Neidlinger, Paul-Henry Koop, Carolin V. Schneider, Daniel Truhn, Wilfried Roth, Moritz Jesinghaus, Jakob N. Kather, Sebastian Foersch
medRxiv 2024.12.11.24318840; doi: https://doi.org/10.1101/2024.12.11.24318840
Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools
A pen mark is all you need - Incidental prompt injection attacks on Vision Language Models in real-life histopathology
Jan Clusmann, Stefan J. K. Schulz, Dyke Ferber, Isabella C. Wiest, Aurélie Fernandez, Markus Eckstein, Fabienne Lange, Nic G. Reitsam, Franziska Kellers, Maxime Schmitt, Peter Neidlinger, Paul-Henry Koop, Carolin V. Schneider, Daniel Truhn, Wilfried Roth, Moritz Jesinghaus, Jakob N. Kather, Sebastian Foersch
medRxiv 2024.12.11.24318840; doi: https://doi.org/10.1101/2024.12.11.24318840

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Pathology
Subject Areas
All Articles
  • Addiction Medicine (403)
  • Allergy and Immunology (712)
  • Anesthesia (207)
  • Cardiovascular Medicine (2969)
  • Dentistry and Oral Medicine (336)
  • Dermatology (253)
  • Emergency Medicine (445)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (1049)
  • Epidemiology (12807)
  • Forensic Medicine (12)
  • Gastroenterology (830)
  • Genetic and Genomic Medicine (4621)
  • Geriatric Medicine (423)
  • Health Economics (732)
  • Health Informatics (2941)
  • Health Policy (1073)
  • Health Systems and Quality Improvement (1092)
  • Hematology (393)
  • HIV/AIDS (932)
  • Infectious Diseases (except HIV/AIDS) (14143)
  • Intensive Care and Critical Care Medicine (853)
  • Medical Education (430)
  • Medical Ethics (116)
  • Nephrology (475)
  • Neurology (4408)
  • Nursing (238)
  • Nutrition (649)
  • Obstetrics and Gynecology (817)
  • Occupational and Environmental Health (739)
  • Oncology (2295)
  • Ophthalmology (652)
  • Orthopedics (260)
  • Otolaryngology (327)
  • Pain Medicine (281)
  • Palliative Medicine (84)
  • Pathology (502)
  • Pediatrics (1200)
  • Pharmacology and Therapeutics (509)
  • Primary Care Research (502)
  • Psychiatry and Clinical Psychology (3799)
  • Public and Global Health (7005)
  • Radiology and Imaging (1544)
  • Rehabilitation Medicine and Physical Therapy (920)
  • Respiratory Medicine (921)
  • Rheumatology (444)
  • Sexual and Reproductive Health (445)
  • Sports Medicine (386)
  • Surgery (491)
  • Toxicology (60)
  • Transplantation (212)
  • Urology (182)