Materials in Paintings (MIP): An interdisciplinary dataset for perception, art history, and computer vision

Mitchell J.P. van Zuijlen*, Hubert Lin, Kavita Bala, Sylvia C. Pont, Maarten W.A. Wijntjes

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

4 Citations (Scopus)
84 Downloads (Pure)


In this paper, we capture and explore the painterly depictions of materials to enable the study of depiction and perception of materials through the artists’ eye. We annotated a dataset of 19k paintings with 200k+ bounding boxes from which polygon segments were automatically extracted. Each bounding box was assigned a coarse material label (e.g., fabric) and half was also assigned a fine-grained label (e.g., velvety, silky). The dataset in its entirety is available for browsing and downloading at We demonstrate the cross-disciplinary utility of our dataset by presenting novel findings across human perception, art history and, computer vision. Our experiments include a demonstration of how painters create convincing depictions using a stylized approach. We further provide an analysis of the spatial and probabilistic distributions of materials depicted in paintings, in which we for example show that strong patterns exists for material presence and location. Furthermore, we demonstrate how paintings could be used to build more robust computer vision classifiers by learning a more perceptually relevant feature representation. Additionally, we demonstrate that training classifiers on paintings could be used to uncover hidden perceptual cues by visualizing the features used by the classifiers. We conclude that our dataset of painterly material depictions is a rich source for gaining insights into the depiction and perception of materials across multiple disciplines and hope that the release of this dataset will drive multidisciplinary research.

Original languageEnglish
Article numbere0255109
Number of pages30
JournalPLoS ONE
Issue number8 August 2021
Publication statusPublished - 2021


Dive into the research topics of 'Materials in Paintings (MIP): An interdisciplinary dataset for perception, art history, and computer vision'. Together they form a unique fingerprint.

Cite this