On the Painterly Depiction of Materials: An Interdisciplinary Study on the Depiction and Perception of Materials within Paintings

The world around us is filled with materials. Our ability of visual material perception informs us how to navigate and interact with our environment. It tells us, for example, whether food is fresh, if a chair is strong enough to sit on, how much force to use to pick up a glass, etc. Painters have studied how to depict the world and the materials therein for thousands of years. We believe that the material depictions within paintings can be leveraged into insights for the scientific understanding of material perception. In this thesis, we studied the perception of painterly depictions of materials and aimed to make the study thereof accessible to other researchers with the release of the Materials In Paintings dataset. We collected a large set of paintings from museums and galleries. Then, we used an online crowd-sourcing approach to annotate material identity (fabrics, stone, etc.,) and gather spatial material segmentations (i.e., “cutting out” piece of the painting that depict the material). In the first study, we measured the perception of material attributes (soft, rough, fragile, etc.,) across a range of materials and found that painterly materials trigger distinct distributions of perceived attributes and we furthermore compared these distributions to those for photographic materials. In the second study, we continued crowd-sourcing annotations on material identity and material segmentations and combined these into the Materials In Paintings dataset. In a number of cross-disciplinary demonstrations we presented novel findings across art history, human perception, and computer vision. While these demonstrations are useful in their own right, the main focus here was the release of the dataset. Next, we used the dataset as a source of stimuli for two studies into specific materials. First, for fabrics, we studied the perception of satin and velvet and the effect of presenting only local or, both local and global information, and found that the perceptual distinction between these two fabrics becomes more ambiguous when removing global information. Furthermore, we showed that local image cues can affect perceptual responses for shininess but not for softness. Lastly, we studied the perception and depiction of pearls by identifying three image features that might trigger the perception of pearliness. In a series of experiments, we confirm the role of these image features but find that the presence of only one of these image features, highlights, is already sufficient for naive participants to trigger the perception of pearliness. Conversely, expert participants (art historians or pearl experts) perceive depictions with all three features as more pearly, which implies the existence of visual expertise for pearl perception. All in all, in this thesis we show the benefits of studying material perception through painterly depictions of materials and enable further study with the release of the MIP dataset.