In my last post, we examined the topics of artificial intelligence and machine learning in museums. Today, I’d like to continue this thread and focus on machine vision. It couldn’t be more timely, as Google recently announced their public beta of Cloud Vision API and it has us all dreaming of interesting ways that machine vision can be used to help museums.
What is machine vision?
Machine vision is the ability for a computer to understand what it is seeing.
Back in 2014, the Museum of Arts and Design in New York, hosted a panel examining the “Cultural Impact of Computer Vision” from the eyes of artists. Today, let’s take a look from the perspective of museums.
Machine vision can be used to inspect and analyze images. Imagine being able to classify all of your visual objects with the flip of a switch (actually, a few lines of code).
Let’s take a look at a few examples!
1. Identifying Subject Matter
Machine vision has become advanced enough to detect the subject matter and objects depicted in an image. What is depicted in this painting, photo, video, or sculpture?
We put Google Vision API to the test with Canaletto’s The Grand Canal in Venice from Palazzo Flangini to Campo San Marcuola located at the J. Paul Getty Museum in Los Angeles.
Maybe it was beginner’s luck, but the 4 terms returned (watercraft rowing, rowing, gondola, and painting) were all accurate descriptions of the subject matter and objects.
There is still a ways to go with object classification but it’s worth noting that the more you “train” a machine vision engine, the more accurate it becomes.
2. Exacting Color Composition
Color composition is one meta-tag that you are unlikely to find in most museum collections databases. Running an object’s image through a computer vision tool can extract and output data related to its color clusters, partitions, and histogram data.
3. Sentiment Analysis
If there are unobstructed human faces in an image, machine vision can be used to determine the emotional state of those portrayed by analyzing the facial characteristics.
Rembrandt (circle of), “Bust of a Laughing Young Man” (1629)
Nice to see you, too!
Pablo Picasso, “Femme aux Bras Croisés” (1901)
4. Text / Character Recognition
Want to easily extract text from every object in your collection? This has been possible for many years. The tool commonly known as “optical character recognition,” has become more accessible and faster to use via cloud APIs.
While this might not be absolutely necessary for pieces by Ed Ruscha or Lawrence Weiner (as the title and text displayed in their works are usually the same), this function’s greatest value could come from extracting text from written documents (historical letters, etc.) so that it’s all searchable and easy to classify.
5. Recognizing Similarity and Patterns
Are there other works in your collections that are very similar, not just on subject matter, but visual composition? A computer can see these relationships and quantify the differences and similarities.
These two Clyfford Still “replica” paintings are slightly different, 5.58% to be exact!
Walking into the Impressionism Gallery at the Museum of Fine Arts, Boston, you’ll find these two stunning paintings by Claude Monet, side-by-side. According to computer analysis the two works are 96.81% similar.
(L to R) Water Lilies, 1905. Oil on canvas; Water Lilies. 1907. Oil on canvas. Collection of Museum of Fine Arts, Boston
6. Art Authentication
Back in 2008, PBS NOVA covered the case of computers helping distinguish forged art from original masterpieces. This project was in cooperation with the Van Gogh Museum and challenged computer scientists to build tools to analyze brush stocks and identify forgeries.
So, where do we begin?
There are various tools available today to help you jumpstart your journey into machine vision.
- Google’s Cloud Vision API
- Microsoft Project Oxford’s Computer Vision API
- Amazon’s Rekognition
- IBM Watson’s AlchemyAPI
As you can see, computer vision is a very powerful tool and is more accessible than ever before. In the hands of museums, it can lead to interesting discoveries, rich data, and new paths into your collection.
- Exploring Artificial Intelligence at Museums
- When A Machine Learning Algorithm Studied Fine Art Paintings, It Saw Things Art Historians Had Never Noticed
- Computers can find similarities between paintings
- Augmenting optical character recognition (OCR) for improved digitization
- Label Whisperer (Cooper Hewitt Labs)