Counting Visitors in Museums
Finding a reliable way to count visitors is an ongoing puzzle for museums. As part of our mission to create meaningful experiences, we embarked on a research journey into the technology available for counting museum visitors. We discovered several potential off-the-shelf solutions employing technology like millimeter wave detection, infrared sensors, laser beam breakers and machine vision. In each case, we encountered a variety of trade-offs which ultimately led us to do what any level-headed development team would do: build our own. And thus, Count Trackula, our bleeding-edge solution arose from morbid curiosity.
The Case for a Custom Solution
Off-the-shelf solutions seemed promising but didn’t quite suit our needs. For instance, infrared sensors and beam breakers generally require precise installation and struggle with simultaneous crossings, often missing visitors entirely. Additionally, these systems are often installed in plain sight, making them susceptible to tampering or interference. Systems that use machine vision technology had the potential to be more accurate at distinguishing people, but we had trust issues with many of the products in this space. We wanted a solution that guaranteed no sensitive data or images were stored or shared. Millimeter wave technology, while accurate and private, is often encumbered by proprietary software with unclear pricing structures.
In the end, we opted to try Apple’s built-in computer vision models in iOS for person detection and tracking. Using an iPhone saved us the headache of needing to assemble our own hardware solution and, given its capabilities, a refurbished 12 mini is relatively affordable. Best of all, an app-based solution lends itself to being accessible to others. All of this gave us complete control to hone the accuracy, privacy, and adaptability of our solution.
Embracing a Vision
Through our development journey, we uncovered valuable insights that shaped the functionality and practicality of Count Trackula. While there is still room for improvement, here’s a breakdown of our current technical approach and some of the things we learned:
The app captures video frames as rapidly as the iPhone processor allows, sending each frame to an ML-based body detection model via VNDetectHumanRectanglesRequest. We found body detection to be more practical than other requests that consider more granular features like hands, faces, or poses.
Detection and Tracking
Once bodies are detected in a frame, the app assigns them a random ID and stores their last known position in UV space. Our intended implementation expects the iPhone to be installed out of reach to avoid tampering. The least invasive location for this is above a door, positioning the camera at an overhead angle looking down at the floor. This placement simplifies counting by reducing issues like overlapping bodies. However, it introduces a drawback: the detection model struggles to recognize bodies directly underneath the camera, as the top of a head provides minimal detail for the model to predict.
We figured out that using two models—one for detection and another for tracking—helped address this limitation. VNSequenceRequestHandler excels at interpolating changes to general subjects between frames, so we prioritize it in regions where detection might struggle, such as directly underneath the camera. Conversely, we place higher trust in the detection model at the edges of the frame, where more of a person’s body is likely to be visible due to the camera angle.
Recognition
There is one final piece of this puzzle to solve: recognition. To be clear, we are not interested in “identifying” individuals. We define “recognition” as the ability to ensure that one person is consistently treated as the same person across consecutive frames. Apple’s detection and tracking APIs are not inherently capable of doing this. In fact, the detection API treats each frame as if it contains entirely new subjects, and while the tracking API maintains a lock on a subject between frames, it cannot guarantee that lock and, in our experience, often drifts. In either case, our implementation requires frequent detection cycles to ensure new people are discovered, which means we need a way to determine who is new and who was already there.
After testing a variety of techniques for this problem, we determined the simplest solution was best. During each detection cycle, we compare the positions of each newly detected subject to previously stored positions. By calculating distances between new and old subjects, we can decide which detections belong to the same subject. To prevent errors such as “jumping” subjects, which can occur during coinciding entrances and exits, we limit movement to a maximum distance for matching. We also configure a threshold for when to stop comparing the position of someone who has likely left the scene. While this technique introduces some subjectivity, it has generally proven effective and performant.
Analytics
Our final step involves capturing that actual visitor data. Sticky Culture uses a custom Firestore database to collect statistics in all of our interactive experiences. In the past, we used Google Analytics and tried various other vendor-based solutions, but the result always felt mismatched for our needs. Our custom schema is simple and flexible, enabling us to collect exactly the data we need without having to stretch it around irrelevant concepts like campaigns or conversions. Since we are only concerned with determining the number of visitors for a particular exhibit, we simply transmit the time when one of two actions occurred: an entrance or an exit.
Looking Ahead
The prototype for Count Trackula is already in action at the Portland Exhibit at the Oregon Historical Society. This exhibit, featuring a single entry point and three interactive stations, serves as an ideal testing ground. Admittedly, there is room for improvement, but the initial results have still provided us with valuable insight that complements our standard interaction analytics. As Count Trackula continues to evolve, we look forward to sharing our progress and lessons learned.