Designing a Digital Docent

Oregon Historical Society Digital Collections, June D. Drake photographs, 1863–2001, Org. Lot 678, Box 26, 059

In today’s digital age, we have unprecedented access to vast collections of historical data and cultural artifacts, yet engaging with them often feels passive and impersonal. Traditionally, museum curators and docents have brought these collections to life—sharing stories, drawing connections, and answering questions with deep, contextual knowledge. However, even the most passionate experts have limited availability and can only cover so much ground. Imagine if you could extend that personalized, knowledgeable guidance beyond such constraints—a guide that is always available and capable of tapping into every piece of information at once.

At Sticky Culture, we’re working to bring the idea of a “digital docent” to life by leveraging AI techniques to interpret vast datasets and transform static content into an engaging exploration. Our key experiment focuses on the June Drake collection at the Oregon Historical Society—a repository of over 2,500 early 20th-century photographs.

As we delved into these images ourselves, we uncovered a wealth of stories: vibrant street parades, candid snapshots of pets, seasonal traditions, and even unexpected oddities that illuminate the full spectrum of early 20th-century life. It’s clear that no single curation could capture the depth and variety of these moments—it’s a collection meant to be explored.

Guiding Exploration

Our goal is for every visitor to discover something personally meaningful within this collection. The digital docent serves as a guide, helping to navigate and connect individual interests to rich historical insights. What follows is a high-level overview of how we approached building the digital docent. There is still much to refine in this prototype and we expect the technology to change rapidly, but we wanted to share some of our initial insights.

Embedded Concepts

The foundation of this project is the meticulously curated metadata provided by the Oregon Historical Society. Their efforts to caption, date, locate, and catalogue each photograph ensure a rich and reliable dataset which provides an essential starting point for meaningful AI-driven exploration. Detailed captions provide essential context by naming individuals, events, and significant details that might otherwise be missed in the photograph.

In order to take advantage of these context clues, we embed the metadata in a vector space using the openai/clip-vit-large-patch14 transformer model. This process places the information into a shared semantic space that captures its underlying meaning and makes it searchable.

Semantic Search

When a visitor asks our digital docent a question like “What did town celebrations look like in the 20s?” we embed that question in the same CLIP vector space as the metadata. Our implementation uses FAISS for both indexing and searching. Unlike traditional searches that rely on exact keyword matches, vector embeddings enable the system to compare concepts by calculating their relative distance. For instance, “town” might connect to terms like “village” or “main street,” while “celebration” could link to “parade” or “festival.” Similarly, ”20s” may surface images reflecting the era’s fashion, architecture, or social customs.

This process is inherently subjective, influenced by the quality and training of the model. We sense that CLIP’s training favors a distinct linguistic style that doesn’t perfectly align with the June Drake dataset. Regardless, it’s an incredibly advanced open-source model that provides a solid semantic baseline for us to refine in subsequent steps.

Re-ranking

From the initial visitor question, we narrow a dataset of 2500+ photos down to 100. The initial 100 results should be, in some way, semantically related. Obviously, not all of them will be, but we’ve created a strong starting point. This is where re-ranking with a cross-encoder comes into play. For this task, we are using cross-encoder/ms-marco-MiniLM-L-6-v2.

Unlike the initial CLIP comparison, which focuses on broad semantic similarity, the cross encoder directly compares each query-result pair, scoring them based on deeper contextual relevance. This re-ranking process refines the order of results, ensuring that the most contextually appropriate images are prioritized for our language model, ultimately improving accuracy and coherence in the final narrative.

Generation

The final step brings the digital docent to life. We supply the visitor’s request, the metadata of the top photo result, and detailed instructions to OpenAI’s ChatGPT, specifically gpt-4o-2024-08-06. With explicit prompts guiding its reasoning, the LLM reviews the metadata and generates a short, engaging description of the image as it relates to the visitor’s specific interest. For example, if the visitor says:

“Show me the cutest dogs!”

Our search results return the following image, captioned with “An automobile is parked on the side of a street in Silverton, Oregon. A dog sits on the back of the car. The dog is Bobbie the Wonder Dog, a Scotch Collie who achieved nationwide fame when he found his way home to Silverton after being lost in Indiana in 1923.”

Oregon Historical Society Digital Collections, June D. Drake photographs, 1863–2001, Org. Lot 678, Box 35, 765

But the digital docent presents this image with the following explanation:

“Bobbie the Wonder Dog is your ideal match! This resilient Scotch Collie’s journey from Indiana back home to Silverton after being lost in 1923 charmed the nation, making him an adorable symbol of determination and loyalty.”

Our primary goal for this prototype was to ensure the model strictly adhered to the metadata it received, avoiding any fabrication while effectively connecting it to the visitor’s interest. If a photograph doesn’t fully pertain to a visitor’s request, the assistant aims to provide an informed rationalization rather than invent details.

Discovery

To further guide a visitor’s journey, we are experimenting with a feature that provides content hints in the form of suggested questions for the digital docent. If a visitor is unsure what to explore next, these prompts serve as entry points to related content.

Under the hood, these hints are generated during the re-ranking phase. When we refine our search results, we sample two additional images with increasing distance from the top-ranked result. The metadata from these images is then sent to the LLM, which reverse-engineers potential visitor queries instead of generating direct answers. This approach accomplishes two things: it encourages exploration by presenting topics related to the visitor’s initial interest, and it guarantees that the suggested questions will always have an answer, ensuring a seamless and engaging experience.

A Dynamic Response

Each of these steps plays a crucial role in transforming a static collection into a dynamic, interactive experience. By designing a system that balances precision with storytelling, we enable a more intuitive way to explore historical records—one that feels more like a conversation with an expert than a simple database search. But this project is just the beginning. The methods we’re developing have broader applications, offering a blueprint for how AI can be used to enhance the discovery of and engagement with cultural archives.