Museums sit on mountains of data they don’t fully use. Visitor feedback piles up, focus group sessions that get recorded and filed away … A lot of the time, there’s a first pass after the work is done to make sense of it all, but it may be incomplete or hurried because people are stretched thin. Or it’s not consistent — there’s a nagging feeling of “did we miss anything?” And, let’s be honest, sometimes there’s not really any synthesis done at all — the work is slow, and there’s always something more urgent.
Break down the problem
Big picture: We want to help museums overcome what we see as a critical challenge: Aligning around who their institution is for. To address a big, hairy problem like that, you have to break it down into parts.
A subcategory of that problem is understanding the institution’s users and potential users. It’s hard to align teams around who an institution is for (other than just saying ‘we’re for everyone’, which is a band-aid that only introduces more problems) if teams don’t have regular exposure to what users (communities) need and how they think about the goals that matter most to them.
Within that subcategory of ‘needing to understand (and not operating off of assumptions)’, there are many moving parts as well. For example, data collection and study framing are important challenge areas that are interior to the ‘understanding, not assuming’ subcategory, and those are both things that come before you ever get to the ‘let’s make sense of this data’ stage. Nonetheless, Making Sense of It All is an important step, and it happens to be an area that is highly relevant to some of our MaP members at this moment — specifically, our Value Realization Collaborative participants who have done the hard work of collecting data through deep listening and now need to turn that raw data into material that can actually support institutional decision making.
Address the problem in a way that supports a small group of real humans/institutions
If you’re unfamiliar, the Value Realization Collaborative is a cohort program where museum teams identify a community goal their museum would like to better support (in the case of our current cohort, that goal is supporting newcomers to the museum’s city or region who are seeking to “feel a sense of connection or belonging with members of my group”), conduct listening sessions with people from that group, and produce opportunity maps that describe how their respective institutions can support people who share that goal. Current participants are in the thick of that translation work: they’ve done the listening, collected the data, and now need to turn it into something that supports real choices about who their institution is for.
Participants in the program move through several stages, which are reflective of all kinds of research, not just Progress-Space Research:
Frame the study — answering critical questions like:
What do we want to learn and why?
What is the potential value of new information to our institution right now?
What will change as a result of this learning? (If the answer is ‘probably nothing’, you can stop here!)
Who will we need to hear from to make a sound decision? (all the scoping questions that come with that)
What signs along the way will tell us we can stop or may need to change course?
What approach(es) will best support our goals (methods)
Data collection
Synthesis / Sensemaking
Application
Integration (a week or five years from now, the work you’ve done may be relevant to others in your institution, making the same or adjacent decisions — ensuring that what you’ve learned is available and discoverable by them is a way to pay it forward)
Where’s the value?
Each of the steps outlined has value to the individuals involved and to the institution, but some have more value than others. Each step in the process is a beloved child to a die-hard researcher — we’d never want to give any one of them up for adoption — but the fact is, sometimes you have to choose. Ultimately, the goal of any research is to produce better results for an organization and the people it supports. If rigid adherence to a methodology — carefully attending to every single step equally — ultimately results in an organization’s failure in its learning initiatives, then we’ve traded ‘doing it the right way’ for the real goal.
So, we can look at these steps and ask: Where do we get the most value? What’s less valuable? It’s in the latter area that we can seek efficiencies.
Steps 1 and 2 are highly valuable. Framing determines whether you’re asking the right questions in the first place — get it wrong, and everything downstream is wasted effort. Data collection is where you actually hear from people. Human-to-human connection has all kinds of first and second-order benefits, including the relationship-building that deep listening sessions can foster with groups that might not otherwise relate to your institution. No one should try to shortcut relationship-building in the name of efficiency.
Steps 4 and 5 are also very valuable. Application is where insight becomes action — where diverse stakeholders make meaning together, and the institution actually changes. Integration ensures the work compounds over time; future decisions benefit from what you learned today. Without these steps, research is an academic exercise.
Step 3 is valuable — but less so. Synthesis is a processing step, not a generative one. The judgment it requires is largely pattern recognition — flagging what matters, distinguishing behavior from cognition. The real meaning-making happens in Application when people debate what the patterns mean. Much of synthesis is tedious extraction work. That’s where we see an opening.
Put the least valuable child up for adoption
Sorry, data synthesis. There’s only enough food in the house for four children.
We’re working on a tool to handle the synthesis piece.
It’s not a novel idea. In fact, you might think there must be existing solutions that can already handle this. But there aren’t, and I’ve tried building custom solutions in the past — it’s never worked.
Progress-Space Research is complex. Making sense of long discussions (listening sessions) where people talk about how they’ve approached their goals requires a lot of judgment to discern what should meet the bar for inclusion in a study.
To take just one example: Someone in a listening session says they “always” do x or usually prefer a over b. These are things that a market research firm would accept, but they don’t meet our requirements. What people say they do or say they prefer is often not aligned with behavior … in short, it’s unreliable. In our work, we need data that:
is rooted in specific past, personal history (here’s how i thought/felt at this specific time when i was trying to do/achieve this specific outcome)
pertains to the goal the institution wants to support (the framing step really is important)
is an example of what’s called interior cognition (inner thinking, emotional responses, or guiding principles — each of which have their own qualifying factors)
All the above must be true for a piece of data to meet the qualifying threshold for inclusion in a dataset, and many other subfactors and edge cases must be accounted for when synthesizing transcripts from listening sessions.
All this resists automation, which is why past attempts have failed. What’s different this time is that we have new capabilities — AI is certainly part of that, but other systems have become available, too — and we have a more pressing need this time around. Previously, I’ve been interested in this problem because it solves a problem for MaP (synthesis is a real time suck), but now we have community members who are struggling with it, and it’s up to me to find ways to support them. Besides, who says there aren’t more broadly applicable lessons to be learned here?
See if the solution might be useful to others beyond your small group
The point of this experiment isn’t to eliminate synthesis altogether — again, there’s no question that getting people together to talk about the meaning of specific data points has value, but most of that meaning-discussion can and should happen in Stage 4 (Application), and it should include a diverse group of stakeholders, not just researchers.
If the results of this experiment can help our VRC participants, can it help others in our community, too?
What would it be like to be able to invite people to bring in existing data and quickly help them ascertain its relevance to a particular challenge? This could happen with staff from a specific institution or perhaps across multiple institutions that share a common question/challenge and have data to contribute …
For now, you can follow our progress in this community space as I try to get this thing working for our VRC members. Feel free to leave comments or ask questions along the way.