Start With the Bet, Not Data

I was scrolling through the Times over the weekend when an ad for a Paul Klee show at the Jewish Museum stopped me.

It’s got five elements: a painting, the artist’s name, an exhibition title (Other Possible Worlds), dates/location, and the museum’s name.

That’s an interesting bet to unpack. If we assume the purpose of the ad is to get bodies through the door (and downstream results like membership or donations), then this ad suggests that the artist’s name or the museum’s name alone is enough to achieve that goal. The value, it seems to say, is self-evident.

Communication is always a bet. It’s true for individuals as well as organizations. For example, there are lots of bets wrapped up in my writing this newsletter every week for years on end. Within that activity are bets like: Kyle is able to say something worth people’s time once a week; Museum folks will be interested in it; Relationships can grow from a regular writing practice; and so on.

You could argue that everything is a bet. Returning to the ad: the museum didn’t start placing bets when it decided to articulate the value of this exhibit in this particular way. There were many bets placed before this one, including the decision to host this particular exhibit at all.

Bets (decisions) can be more or less risky, and there are steps you can take to reduce risk. They don’t have to be expensive steps — a lot can be improved just by changing the habits we have when it comes to placing bets.

The Black Box

Maybe the ad caught my eye because some part of my brain was still thinking about a conversation I had with a client last week about the challenge of integrating evidence into decision-making.

Many museums want to adopt a more evidence-based approach to decision-making (good!), but they hit the same wall we discussed on our call: they start with whatever data they have and try to pull a decision out of it. It’s an understandable default that most of us inherit. But the order in which we do things often has a surprising impact on results, and if we start with the data, the conversation usually dead-ends in attendance figures or NPS scores that nobody knows what to do with.

I suggested an experiment to help them break this data dead-end cycle that might be useful to you in your work:

1. Put a black box in the middle of your staff meetings.

2. Anytime you're trying to make a decision of any consequence, ask each other what would need to come out of that black box in order for you to know whether this idea or path was a good one to follow.

Most of the time, there’s not going to be any one single answer, and nothing will be definitive. You’re not going to have a single piece of evidence that you could pull out of the box that will tell you with 100% certainty a particular bet is going to pay off. Instead, you’ll hear yourself or your colleagues saying things like, “If we had put on an exhibit about this before and attendance was below average, then that would suggest this is a slightly more risky bet” or “If the black box contained transcripts with a dozen people talking about how they decided to become a member after they visited with a friend, that would mean we could feel a little bit more confident in going down this path”. No one piece of evidence from the black box would kill an idea any more than any one piece of evidence would guarantee you should choose it. That’s good because you want people to get comfortable and not feel threatened by having the black box in on your conversations.

The black box helps you name what would be helpful in choosing one path over another. You’re no longer trying to pull a good idea out of data. You’re trying to understand what data would tell you more about the value or potential of the different options before you.

Avoid filters: don’t ask whether you have the data or not at this point. Don’t ask whether it would be too difficult to get that data if you don’t have it. Give yourself permission to take five or six steps through that dark room before you turn on the lights and subject things to what’s ‘reasonable’ or practical. Someone will say, “We don’t have survey data on that,” or “We don’t have time to interview 10 visitors about that”. Resist that resource-based mentality for just a few more minutes. Make notes about what evidence people are citing — you’re collecting data just by having the discussion! By having these conversations regularly, patterns emerge. The same kinds of evidence keep surfacing because they represent institutional knowledge gaps — things your team needs to know but hasn’t yet invested in knowing. You’re not just solving for today’s meeting. You’re changing how decisions get made.

What Would the Box Say?

With that black-box conversation in mind, I paused my doomscrolling to consider the Paul Klee ad.

The first thing I thought was, how many people know who Paul Klee is? I know who he is, but I have an MFA, which is a rare condition most people don't suffer from. Klee is a major artist, but he's not a household name. In addition to knowing who Paul Klee is, the museum seems to be betting that people will appreciate the work itself — that the painting is compelling enough, even without name recognition, to make someone take the next step toward visiting.

What would the black box have said?

Sitting in the room around the black box with this ad in front of them, the team would have said, If the black box had data on how many people who see this ad will also recognize Paul Klee's name or have some appreciation of his work, that would help us gauge whether this framing is the right bet. If the number is large, we'd feel good about leading with the artist. If it's small, we'd want to think about what else the ad could offer.

There are lots of other things you might pull out of the box. This exhibit is about Klee’s late period; If the museum, or perhaps other museums, have put on an exhibit of his earlier work, for example, it would be nice to be able to open the black box and see what the results were. Neue Galerie is six blocks south of the Jewish Museum and hosted an exhibition on Klee’s work twenty years ago. How did they frame the value of the exhibit at that time? If the black box held a detailed post-mortem on that exhibit, and the museum had taken a similar approach to marketing the show (i.e., banking on the artist’s name recognition), that would shed light on the success of choosing this bet over another.

But the ad isn’t just about the artist. There’s also a bet on the image — the implied bet is Some people who see this ad will have an appreciation for art/painting in general and even if they aren’t terribly familiar with Paul Klee, the image itself will resonate. The black box would need to tell us how many people who see this ad might be art lovers — the kind that just love painting or a certain style or period, and this image speaks to those interests.

And then there’s the museum's name. We’d want to be able to reach into the box and pull out evidence that shows: There is a significant portion of people (insert range) who are enthusiastic fans of the Jewish Museum, so this approach to advertising the exhibit is a safe bet because we really just need to refresh people’s memories — it doesn’t matter if we’re speaking to people’s interest in a particular artist or art history or any particular individual outcomes. (More on this frame below.)

These are the main elements of this particular bet: Two name-brand bets (Paul Klee and the Jewish Museum, respectively) and what we might call a professional/hobbyist bet to borrow from John Falk’s typology (i.e., people who love art and art history will find the image compelling enough to influence their behavior). The other elements in the ad — location and dates — are informational. Perhaps someone who sees the ad will be in New York this spring or summer, and maybe this will suggest to them that this is something that they could do while they’re in town. But even if that is the bet, the black box can be useful there, too. We would want to open the box and find a well-calibrated range for how many people who see the ad will travel to New York during the given period, for starters. The black box gives us the tools to begin decomposing the underlying assumptions around what would have to be true for a particular choice or path to be better than alternatives.

At this point in our meeting, everyone accepts that there is no discussion of whether the museum has the data we’re hoping to find in the box or whether it can get it. We’re just describing the shape of our uncertainty. It doesn’t matter that there’s likely no ready-to-review post-mortem from Neue Galerie or precise numbers as to how many long-suffering MFA graduates will see the ad. Because the black box¹ helps us own what we don’t know and, even if we decide that the evidence can’t be found or produced, it shows us the degree of uncertainty we’re accepting by making particular choices.

What isn’t in the ad has to be accounted for as well.

The exhibition traces Klee’s final decade and presumably contextualizes what was happening at that time, drawing more or less overt parallels between Klee’s context (Nazis, fascism) and our time (Nazis, fascism). Still, none of that is in this ad. If we’re betting on timeliness as a draw, we’re also betting on some deep knowledge of history and a long-lingering, contemplative consumer of advertising. If we want the black box to estimate the number of people who will see the ad and, based on what it shows alone, will begin to reflect on the implications of Paul Klee’s struggles under fascist repression and our own time and how they might want to go see the exhibit because that’s something of interest, it's safe to assume that number is small enough that advertising probably isn't the right vehicle for that kind of resonance — which doesn’t mean the connection isn’t real once someone is in the gallery. The absence of any reference to how Klee’s time or work or experience relates to our own is a bet that that is not what will draw crowds.

Similarly, we don’t see anything about individual outcomes that the exhibition might support. There’s nothing to suggest that this experience affords social connection², for example. By not talking about or showing anything about personal outcomes, the museum is betting that art-historical interest is a stronger draw than the personal outcomes the experience affords.

Feed the Box

You may find that the data you want the box to contain isn’t as difficult to produce as you might think.

Let’s try it with the Klee ad. The core question the black box needs to answer is: How many people who see this ad will visit the exhibition because of it?

We don’t have that number. No one does because the data doesn’t exist yet. But we can reason our way toward what’s needed.

Start with the people most likely to respond. Let’s round down a bit and say there are 20 million people in the NYC area, and say that about five million people live within easy travel distance of the Jewish Museum. So, a quarter of the total population could take public transportation without too much effort. How many of them will see this ad? Of those, how many recognize Paul Klee’s name? Of those, how many feel enough pull to look into it further? Of those, how many actually make the trip? Each filter shrinks the number, and when you stack them, even generous assumptions leave you with a small fraction of the population you started with. Widen the circle to the broader metro area and the conversion thins further — more people, but weaker pull. Then there are the tourists: roughly 22 million people will pass through New York during the exhibition’s four-month run, but for most of them, a Paul Klee show is’'t why they came. (There’s a lot of competition even just for art lovers during a several-day trip to NYC, and as much as we might like to think that people stroll down Museum Mile the way they might walk through a mall, you're going to be hard-pressed to visit the Met, the Guggenheim, and a third museum in an afternoon. That kind of casual, pop-in-for-an-hour behavior is not something we’d associate with the professional/hobbyist type that this ad seems to be geared toward.)

When you run the numbers — even loosely — you end up somewhere in the low tens of thousands of exhibition-motivated visitors over the run of the show. That gives us a first sketch of what the box might contain.

Now sanity-check it. A mid-size museum on Museum Mile might see somewhere around 200,000 visitors a year. Over a four-month spring-and-summer window, maybe 70,000 to 80,000 people walk through the doors regardless — members, school groups, foot traffic, tourists who wandered in. If our estimate is in the right ballpark, it suggests that roughly 15–20% of the museum’s traffic during the exhibition period would be exhibition-motivated, and the rest would have come anyway.

That’s a useful thing to know, even if the numbers are rough. The numbers don’t need to be precise to be useful. The approach is named for Enrico Fermi, who estimated the explosive yield of the first nuclear test by dropping scraps of paper into the blast wave. If structured estimation can get you in the ballpark of a nuclear explosion, it can get you in the ballpark of exhibition attendance. Moreover, you’ll be doing far less of this kind of calibration at your museum because you have baselines or can get them.

The point isn’t to produce a number you’d stake your career on. The point is to know whether you’re talking about five thousand people or fifty thousand — because those are different decisions. A range that’s honest about its uncertainty is more useful than a single number that pretends to be exact, and infinitely more useful than no number at all, which is what many museums are working with when they place these bets.

Remember that any decision-making approach should be evaluated against the status quo. If you’re comparing black-box, Fermi-style calibration against plain old guessing or hoping that the person next to you in the meeting has an opinion based on some data (hint: they don’t — they're assuming or hoping you do), then this wins every day of the week.

All that said, there’s a problem with the path I just sketched out.

The entire calibration exercise assumes the ad is trying to convert — that its job is to take someone who sees it and turn them into someone who buys a ticket. That’s the lens I used, and I think it’s fairly common — I mean, after all, to say the goal is to just get the person to click on the “learn more” button seems misguided. The museum isn’t investing in the exhibit so that people will click on a button to learn more. Ideally, under this frame, we’d be able to trace a line from impression to admission (or joining/membership, donating, and so forth).

It’s reasonable to expect marketing to deliver direct business outcomes, but there is a credible argument that this is the wrong lens.

Byron Sharp, a marketing researcher and Director of the Ehrenberg-Bass Institute at the University of South Australia, argues in his book (it’s good — give it a read) that most ads don’t work by persuading anyone to do anything. They work by keeping a brand mentally available — lodged in your memory so that when the moment of decision arrives (Saturday morning, nothing planned, someone asks “what should we do?”), the brand surfaces. You don’t remember being persuaded. You just remember that the Jewish Museum exists and has something on.

From this perspective, the ad we’ve been dissecting doesn’t fail to make a case. It’s not trying to make a case. The artist’s name, a compelling image, the museum’s name — that’s not a thin value proposition but rather a memory device. It’s a deposit into a mental account that pays out later, in a context the museum can’t predict or control.

If Sharp is right, then measuring this ad by conversion — how many people who saw it bought a ticket? — is like measuring a billboard’s success by counting how many drivers pulled over. You’d conclude that billboards don’t work, and you’d be wrong about why.

This matters for the black box. Because if you’re sitting in that staff meeting and the question you pull from the box is “how many people will this ad convert?”, you might be asking the wrong question. Under a mental availability framework, the better question is: Will people remember us when the moment comes? And the evidence you’d want from the box changes accordingly. You’d want to know about recognition, not conversion; Salience (one of Sharp’s favorite words), not persuasion. Whether the image is distinctive enough to stick, not whether it’s compelling enough to motivate a trip.

And yet …

The Fermi estimation still works under this lens because you’re still decomposing an uncertainty into estimable parts. But what you’d pull out of the box is different. Instead of “how many people convert,” you’d ask: “how many people, the next time they’re deciding what to do on a Saturday, will have the Jewish Museum surface as an option — and is this ad the kind that builds or refreshes that availability?”

That’s harder to estimate, but Douglas Hubbard would say a range isn’t impossible (another good book). Besides, the discussion itself has value. The black box doesn’t care which framework you bring to it. It just asks: what would you need to know? And if your team is sitting around that box, with half the room thinking about conversion and the other half about mental availability without realizing it, the black box will make those assumptions clear sooner rather than later.

Try the black box yourself. You can literally take a cardboard box, paint it black or whatever color you fancy, and put it in the middle of the conference room table. Ask: What would we need to find in that box to better understand whether this bet that we’re discussing — whatever it may be — will be more or less effective?

The institutions I work with are building the habit of asking “what would we need to know?” before “what data do we have?” If you’re trying to make that shift — or you’ve tried something like this and it changed the conversation — I'd be glad to hear from you. Just reply to today’s newsletter.

Kyle