In my previous post I wrote about the pitfalls of evaluating a mobile media tour based on pick-up rates. Pick-up rates measure the effectiveness of sales and marketing efforts but don’t in the least tell you if your visitor “gets” your message. Is it memorable? Understandable? Did it advance your mission? Are you trying to affect visitor behavior in some way? Did you reach the audience you wanted to reach?
Another ineffective measurement is bland guest feedback. “Our visitors tell us they love the tour!” Well, guess what? Everyone who’s ever attended a party at my house tells me they love my cooking, think my decor is divine, and that I “throw the best parties ever!” I love hearing it, and it even may be true, but it’s not an evaluation or even useful feedback. It’s being polite. The opposite of that isn’t the person who doesn’t like anything–though they do exist; it’s the person who, when asked what they think, feel compelled to say something negative simply because “well, you asked.”
What you need from an evaluation tool is specific to your tour and your organization and requires specific answers to predetermined questions. In short, what result were you trying to achieve when you decided to spend the time, money, and effort it takes to develop, produce, equip, and maintain this particular interpretive program?
The solution is to develop your evaluation plan while you’re developing your content and doing it the S.M.A.R.T. way: Specific, Measurable, Attainable, Relevant, and Time-focused. As you and your creative team develop your content and craft the tour experience, answer these questions along the way. When the project is ready to launch, you’ll have all the information you’ll need to know if your tour is doing what you want it to do.
SPECIFIC: Who, what, when*
Who are you primarily trying to reach with this tour? No program is going to reach or please everyone. It just isn’t. Not your exhibits, not your live docents, not your films/videos/interactives, not your website, not your blog, not your app. No one thing will appeal to all of your adult visitors, nor to your kids, nor to your foreign visitors, and on and on and on. But you can decide who you want the primary audience to be so that you can focus your evaluation efforts on that group.
Once you’ve decided who, what effect on that visitor do you want the tour to have? Do you want them to be informed? (By the way, “informed” is the weakest of goals. You might as well substitute the word “bored.”) Inspired? Connected? Engaged? The important thing is to set the bar high and to be specific.
When do you want this effect to happen? Immediately, while the person is at your location, or are you trying to inspire some future behavioral or intellectual change? Obviously, the answer can be both but it helps to pick the most important to you now so that you can develop a way to measure the success.
For example, if your primary purpose is immediate (“We want them to really connect with the artifacts on display as part of people’s daily lives.”), then an exit survey or even a feedback wall at the exit where visitors can write their impressions will give you the information you need. If it’s more future based, such as wanting to inspire someone to seek out more arts activities, then social media or getting an email address for a future survey should be part of the design.
MEASURABLE: How will you know when you are done?
Obviously, when it comes to interpretation we rarely consider our work “done”; however, it is important to set a marker so you’ll know you’ve moved the bar in the right direction. It could be as simple as collecting visitor feedback cards and looking for words that reflect your original intent. Or it may be encouraging a visitor action such as “sign up for our newsletter!” or “become a member!” and then tracking the completed action through signup forms. Whatever it is, it should be something you can count, not something that you simply feel or “seems like” a positive result.
ATTAINABLE: Do you have the ability to get the feedback you need?
Not every organizations can do major evaluation programs but every organization can do something to gather information. Can you develop and conduct simple surveys? Is your staff willing and able to encourage feedback or to get visitors to sign up for social media or whatever else it is you want them to do? If the answer to either of those is no, do you have another option? Does someone on your staff know how to use the app or audio guide software to get the information on visitor behavior that should be found there or can/does your vendor provide it?
RELEVANT How, exactly, does it fit into the broader mission?
Take a look at your annual and five-year goals and articulate how mobile media tour program is supposed to help you achieve them. Measuring effectiveness takes time, effort, and resources IN ADDITION to the costs of the audio tour itself, so it has to be worth it. Is there an additional emphasis on foreign visitors this year? Are you launching or expanding your accessibility goals? Are you expanding your interpretive programs to include more learning styles? Whatever they are, know how this fits into the picture.
TIME-FOCUSED: When will you take your measurements?
Most of the time, this section is called “time-bound” and is intended to set a deadline. However, I switch it to “time-focused” since a) most mobile tours last as long as they are relevant to the content, and b) evaluation should be an on-going activity. So how often will you gather data? Ongoing? Once a week, once a month, once a quarter, once a year? To whom will the information be reported and what are those people supposed to do with it? The important thing here is an established timeline and reporting structure.
CONCLUSION
One reason pick-up and attendance rates are so misused is because they are so easy to measure. More complex programs–those designed to impact a visitor on an emotional, intellectual, and sometimes even spiritual level–are more challenging to evaluate, but not impossible. Most important, this feedback is more valuable because it goes beyond sales and marketing efforts and assesses the organization’s ability to achieve its mission, its reason for being.
When you are developing your mobile tour, develop your evaluation tool at the same time. It will make a huge difference in both the effectiveness and your own enjoyment of your tour.
