Interview with Justin Wyatt


In October 2011, Justin Wyatt, VP of Primary Research Analytics for NBCUniversal sat down with MIP for an interview. In the below excerpts, Wyatt discusses how market research methods have evolved in the last decade and draws on his academic research interests to reflect on the current marketplace for indendent film and the potential of academic-industry collaborations.

Justin Wyatt is Vice President, Primary Research Analytics at NBCUniversal.  He has worked on both the client (ABC TV Network) and supplier (Frank N. Magid Associates, Hypothesis Group) sides of the market research industry.  He is the author of High Concept: Movies and Marketing in Hollywood and the co-editor of Contemporary American Independent Film: From the Margins to the Mainstream.  Wyatt holds a Ph.D. in Film and Television Studies from UCLA and a B.A. in Economics from the University of British Columbia.

Market Research and Digitization

Monday, October, 17, 2011 

We want to ask you about the nature of your job and how it’s changed in the last 10 years. What is market research like in the context of digitization?

It is a big question. I’m going to talk about it in terms of quantitative versus qualitative research. And then I can also talk about genres. In terms of quantitative, I think the major change is really a shift in method. When I started 10 years ago, the majority of studies were being done with phone surveys. And before that, of course, it was mail. Over this period, it really has shifted. The vast majority of studies are now done online. I won’t say all, because certain studies are still done on the phone. It started to shift around 2001. The benefits are obvious: There’s a speed to the data coming through. Before you would literally have a phone room and you’d be checking in either late at night or first thing in the morning about the number of “completes” they have for the surveys. You have 12 women age 34 and you’ve got 16 women 25 to 45 and so on and so forth. You’ll be looking at your quotas, and for TV studies it takes perhaps a week to 10 days for it to complete fielding. But now, with online surveys, you can send a sample out and finish a study in a day. It can be done really, really quickly.

So speed, and the cost is usually quite a bit lower, too. And the length of a survey tends to be longer. Now, I would say one of the things that really has suffered in a way is this notion of open-ended responses. Because one of the real benefits to phone surveys was you could get pretty detailed open-ended responses. You were able to get respondents having a rapport in a way with the interviewer, a kind of back-and-forth. You’d read these transcripts to open-ended response and you’d go, “OK, this is a lot of data.” And then we start doing online surveys and the open-ended response is three words. They realized they had to type something or they couldn’t continue.

Has it gotten better? Yes. It is anywhere near where it could be?  No. You do suffer in terms of the quality. I think there’s no way around that, though. The other thing that’s a real issue are the databases of respondents. Ask yourself, how many of my friends take online surveys? You’re upscale, educated people. Most of your friends aren’t spending their Saturday nights doing online surveys, you know? What happens is you get these survey houses and they have these incentive systems where you can win prizes, sweepstakes. So then you start to think after awhile, how seriously do people take these surveys? Around 2005, 2006 the big thing was understanding cheater algorithms. You build in cheater algorithms to figure out people doing diagonals, doing all ones or all fives, etc. Or who did the questionnaire so quickly, there’s no way they could have read it. Or they took so long, they must have been having coffee with a friend or something.

There could be ex-post analysis that goes with the data that’s pretty intricate. I know you took out that respondent because they took too long. But was it the right thing to do? Or maybe they really did feel like it was a diagonal. So there are all those kind of judgment calls, I think, that are still out there as issues.

I would say the other big thing is what they call “streaming respondents.” What happens with streaming respondents is a client will work with a third-party research vendor doing a study of, for instance, women ages 18 to 34. They may be working on TV soap operas. At the same time, the vendor may be doing studies for a variety of different clients: Tide detergent, Motorola, The Wall Street Journal and so on.  So you have different types at once. What they will do is they’ll see which of the surveys the respondent qualifies for and if they don’t qualify for yours, they will throw it to someone else. So it really isn’t an unbiased method of creating your sample.

So in the last few years, really, we’ve been saying we can’t stream respondents. If somebody doesn’t qualify for a study, you can’t use him or her again. Or if they did qualify for a study, you still can’t use them again. I found examples of this not at NBCU but at other places I’ve worked where, I’d go and read the open-ended responses for a long-length survey, and I’d read something like, “Well, last time you ask me this question …” and I’m like, oh, my goodness. You know you’re reusing your people when they have a rapport with the survey instrument! How scary is that?

So the quantitative research is kind of a work in progress. You need to have a large sample to be able to extrapolate to the national population if you’re making commercial entertainment. How are you able to do that except through online surveys? There are other methods like mall testing, dial testing and so on. But online, even with all its plusses and minuses, is a big shift that is huge, in my opinion, in terms of quality.

In terms of qualitative, what’s interesting is more of a shift in focus. Qualitative research in the TV industry really grows out of Nielsen ratings. We’ll look at minute-by-minute ratings and we’ll see where there are peaks and valleys. Why do people drop off at this moment in the show? And if we can’t come up with reasons why, then a network may think, “OK, we better try and talk to our viewers and see what’s going on with them.” For something very subtle like that, it’s difficult to phrase that in an online survey without being leading. So inevitably you have to use another method that is more qualitative. I would say one of the most interesting things for me with qualitative in the last 10 years is a shift to more of the “lived experience.” This is something that I developed a little bit when I was working for the ABC TV Network and that came out of my Ph.D. work.

I worked doing market research for Good Morning America, which is the biggest money-making show for ABC given the number of commercial pods.  We had tons of Nielsen data and that, in terms of the peaks and valleys, you could spend years going through one day’s show in terms of the trajectory for when people are engaging or not engaging. So I put together a “lived experience” study: at-home viewing sessions. It came from my own experience. I was in Dallas visiting friends – I used to teach there – and at a dinner party I sat next to a woman who runs a construction company. She was in her early 40s and kind of the target Good Morning America viewer. She had so much to say about Charlie Gibson and Diane Sawyer and the segments on the show and it was so rich and so in-depth and she was so smart about it all. But I was thinking, she would never do a traditional focus group. She would never show up at a focus group facility and get paid $100 to sit there for two hours because, you know, her daughter has to be run to soccer practice, she has to cut a deal with concrete suppliers, or whatever it is. So I thought, what I’m going to do is figure out a way to get her 'lived experience,' to understand it.

I moved into her home and I watched Good Morning America with her and her friends for two mornings. I told them, “I don’t work for Good Morning America, but I do work for ABC Research. My job is to understand what you think and feel.” First thing, on the Sunday night I had dinner with all of the participants so they could get comfortable with me. The goal was to get their barriers down, really. And then I watched the show with them and it was amazing how after the first 15 minutes, they were so unfiltered with what they would say because they were a group of friends. It went exactly against all of the “rules” of focus groups, where you don’t want to have a group bias. But I wanted to hear exactly what they’re feeling. So they would say stuff, they would interact with the commercials, and out of these two mornings came a 35page report that I authored on “The Lived Experience of Morning Television.” ABC News President David Westin asked me to do this Lived Experience project for all of the news except Nightline.  He felt that it was a little creepy to show up at someone’s house so late!


So I did. This is a long way of saying that breaking the rules of traditional research was something that I did to get closer to the viewers and what they were thinking and feeling. It worked out really well. We were able to get insights, qualitative insights, in a much deeper manner than we could through a focus group with an “objective moderator.” The thing to realize about all of this research is that it’s useful as a bellwether, nothing more.

Also, I would say with qualitative that engagement by viewers is a crucial issue nowadays. Are people watching? Yes, they’re watching. Are they watching and paying attention?   Are they engaged? This notion of engagement is something that we really spent a long time grappling with. It is qualitative research and cross-platform, especially in terms of news and information. As I said, I did a lot of work for ABC News, and I think ABC News is one of the savviest consumers of market research out there. Because their job is news and information, they are very comfortable with research. We did a large amount of research around the experience of consuming news and information in a cross platform world.

The news stuff is fascinating because these executives are really trying to understand technology and to move forward in an informed manner. The notion of the evening newscast is clearly an icon from the 50s-70s, but what does news mean in our current media environment? We experimented with a kind of vaunted idea of what ABC News could be and how people consume bite-size information. Even watching just two or three segments was a big commitment to people. Is the newscast of the future one that you select modules of content? And then at 6:00 p.m., or whenever, it’s delivered to you, that module you selected becomes your evening news? Is that something that could happen? Those are the kind of the experimental questions that we grappled with. The challenges are ongoing.  I’m sure that ABC and the other broadcast news operations are still grappling with them.

Indie Film

Monday, October 17, 2011

We want to ask you some questions about your research on independent film and ways that you see developments in the contemporary marketplace for people working outside conglomerates. 

It’s so grim, isn’t it? I think what was interesting to me about independent film in the early ’90s ceased being interesting by the midto late ’90s. I wrote an article about Miramax and New Line in an anthology by Steve Neale. I knew for me that when you start having The English Patient being called an independent film then, like, oh, my God. That’s like Dr. Zhivago. So big, so much the product of a major studio that “independent” really just becomes a marketing term at this point. And the funds and the sources of supporting independent film just over the last two decades have been cut over and over again. I look at the small distributors. Take a company like Zeitgeist Films. They still do really interesting stuff but for them, a good documentary like Bill Cunningham New York, it’s a million dollars in total gross and it’s a huge success for them. So the scale is just that, doing things so small, so tiny.

Do you see new business models emerging that take advantage of digital technologies?

I think the challenge with digital at this point is breaking through, because there’s so much information and there are so many options and so many places to go and things to look at. It really is a time of information super overload. To get on the radar is a very tough thing at this point. Obviously, the potential for reinventing independent film through digital is there, but it hasn’t really happened yet.

I kind of feel let down by film in general, independent or mainstream, to tell you the truth. I’m a child of the ’70s, so I grew up with the films of Robert Altman, Martin Scorsese, Peter Bogdanovich and Francis Coppola. And part of what I was learning all those years ago was really about my falling in love with film. In High Concept, I drew the connection between Nashville and Jaws. I loved Nashville. I saw it over and over again. I wasn’t that fond of Jaws. And yet Jaws is really about the trajectory of Hollywood production for the next 35 years, you know?

I kind of feel like film failed me from the mid-’80s onwards. There are so few movies that I find interesting now. I’ll get through half an hour and decide I’d rather read a magazine or a book. Although I have to admit there is a kind of split in my household now because [my partner] Jeff, who’s not the film historian, will only watch arthouse stuff.


He’ll watch Bergman, Truffaut, Godard and Antonioni or imports on BBC America.  His favorite this year was Lars Von Trier’s Melancholia.  We’ll be sitting together after dinner, and my initial instinct is to scream, “I want to watch Rachel Zoe!”  That doesn’t go over so well.

One last question: What are some ways in which scholars and industry practitioners can foster better, more productive dialogue?

Well, I think that having people like yourselves become “consultants” in the industry, who could offer advice on aesthetics, content, and history, and how those things come together. I’m constantly amazed at how much I’m able to invoke media theorists and historians in my everyday work.  Of course, I never quote media studies verbatim to my internal clients but writing on narrative, media consumption, stardom, and celebrity are continually useful to me. And I can certainly see that an academic who had knowledge of a storytelling and history – media history, especially – could be incredibly useful.

While I was a grad student at UCLA, I completed three separate internships in the media industries. I really encourage M.A. and Ph.D. students to follow suit. The benefits are in both directions. If your interactions are meaningful, you’re able to raise your level as an executive through interaction with talented and engaged interns. And that’s really what you want to be doing: You want to be interacting with smart people. So having the chance to have an exchange with people in the academy who understand narrative, understand storytelling, understand history, and who understand celebrity and emotional connections to dramatic characters.

I think what’s most important is that a tiny shift has to happen in terms of the vocabulary. You really need to be speaking the vocabulary of people in the industry. It’s not that different. The people that I’m dealing with in media research and production are all smart and incredibly accomplished people. However, they’re going to be thrown off by terminology that’s seems to be “too academic.” It’s about finding a way to translate these theories to encourage the greatest possible appreciation and impact.

More From Critical Conversations

VFX artist and activist Dave Rand recounts his experience working at Rhythm & Hues when the firm filed for Chapter 11 bankruptcy. 

Steve Kaplan

Steve Kaplan discusses the challenges he confronts trying to convince visual effects artists of the value of organized labor.

Thomas Gewecke talks about the value of digital ownership for consumers and distributors.

Mitch Singer talks about taking advantage of "disruption" in developing tools for digital entertainment distribution.

Dick Wolf talks to MIP about launching Chicago Fire in a changing media environment.