Mary Ellen Gordon of Market Truths is by far the single most knowledgeable and experienced person (and avatar) doing market research in Second Life. She is highly respected by both the business and academic research community in Second Life - and she has been very generous with her knowledge within the SL Business Communicators community in-world. I've been after her for a while to allow me to interview her for this blog - and she has finally agreed. What follows are nine questions I posed to her about her work and research in SL.
By way of disclosure, Mary Ellen has become a friend and a colleague through our association in SL. She and I collaborate from time to time on business projects and my respect for her expetise continues to grow. I'm delighted she has agreed to share this space with me briefly. I hope you find it beneficial:
1) Mary Ellen, tell us a bit about your background as a market researcher.
I've been involved in research for twenty years now. I have a PhD in marketing, and have Professional Researcher Certification from the Marketing Research Association. The first project I did was the first study of the US IVF market back when I was working in the pharmaceutical industry. I subsequently completed my PhD program with a focused research component, and then spent six years teaching (marketing research, among other things) and doing academic marketing research.
The idea behind Market Truths was to blend the type of rigor that is demanded by academic research with the real world relevance demanded by applied market research. We've been going since 2001, and have done a variety of different types of research projects for clients in a range of industries in the US, UK, and New Zealand.
2) How long have you been conducting research in Second Life - and what type of research have you done?
We started doing virtual world research in Second Life in October 2006. Since then, our SL research has included surveys, interviews, focus groups, and observational research. We maintain our own panel to use as a sampling frame for SL research, but we also survey visitors in client locations, at events, or utilize "real world" participants. Our experience in SL has allowed us to create some unique methods of gathering and validating research there, as well.
3) Why do research in virtual worlds at all? What characteristics of the environment of Second Life - or virtual worlds in general - makes them valuable to conduct research via those environments?
There are several things that make virtual worlds valuable from a research perspective:
a. They make it possible to evaluate reactions to three dimensional objects easily, quickly, and (relatively) inexpensively. For example, if a company wants to test a prototype of a product or a package, everyone can see that in 3D and you can even manipulate it in real time in 3D to interactively create the ideal version in cooperation with the research participants.
b. Access to target markets. At present the SL environment is quite unique in that it is very international and it has a high concentration of people in technical, creative, and marketing jobs. Technical and creative people are generally not the types of people who jump at the opportunity to participate in market research, yet we find that they are willing to cooperate in SL. That's certainly important for companies targeting those types of people, and can also be important for companies that need technical or creative input into new product development. The international aspect is important for companies targeting customers in more than one country since it makes it possible to get information from people in a lot of different places in a short time.
c. It removes RL geographical constraints. For example, it's not uncommon for companies to make decisions for all of the US based on focus groups conducted in just a few cities. Obviously that's not appropriate if there are regional differences, but in RL you just can't easily get people from Boston, Iowa, Alabama and Los Angeles together for a focus group. In the virtual world you can get people from all over the globe together at once. While it's been possible to do that sort of thing with other forms of online communication for a while, there's definitely something more immersive about being "together" in the same room in a place like SL -- and that influences the quality of data.
d. The quality of data collected tends to be high. People are really forthcoming in SL, and comments made in focus groups, responses to open-ended survey questions, etc. all tend to be thoughtful and helpful. This is probably attributable to a combination of the types of people who are involved, the fact that they tend to be relaxed while in SL (they don't have to think about how long it's going to take to drive home after being interviewed, etc.), and (in the case of any research that relates directly to SL) because many people are really passionate about SL.
4) There seems to be a general misunderstanding about sample size relative to the validity of research. Can talk a bit about sample quality and quantity?
Yes, there's a huge amount of misunderstanding about sampling in general and sample size in particular!
To start at the beginning, it's important to remember that most statistics were developed for use with a "probability sample." That means that every member of the population has a known - non-zero - chance of being included. That's fine for crops or lab rats, but is problematic when it comes to humans because any research that enables people to opt in or out will have some people with a zero chance of inclusion. That, combined with things like variation in the number of phone lines in a single house or the number of avatars associated with a RL person, means the "known" criterion is also seldom fully satisfied.
While almost no market research can fully meet the gold standard of a true probability sample, with everything we do we should be trying to approximate that to the extent that we can and be aware of potential problems and biases to the extent that we can't. So, for example, one way to maximize the size of a sample would be to just let everyone who wants to do a survey, and do it as often as they like (like the audience voting someone in or out on a reality show).
I've seen a lot of SL surveys being done that way, but from a sampling standpoint that's not good as the type of people who participate are unlikely to be representative of the whole population that you're interested in. Also, the observations are unlikely to be independent (e.g., if the same person responds to a survey with three different avatars, you are really just getting one person's views three times rather than three independent opinions), which is another one of the fundamental assumptions underlying most statistics. So knowing the absolute size of a sample really doesn't tell you much of anything unless you know how it was selected and what was done to try to control for things such as self-selection, and non independent observations (and in particular in the case of SL, alts - multiple avatars owned by the same person).
Assuming a sample has been drawn in a way that's likely to reduce those problems, the size that's required to get good information is smaller than people often think. Larger samples tend to make estimates more precise; however, most marketing decisions don't require the same degree of precision as, say, determining the appropriate dosage of a drug. The requirement for precision is important because once you pass a certain threshold the sample size needs to increase a lot to get a meaningful increase in precision.
For example, the following graph shows how the maximum margin of error associated with the estimate of a proportion (based on a population of 5,000,000 and a 95% confidence level) changes based on the sample size:
The exact numbers change, but a similar pattern holds for estimating different types of statistics, and with different population sizes, confidence levels, and sampling methods.
For many marketing decisions - especially in an area that's evolving as rapidly as virtual worlds, a 5 or even 10% margin of error is fine. All other things being equal, it might be nice to go for even greater precision, but all other things are not equal. As shown in the graph, going down to something like a 1% margin of error requires an extremely large sample. Getting that sample increases the cost of the project - particularly if adequate controls are put in place to reduce the chances of the types of sampling problems described previously (and if those controls are not in place the larger sample size is pointless). In an environment that's evolving as rapidly as virtual worlds, clients would often get more information and insights for their money if they conducted a survey at regular intervals with a smaller sample than with a one shot survey with a larger sample.
4A). So, then small samples (if chosen correctly) can be more "accurate" than large samples, and that the size of the sample isn't quite as important as the method used to include the participants? It sounds like you are also telling marketers to at least look at how a the data was gathered before taking a survey as relevant. Right?
Exactly. Quality is more important than quantity as far as samples go. If a poor quality process has been used to generate the sample, then it’s pretty useless no matter how big it is. If, on the other hand, great care has been taken in the sampling process, then you can get a lot of information from even a relatively small sample.
5) What is a surprising or unexpected issue, outcome, or finding you have run across?
Perhaps the most surprising thing is the extent to which people's behavior in world corresponds to their behavior in RL. Prior to beginning research in SL I had read a lot of speculation about people using avatars to enact some sort of fantasy life, and I still hear and read that sort of conjecture fairly frequently (though typically not from people who spend a lot of time in virtual worlds themselves). This is an issue we've looked at in a number of different contexts, and what we've found is that some people may use avatars to enact fantasies some of the time, but that does not apply to most people most of the time in SL. Role playing may be more common in other virtual worlds, but at least in SL there is a fairly close correspondence (in attitudes and behaviors - not necessarily physical appearance) between the majority of RL people and their avatars.
6) There seem to be a lot of companies and universities beginning to conduct research in SL with varying degrees of success. What is the most common "mistake" you see when conducting research in this environment?
The most common mistake is conducting research before really understanding the SL environment. That can manifest in different ways. One is not putting in place any controls for alts when the unit of analysis is supposed to be the RL person. It seems to me that dealing with alts is the #1 issue facing SL researchers. We spent hundreds of hours on it last year, finding solutions to keep our research valid.
Lack of understanding of SL can also manifest itself in the conceptualization and design of the research. For example, there was an article by researchers at well known universities that was published in an academic journal and that was cited in a lot of mainstream media that investigated eye contact among avatars without addressing things like the fact that the RL person may be using the camera to look somewhere entirely different from where the avatar is looking; there are SL settings that make avatars' heads automatically turn to whoever is speaking; and to manually get one avatar to have eye contact with another relies at least as much on dexterity with a mouse as it does on intention, empathy, etc.
Lack of understanding can also come through in companies not applying the same standards for privacy protection in SL as they would when collecting data in RL because they're under the (false) impression that they don't have to.
7) We see a lot of different kinds of "mainstream" research companies like comScore, Nielsen, etc., releasing research about virtual worlds. What is useful about these tools for marketers?
They're useful in terms of getting a general sense of what proportion of the population participates in virtual worlds because they're including virtual world questions in surveys of the general population. They're much less useful in terms of uncovering the details of what virtual world participants think, do, etc. because the proportion of the general population that participates in virtual worlds is still quite small, so even if their general population survey starts with a big sample the number of virtual world participants that they end up with tends to be too small to use for detailed analyses of just that sub group.
8) What are three things you recommend a marketer consider about vw research?
First, just that they consider it at all. As described above, done properly, it does yield information that's relevant to RL marketing decisions and has some benefits even compared to more traditional forms of research.
Second, that you don't need to have a virtual world presence, or even be considering one, to benefit from virtual world research. It can just be the venue for gathering data to make decisions about marketing strategies and tactics that will be executed through other (more traditional) channels.
Third, that marketing is changing. While none of us know exactly where that change will lead, it seems clear that traditional channels are diminishing in importance and virtual worlds and other social media are increasing. There may not yet be a lot of "eyeballs" present in virtual worlds, but if this is indeed the general direction that marketing is going, it may pay to consider some exploratory research now to provide a foundation for future decisions and to be prepared if / when virtual worlds do become a more dominant channel.
9) What trends do you see with marketing in virtual worlds this year?
I think we'll see more variety in who is using virtual worlds and for what purpose. Up until about the middle of last year it seemed that most companies with a virtual world presence were big consumer brands there for PR reasons. Toward the end of last year when our most recent RL Brands in SL Report was published, we noticed that some brands (particularly high tech companies) were using SL not for general PR purposes, but as a way of communicating with a very specific target audience (in the case of the high tech companies, that was software developers). I suspect that trend will continue this year, probably extending into other populations that are disproportionately represented within SL and to other purposes (e.g., new product development, product customization, etc.). I also think we'll see more small and B2B brands coming in to SL and that their aims will also be more specific and varied than just generating buzz.
January 23, 2008