News Publications

Developing an Opt-in Panel for Swedish Opinion Research

Publicerat av Demoskop7 January 201622 September 2023

Developing an Opt-in Panel for Swedish Opinion Research

7 january 2016

Karin Nelsson, Inizio, Inc., Sweden
Stefan Pettersson, Inizio, Inc., Sweden
Lars Lyberg, Lyberg Survey Quality Management, Inc., Sweden
Paper presented at the WAPOR regional conference
Doha, Qatar, March 7-9, 2015

1. Introduction

In Sweden the year 2014 was called the super election year because of the elections to the EU parliament, the Swedish parliament, the regional governments, as well as numerous local referenda. These events have generated a demand for constant polling resulting in almost daily predictions from different survey organizations. Most of the survey organizations use initial probability samples resulting in estimates or predictions with overstated confidence levels due to oversampling until certain sample sizes are achieved, nonresponse biases and other nonsampling errors. Other survey organizations use online panels developed in different ways, where confidence levels cannot be established due to a lack of theory. One critical aspect in most opinion surveys is timeliness. Most large media organizations as well as the political parties demand data within extremely tight timeframes leaving very little room for traditional nonresponse follow-up and other measures that might mitigate nonsampling errors. Another critical aspect is the current strive for data-driven journalism, i.e., media’s demand for opinion and other data than can add value to reporting of news and events and political commentaries. Obviously such a demand must be met by nontraditional survey methods such as panel approaches. Typical survey topics include confidence in government agencies, opinions about movies, attitudes toward EU membership, and perceptions of leaders of political parties.

In this paper we describe our development and implementation of an opt-in panel that is eventually supposed to be run entirely by our client, Aftonbladet, which is the largest newspaper in Sweden. Aftonbladet is part of Schibsted Media Group- a large media group with 7,000 employees in 29 countries. We have developed a process for panel recruitment among web sites associated with Aftonbladet and its Swedish partners. Currently the panel has more than 30,000 members and numerous surveys have been conducted since its inception. To adjust for representation problems we have used various weighting procedures and we have received guidance from a scientific board. Procedures are in place to identify satisficing behavior and mischief. Panel data have been calibrated with official statistics and we have trained one of Aftonbladet’s journalists in survey methodology so that a transition of responsibilities is possible. We will discuss general concerns and challenges associated with opt-in panels.

We follow the ISO standard 26362 for access panels in market, opinion, and social research. This standard is a set of criteria against which access panel providers can be evaluated and the quality of such panels be assessed. We will also briefly discuss the fact that both traditional approaches and opt-in approaches have problems. Recent exchanges on AAPOR-net and other discussions illustrate this.

2. A changing survey landscape

During recent years the survey landscape has gradually changed. Several factors have triggered this change. Traditional surveys using the interview mode have become increasingly expensive and time-consuming. Nonresponse rates in Swedish government surveys of individuals have reached unprecedented levels, such as 30% in the Labor Force Survey. Web surveys based on some kind of population frame notoriously experience very high nonresponse rates as well. Private marketing and polling institutes often end up with nonresponse rates from 80% and up. At the same time new devices, such as smartphones, suitable for data collection have entered the scene. Social media such as twitter and facebook can also be used as survey modes. Also decision making can now, more than before, be based on several data sources that might complement surveys. Big data and administrative records are examples of such sources. AAPOR (2015) is a task force report on the potential impact of big data on survey research. AAPOR has also released task force reports that point out the potential of mobile devices (AAPOR 2014a), the role of social media (2014b), statements on opt-in panels (2010), and nonprobability sampling (AAPOR 2013).

The classic sampling theory developed by Neyman (1934) relied on the assumption that the only errors that existed were the sampling errors. Of course Neyman and his peers knew that this was an unrealistic assumption but the pressure was intense to develop a data collection strategy that could be restricted to a sample of the population of interest rather than investigating the entire population. This was mainly done for financial reasons. The fact that a sample also allows a more controlled survey environment was not really considered. In those days very little was known about measurement errors and data processing errors. Nonresponse was a minor concern, since people were quite willing to participate in surveys. With small nonresponse rates a sample really was representative of the target population. In the 1970s response rates started to decline in most western countries. Reasons for this development include (a) an increasing number of surveys conducted by an increasing number of service providers, (b) badly designed questionnaires, some of them containing many questions, (c) big brother concerns, and (d) a dislike among service providers to conduct refusal conversion. Now we are in the unfavorable nonresponse situation described above. Thus, most samples are nonrepresentative due to high nonresponse rates and this state of affairs is not likely going to change. It takes adjustment to compensate for any biases resulting from nonresponse.

Time is also a factor. Clients want results very fast these days. Most survey clients do not want to wait several weeks or even months to get results. A long data collection period is necessary if a system of tracking people, following them up and trying to convert them is in place. Thus, if time is crucial then survey procedures have to change.

Web is obviously the fastest mode but can be difficult to implement in a probability-based fashion since it requires a basic probability sample of individuals who are asked to enter a website to access the questionnaire. Those who do not use the internet are provided with equipment that makes them able to participate. A less time-consuming and more practical approach would be to use volunteer samples, one example being the opt-in panel. Through different recruitment means, people are asked to join a panel and participate in surveys on a regular basis. Voluntary participation is of course a procedure that violates the classic sampling theory, where it is stated that every member of the population should have a known, nonzero probability of being selected for the survey. However, most probability-based surveys are also voluntary.

The use of opt-in panels has generated a lot of debate during recent years. Opponents argue that inference is not possible, while proponents argue that all samples these days are nonrepresentative due to the high nonresponse rates encountered. The latter agree that nonprobability samples can be heavily skewed and any sample composition needs adjustment of some kind. The debate heated up last year when The New York Times and CBS News told the world that they would start using online panels from YouGov as part of their election surveys (Keeter 2014). For a long time prominent media organizations have had strict requirements regarding surveys and probability sampling as well as publishing results from surveys. New York Times started revising their standards as a consequence of endorsing nonprobability samples.

Nonprobability sampling has recently gained in popularity. Some nonprobability proponents have turned to Bayesian inference, where some of the problems with nonprobability can be addressed (Wang et al 2014, Gelman et al 2014). In Bayesian inference a prior distribution is the point of departure. This distribution is combined with data collected in some way resulting in a posterior distribution. It is possible to produce what is called a credibility interval (AAPOR 2012), which is different from the classic margin of error. Margins of error cannot be calculated for a nonprobability sample and nor can the nonresponse rate. This is clearly stated in the standard for access panels (ISO 2009), which contains requirements regarding the communication between the panel provider and the panel members, recruitment of new panel members, panel structure and size, panel management, panel usage, client reporting, and code of conduct.

Clearly there is yet no formal theory for handling nonprobability samples. Current procedures aim at constructing a representation that mimics the target population. This is done by means of adjustments of different kinds. In the future, efforts will be made to develop robust quality measures for results based on nonprobability samples together with a development of methods that combine results from different sources, such as regular surveys, surveys based on probability sampling, big data, and data from information systems including social media.

3. Swedish Opinions-An opt-in panel

The year 2014 was called the super election year in Sweden. Numerous elections were conducted and we predicted a need for almost daily election coverage by media. Also there was a need for material that could be used by journalists to allow a transfer from case related coverage to a more data-driven journalism. Timely reporting is of course essential in most coverage of current events. Thus data collection had to be fast.

Aftonbladet, the largest newspaper in Sweden, approached our firm Inizio suggesting the development and implementation of a daily panel eventually called Swedish Opinions, which could serve the newspaper’s journalistic needs regarding current attitudes and opinions. We wrote a proposal that was accepted and the panel is now used in the following way.

During the morning editorial meeting various “research” ideas are tossed around. A statistical editor, trained by us, starts thinking about question wording and possible visualizations that might shed some light on the research idea. A web questionnaire is developed or an old one adapted to the research problem. The daily invitation to participate is sent to panel members. The inflow of responses is followed up, analyses are conducted and preliminary and final reports are produced. Adjustment to compensate for nonrandomness is done both at the sampling stage and after the data have been collected.

At the sampling stage it is possible to do prestratification. Typically we use gender, age and region to adjust for nonrandomness. After data collection, adjustments are made to compensate for varying response propensities across prestrata. Here we use poststratification by assigning different weights to different groups so that the distribution of respondents resembles that of the entire country.

Recruitment of panel members is done by using banners that pop up not only on Aftonbladet’s website but also on other sites associated with the media consortium, where Aftonbladet is just one of the members. The panel currently has more than 30,000 members.

During the development of the Swedish Opinions concept we have had access to the Advisory Board mentioned above consisting of a number of high-profile academic survey researchers.

4. Problems and challenges

The main technical problem has to do with the fact that systems available on the market are designed for general access panels rather than panels, where members are reinvited with a certain periodicity. Thus we had to adjust for that in the system we acquired on the market.

Methodological problems were more challenging. The recruitment was in fact so called double opt-in, where those who expressed an interest in panel membership had to confirm their membership before being included in the panel. Various background data were then collected regarding socio-economic status, and estimated frequency of site visits to improve final adjustments after data collection. To adjust for underrepresentation of specific groups we had to move the invitations around from time to time, basically by identifying parts of sites that might be of special interest to those underrepresented.

The main topic of these surveys has concerned party preferences and related parameters such as opinions regarding the political agenda and trust in politicians. Party preferences can be seen as very suitable for opt-in panels. There is a lot of auxiliary information that can be used to validate the results. Our results were in accordance with those of the university-based SOM Institute that specializes in surveys on election coverage and they were also in line with those of most other pollsters.

A number of adjustment methods were tested, including propensity weighting, raking, and even Bayesian estimation. The latter turned out to be too technical, especially for the newspaper staff to handle. That suitability assessment is valid also for raking. We ended up with poststratification using gender, age, region, and the results from the previous election. As a matter of fact all three procedures gave approximately the same result, which was another reason for choosing the relatively simple method of poststratification. Results have been stable and the method can be used by the newspaper staff. A large newspaper such as Aftonbladet has to work with a tremendous speed and any complexities have to be avoided.

Our goal was to create a fool-proof system that could not go wrong, but of course some mishaps have occurred anyway. There have been problems with the manual control from time to time but also some system bugs have been discovered. On at least one occasion results that should have been scratched were published anyway.

One of our goals was to show that our concept was sustainable, i.e., that it is indeed possible to build a survey system based on a panel recruited from the sites of a media consortium that can be handled by newspaper staff. We have come to realize, though, that there is a potential problem here. Survey researchers are interested in data about aggregates of people rather than data about individuals. Traditionally, newspapers tend to focus on data about individuals and reporting of case studies, partly because a lack of survey data. Data-driven journalism should be based on surveys rather than data on individuals, and this change calls for journalists having some basic skills regarding surveys and their error structures. That is one reason why we asked Aftonbladet to assign a designated statistical editor, who in turn can guide other journalists about the strengths and pitfalls of surveys.

Complementing case-driven with data-driven journalism is, however, not an easy transition.

5. Some observations

Most of our surveys are very short with just a few questions. This is good from a measurement error perspective. There is very little room for cognitive phenomena such as acquiescence, satisficing and straightlining. The survey is basically over before these behaviors start to kick in. However, primacy effects might occur. In some surveys we have used partial randomization of response alternatives to reduce the effects of that phenomenon. In any case, a daily panel like ours offers great opportunities for small controlled experiments concerning primacy but also regarding time of the day, stability of response patterns via reinterviews, where the same questions are given the same respondents on two occasions, as well as calendar effects, where response propensity might differ depending on day of the week and time of the day. Unlike many other surveys the response burden is light.

The big problem is of course that there is a lack of a theoretical foundation. In a Bayesian framework this would be less of a problem albeit complicated to set up and difficult to convey to those less mathematically inclined. A new theory would have to address issues of representation as well as measures of error margins. Several statisticians are already working on these things, among them Andrew Gelman and Doug Rivers, and we are sure that theory will eventually appear. Current practice regarding opt-in panels relies on adjustments that have proven empirically stable. One should bear in mind that classic survey methodology relies heavily on modelling and access to auxiliary information both at the design, implementation, and analysis stage.

Since the launch of our panel less than a year ago we have conducted approximately 150 surveys comprising a total of about 95,000 web interviews. The number of newspaper articles based on these results is 126. The number of questions asked to panel members amounts to approximately 500. Soon our work will be taken over by Aftonbladet’s statistical editor and the newspaper’s analysis unit.

To sum up, opt-in panels offer convenience, low costs, and light response burden but adjustment is necessary.

6. References

AAPOR (2010). Opt in online task force report.
AAPOR (2013). Report on nonprobability sampling. AAPOR web site.
AAPOR (2014a). Mobile Technologies for Conducting, Augmenting and Potentially Replacing
Surveys: Report of the AAPOR Task Force on Emerging Technologies in Public Opinion
Research. 
AAPOR (2014b). Social Media and Public Opinion Research: Report of the AAPOR Task Force on Emerging Technologies in Public Opinion Research.
AAPOR (2015). Big Data Task Force Report. 
Gelman, A., Carlin, J., stern, H., Dunson, D., Vehtari, A., and Rubin, D. (2014). Bayesian data analysis. Chapman and Hall.
ISO (2009). Access panels in market, opinion and social research. Standard 26362.
International Standards Organization.
Keeter, S. (2014). Change is afoot in the world of polling. Amstat news, October.
Wang, W., Rothschild, D., Goel, S., and Gelman, A. (2014). Forecasting elections with non-representative polls. International Journal of Forecasting.

---

Further reading within:News

View all within: News