Communications and Impact Metrics for Think Tanks

Based on a presentation made at the conference “Think Tanks – Facing the Changing World,” hosted by the Chinese Academy of Social Sciences in Beijing, June 17-18, 2013.

Today, many of the world’s 5,500 think tanks are seeking more effective ways to communicate, to increase their impact – and exploring better ways to measure that impact.

My views on these tasks are shaped by 35 years in communications, including in newspapers and news websites, as well as my work these past three years with an independent, non-partisan global think tank, The Centre for International Governance Innovation (CIGI).

For any organization, including think tanks, good communications begin with

the creation of an overall strategic plan. This may seem obvious, but any enterprise is more likely to succeed with a clear mission and goals (many of us can identify cases where a muddy plan led to poor results). Mission is a definition of purpose. Goals define what success will look like: the desired impact. Tactics are the actions necessary to achieve those goals. It helps everyone in the organization if a strategy combining these elements in a logical fashion is written consultatively, then shared internally, so that each person can see how his or her work contributes to the overall plan.

A traditional view of think tanks is that their strategy requires them to conduct research and analysis to develop policy ideas, and then communicate their policy ideas both directly and indirectly. They can communicate directly, to policy makers who exercise power by making decisions. They can also communicate indirectly, to policy influencers, such as the media, scholars and citizens.

One challenge, however, is measuring the influence of think tanks, especially in the areas of policy impact, to assess whether the strategic plan was successful. The problem is one of attribution — who gets the credit for a policy that is implemented? Policy input comes from many places. Public or governmental policy development is a complex and iterative process in which policy ideas are researched, analyzed, discussed and refined — often through broad consultations with many stakeholders. When a policy is finally adopted, it may wear the fingerprints of many hands. For these reasons, a think tank cannot always claim success and say, “this policy was our idea.” In many cases, it would be highly unusual for a political leader to give credit to a particular think tank for a specific policy; such leaders must take ownership of their own policies, to be accountable for them.

In creating impact, a think tank can extend its role beyond that of conducting research, analyzing and identifying policy problems or sharing policy ideas. For example, think tanks also have the ability to convene meetings of different groups at conferences, seminars and workshops — to connect people and to facilitate dialogue. As conveners, think tanks have the ability to build bridges among diverse groups such as policy makers, non-governmental organizations, academics, business leaders and the media. In this way, think tanks can create a sort of “Track II” process — a catalytic role in which the think tank’s own influence is, once again, hard to measure. Think tanks may also have a role in education; through training programs, education and outreach, think tanks can help to develop the next generation of diplomats, bureaucrats and political leaders.

In communications, it is important for think tanks to reach the right people, with the right message, using the right method. Think tanks use a variety of communications channels — as different channels may be more effective with certain audiences. To reach top leaders, for example, an ineffective method would be to rely on academic-style research papers — because high-level leaders are busy and have little time to read.  The best method of outreach to senior leaders might be small meetings to present research findings in person – but this depends on having access to leaders, through a think tank’s network of people with excellent connections. Meanwhile, middle-level officials can be reached through multiple channels, such as conferences, workshops, papers and policy briefs (research papers might be 5,000 to 10,000 words, or more; but policy briefs are shorter documents of 1,000 to 1,500 words, which distill the key policy recommendations into a few concise findings or policy recommendations). Academics and scholars are more easily reached through well-written research papers and scholarly books. The wider public can best be reached through accessible websites and through the news media. For outreach through news media, think tanks must deploy skilled communications specialists to create and send news releases written in journalistic style, and who will follow up personally with journalists with whom they have developed relationships through regular contact. Other channels of communications include social media, newsletters (including email newsletters) and annual reports — each suitable to a particular audience. Good communication plans use a combination of all of these channels to achieve the greatest impact.

The “Cycle of Impact” for a think tank has three phases. The first phase is to Plan. Researchers within think tanks consult with policy makers to better understand the challenges and issues those policy makers are facing; they design projects to address those topics, and the design includes an allocation of resources, budgets, staff and timelines. The second phase is to Engage. The think tank may engage in deep research and analysis of the topic, including the historical context and policy options; it may also convene conferences and public or private meetings as necessary; and it may communicate its findings through publications, websites and social media. The final step is to Measure. The think tank may track the quantity of outputs in publications, media mentions, website traffic and social media hits; it may evaluate the quality of the outputs (even if this is a subjective judgment) and it may even try to assess the actual impact on public policies (although this raises the difficulty of attribution, as discussed earlier); and it may report on these measurements to stakeholders, such as funders of the think tank. The third phase is the easiest to overlook, but measuring outcomes can yield valuable lessons to help a think tank improve its work.

We can think of many things to measure at a think tank. What follows is a list of 15 possible metrics, as suggested by various experts on think tanks — and unfortunately, the more useful ones to consider may also be the hardest to measure in exact numbers. These metrics can be grouped, with the first five metrics being measures of Exposure, based on an assumption that more influential think tanks are more exposed to public view.

1.Media mentions: These are citations of the think tank, by name, in media such as newspapers and news websites. Some third-party services can be hired to measure citations, or think-tank staff can search the Web with Internet search engines. Online searches are imperfect, however; they may not capture references that occur in traditional print only, or on television or radio; and they may miss citations behind pay walls or other security measures.

2.Number and type of publications. This is strictly a quantitative measure of the think tank’s publications, and does not evaluate the actual content of the publications as being of a high quality or not.

3.Scholarly citations. These include citations of the think tank’s work in academic journals.

4.Government citations. These include citations of the think tank’s work in government meetings or official party proceedings.

5.Think tank ratings. How did the think tank fare in annual ratings, such as those produced by the University of Pennsylvania? Some critics see such rankings as mere popularity votes, based on perceptions only, with methodologies that do not take into account different structures, funders, missions or other characteristics of think tanks. Nevertheless, the ratings do garner considerable attention.

The next group of metrics looks at Resources, based on the assumption that more resources allow a think tank to exercise more clout and, hence, achieve more influence.

6.Quality, diversity and stability of funding. The source of its money may reflect on a think tank’s independence, support and connections.

7.Number, experience, skills, reputation of experts, analysts and researchers. It’s easy to count heads, but reputation is a subjective quality and harder to measure.

8.Quality and extent of networks and partnerships. Influence is not just a question of who you are, but who you know.

The next group of metrics concerns Demand — that is, does anyone actually want to see or hear from a particular think tank?

9.Events. The number of conferences, lectures and workshops, and the number of attendees (both of these are a simple quantifiable measure). Harder to measure is the quality of the attendees. Are we just filling the room or are we attracting influential opinion leaders, powerful policy makers and top-level experts?

10.Digital traffic and engagement. Number of website visitors, page views, time spent on pages, “likes” or followers.

11.Official access. Number of consultations with officials, as requested by the officials themselves.

12.Publications sold or downloaded from websites. This is not the measure of output, but rather the external “pull” on the publications.

The final group of metrics considers Policy Impact and Quality of Work. These may be the most important things to measure, but also are among the most difficult to quantify.

13.Policy recommendations considered or actually adopted. As discussed previously, this is a problem of attribution. A think tank may say it put forward an idea, but if others had the same idea, who gets the credit if a policy is implemented?

14.Testimonials. Praise, criticism or other assessments of a think tank’s work can be collected through interviews with policy makers or recognized experts; this work can be done by external, independent evaluators, reporting to the think tank’s board or funders. As well, opinions about the think tank can be collected through formal surveys of the organization’s event attendees or subscribers to its newsletters and publications.

15.Quality of the think tank’s work. This is the most subjective of all metrics, but criteria for quality can be developed and defined, and placed on scales (such as from 1 to 10). How good were the publications in terms of readability and insight? How relevant were the projects and outputs to real-world problems and issues? How effective is the think tank in communicating its messages? Again, external and independent evaluators can be hired to make these highly subjective judgments.

In summary, to achieve maximum impact, think tanks should develop an overall strategic plan for the organization, plan their research projects consultatively with policy makers, engage their audiences through channels that are carefully designed to reach the right people using the right method and, finally, measure the outcomes of their work to ensure the goals were met.

Source: CIGI publications.