PHILADELPHIA, PA  – Organizations entrusted with the evaluation of press freedom need to evolve to keep pace with new technologies for disseminating information, such as cell phones and other mobile communication devices, as well as evaluate their methodologies to ensure their ratings systems are valid and useful.

            Those are some of the findings of a study, “Evaluating the Evaluators; Media Freedom Indexes and What They Measure,” undertaken by the Center for International Media Assistance (CIMA) at the National Endowment for Democracy and the University of Pennsylvania’s Annenberg School for Communication. The report, written by former Washington Post reporter John Burgess in cooperation with the Annenberg School for Communication’s Center for Global Communication Studies (CGCS) and CIMA, is based on a collection of research on this subject conducted by Annenberg since 2007.

            “All over the world, studies that rank countries by media freedom figure prominently in civil liberties debates, help determine levels of financial aid … and impact an array of foreign policy decisions and academic research,” Burgess wrote.

            As use of these indexes expands, they draw increasing attention from academics who are trying to judge the quality of the underlying social science. “Precisely because these are such important institutions, it is desirable to try to shape a critical discourse about their work,” said Monroe E. Price, Director of CGCS at Annenberg.

            The paper examines the practices of the “big three” evaluators – the International Research & Exchanges Board (IREX), Freedom House, and Reporters Without Borders.  Its recommendations, which are applicable to governments and financial donors as well as the three organizations, include:

• Evaluators should continue to increase technical sophistication, validity of their data across time, and transparency of sourcing. Longitudinal data is valid only when questions are consistent.
• Organizations that fund the studies should ensure that financing is adequate.  Freedom House, for example, has recently had to cut back on its evaluations due to a lack of funding. 
• Governments should resist the temptation to dismiss studies and ranking of media freedom in their countries as outside interference and should consider the findings seriously when crafting media policies.
• Media aid practitioners should be careful when linking a country’s performance with a specific project.  
• Evaluators should find ways to measure conditions of freedom with respect to the Internet, mobile phone texting, and other digital technologies.  The report notes there are already 4.6 billion mobile phones in use; how is their application affecting press freedom?

The full report is available by visiting the CGCS web site.