The good news is that there are more than a dozen product development partnerships (PDPs) working to develop products to diagnose or treat the diseases that afflict many of the world’s citizens. Started in the 1990s and funded primarily by donor dollars (a total of about $450 M through 2007), the PDPs have been making incremental progress on more than 60 projects with a handful of products now in use/commerce (Dahlberg/IFPMA). While their progress is appreciated, it is reasonable to expect the PDPs also use their funding as effectively as possible and report on their overall performance, as was pointed out last November by Dr. Melinda Moree, CEO of BIO Ventures for Global Health (Moree Post) and last week by her colleague, Dr. Orin Levine (Levine Post). Such performance measurement would be a useful tool for PDPs to evaluate their internal project portfolios and operations in the near term and, in the long term, should we be so lucky as to have multiple PDPs (or companies) pursuing similar targets and competing for funding/investment, provide a way for donors/investors to make more rational donation/investment decisions.
I found one published study of the evaluation of PDP performance, a 2007 report by FSG Social Impact Advisers (FSG Report) which was supported by the Gates (thanks, again, Bill and Melinda) and Rockefeller Foundations on behalf of the “PDP Funders Group,” apparently a group of PDP donors formed in 2004 [but now missing-in-action; anyone have an update on this group?]. The authors conducted extensive research (interviewing representatives of more than 50 organizations and companies) and found:
-donors vary widely in the breadth and depth of performance oversight;
-current PDP performance measurement methods lack structure and comprehensiveness; and
-although PDPs monitor R and D project performance, they differ considerably in the way they do it and in the way they use the results in managing or explaining their activities.
However, the authors step back from proposing a evaluation methodology; rather, they propose adoption of an “evaluation framework” that they say “serves as the first tangible manifestation of a more integrated and collaborative approach to performance measurement” and “provides a platform to enable each PDP and its donors to engage in discussions around performance issues and decide jointly on the specific metrics that best reflect the unique priorities of individual PDPs.” The FSG bottom line recommendation is yet another group be formed (“a new ‘performance measurement partnership’ be established between donors and PDPs to improve the value of donor-sponsored evaluations and internal management reviews by PDPs”). Not useful.
Surprising to me is that FSG Advisers (or others in the global health field as far as I can tell) did not reference the most obvious source for measuring of R and D performance: the pharmaceutical industry. Many techniques have been developed and used internally to evaluate performance and to manage R and D projects and portfolios and downsizing/merging. Externally, similar techniques are used by professional investment firms to rate investments and conduct due diligence. (A few source of information are below.)
I am assuming that PDPs have adopted of these methods to evaluate, reassess, add to, and trim their project pipelines, but don’t publicize their methods. After all, one of the mantras of biotech/pharma project management is that it is better to eliminate projects quickly and early and redirect resources to other opportunities than to concentrate on a few, increasingly expensive projects. PDPs are/should be doing this. In addition, a widely-used external evaluation method will be needed as new sources of funding for the PDPs emerge to allow inter-PDP comparisons. These include:
-a PDP “Financing Facility” as proposed by IAVI and other PDPs to the WHO Expert Working Group for Financing R and D (http://www.who.int/phi/public_hearings/second/contributions/HollyWongInternationalAIDSVaccineInitiative.pdf);
-program-related investments as being implemented by the Gates Foundation (my posting of January 14).
PDP performance measurement: what am I missing?
Business Insights report 2005: http://www.globalbusinessinsights.com/content/rbhc0138t.pdf
Journal of Commercial Biotechnology article 2008: http://www.palgrave-journals.com/jcb/journal/v14/n4/abs/jcb20086a.html
Portfolio Management Conference 2008: http://www.healthtech.com/Conferences_Archive.aspx?id=78504
Consulting company primer 2010: http://www.ngpharma.com/article/A-Strategic-RD-Portfolio-Management-Primer/