R&D Best Practices: A Study of Top Performers
Which PPM Best Practices give leading R&D organizations a competitive advantage? Michael Menke's ongoing PPM Accelerate Benchmark study was designed to provide the answers.

 

Numerous PPM best practices aim to make R&D more productive and ultimately generate more value for the organization. But which practices are organizations doing well, and which ones are the most impactful?

In 2011, Dr. Michael Menke started an international portfolio management benchmarking study, PPM Accelerate, to gather extensive data on the usage of 50 commonly accepted PPM best practices across multiple industries with the objective of providing insight as to the importance of these practices and their effect on PPM performance.

In this video Dr. Menke discusses how the study was framed, which organizations were originally involved, and what the raw data revealed.

This is the 1st part of a two part series on Michael Menke's PPM Accelerate study. For more information about insights from the study, refer to the video "Critical Best Practices for R&D Portfolio Management".

Planisware has also teamed up with Dr Menke to offer our readers a chance to take an excerpt of the study focused on Resource Management, and review the results with Dr Menke.

 

The link between Project Portfolio Management Adoption and value creation

There are a number of overarching objectives of portfolio management. But indeed one of them surely to create business value. And this is very hard to prove because in R&D, we're only creating the potential for our businesses to really go out and earn money that gets reflected in our earnings, our share price, etc.

But a number my colleagues from Strategic Decisions Group who worked with most of the global pharmaceutical industry, and had a good idea of who was really seriously adopting portfolio management – this was mid-90s to early 2000's – created an index of the strong adopters, which you see here compared to the Dow Jones pharma index, which would be basically all publicly traded pharma companies in the S&P 500 over the same time periods.

And what we can conclude from this graph [is that] there's a strong association of portfolio management usage and value creation in the pharmaceutical industry where managing the R&D portfolio, well, is absolutely critical to future business success. I'm not saying by the way that this proves causality, but there's clearly a strong association, which is – given that we're talking R&D portfolio management and the ultimate business results are pretty far in the future – I think that's about as good a test as we can get.

What influences decision-making?

Now portfolio management's [aim] is to support good management decisions on which projects and programs get funded, and which, although they may be very good, have to be set aside.

What influences decision-making generally, not just in portfolios but any decision? Well the culture and incentives in the organization are probably the most powerful drivers of decision behavior. But then there's behavior itself which is the more explicit manifestation of culture and incentives. Then there could be a formal decision quality framework and I'll use "DQ" in the presentation for Decision Quality. There is the decision making process. And then finally there's your decision making or your decision quality methodology and tools.

In the benchmarking, I'm going to report on today... it's really focused there in the middle: Decision Behaviors, Decision Quality Framework, Decision Process, and the practices that make those things come along. So, the framework applies generally to making quality decisions, but also to portfolio decisions. Now portfolio and incentives are extremely important as I said but they're very hard to change. That's why I decided to focus on decision behaviors and practices, Decision Quality Framework, Decision Process.

Introducing the PPM Accelerate Benchmarking study

And this benchmarking is still underway in a series of round since 2011. I have a European partner, so we got companies from the US, companies from Europe. Even a Japanese pharmaceutical company, Takeda, participated.

And so we now have a database on 50 best practices where we ask people about their importance, how well they're executed, and establish performance standards on those 50 practices.

The 50 practices by the way largely were drawn from prior benchmarking studies, including a very large study done with the Industrial Research Institute  in the early 90s by Bob Cooper and his colleagues that resulted in their well-known book  "Portfolio management for New Products". So these best practices were not something that I invented or created. I really largely extracted them – many of them have been proven in prior studies.  What we wanted to do was test how important people felt they were, and how well they were executing them.

Now to establish a standard of excellence we made sure – and I know a lot of companies and have worked with a lot of these companies – we made sure to include a number companies that were very mature and were leaders in the use of project portfolio management. We now have data on over seventy organizations including many leaders, and we're continuing to enroll new organizations.

This shows you the people who participated in rounds one through three. Some of these organizations have multiple participants. So, for example you see Dow, several different units of Dow, several different units of Bayer, three different units of Rockwell Automation, and so on and so forth. A wide range of industries and functions and, as I said, the study is ongoing.

The 50 PPM Best Practices

So we grouped our 50 best practices into 8 categories. They all have a letter code that goes with them, which you can see on the pyramid here.

  • Category A = Added Value of PPM = How do we create value? Why are we doing PPM?
  • Category B = Analytics, Reporting and Risk Assessment
  • Category C = Management decision behavior
  • Categories D, E and F = Financial Information, Resource Information, and Strategic Value Information. And I think I would call each of those... I would substitute now "Information" with "Information and Management", so it would be "Resource Information and Management", "Strategic Value Information and Management", etc.
  • The foundation of the pyramid: Category G = PPM Organization and Governance and Category H = the PPM Process itself.

Examples of PPM Best Practices

Just to show you an example of what the practices are.

The first one is practically the thesis of Bob Cooper's book, and that is: A1 – Pursue three overarching objectives of portfolio management: 1) Strategic alignment, 2) Strategic balance and 3) Maximum return.

B5 – Show the impact project risk on future project and portfolio value.

C2– (that's the decision behavior category) Decision making by management is knowledge-based transparent and consistent. I'm sure that's true all your organization's!

D3 – (pretty obvious) Measure the strategic and financial value portfolio decisions using a business case. And that actually means a business case for the projects, not a business case for the portfolio.

E3 (one of the most important and most weakly actualized or most poorly executed of all the practices) – Do not overload the project pipeline or the people. Resource the projects adequately. Everybody says: "This is really important". And almost everybody says "We're not doing a very good job of that."

F1 (another practice needing improvement) – Have a well-defined business strategy and communicate it to all employees clearly and often. You would think that that's obvious, and basic, and everyone does it. What people actually say is [that] they're not doing it nearly as well as they think they should.

G1 – Portfolio governance should be clearly defined and understood.

H1– Use consistent PPM process, language, and tools. How did we score the practices? Well basically, we asked each participant four questions with a fairly simple scoring system. First of all: Is relevant or not relevant to achieving your portfolio management objectives. And if it is relevant, is it actually something that's formally part of your portfolio management process?

Scoring the 50 PPM Best Practices

In which case if it was formally specified that you do it, we called it a "Core practice". So this could either be no relevant, or core.

Then we asked for the contribution: if it's relevant, what is its contribution toward achieving your objectives ranging from "not at all important" to "extremely important" on a scale of 1 to 7. So those were the questions about importance of the practice.

Then we asked two questions on usage: 1) How frequently do you use it when appropriate, and 2) how well you execute when you use it. And keep in mind throughout the rest to the presentation that we combine frequency of use and quality of execution in... by multiplying the two of them, each on a scale of 0 to 100 percent to get a number we call "actualization". So actualization, you'll see a lot it's our major performance metric. It's a pretty simple scoring mechanism but it produces some pretty nice insights.

So the participants' answers validate the best practices as important, and they establish a clear performance benchmark. The data on this slide are based on rounds 1 through 3 which had 44 participants. Round 4 had about 30 oil companies in it. That's why we're up to over 70.

Results of the study

So based on rounds 1 through 3, all except five practices are considered core by 15 or more out of 44. And 10 by 30 or more. So that's core, that's not just relevant but it means these are specified is things that we do in our PPM. Every practice is considered relevant by at least 25 of the 44 organizations, and 16 of the 50 practices by 42, 43 or all 44. Remember this is the 44 participants in rounds 1 through 3.

6 of the 50 have an average contribution at 6 or higher on a scale of 1 to 7, and to be between the highest and the second highest on a 7-point scale, people were saying this is really important.

And only 6 had an average contribution below 5. So you can see these answers were strongly skewed toward the extremely important very important rather important... you know the high end of the scale.

On the other hand, actualization – again: the product of frequency of use times quality of execution – drops fairly rapidly from 66% to 33%. Pretty significant range.

The Best Practices can be turned into reality

I then said: "Well let's take for each practice the three best organizations, and average their actualizations." And that top-3 group is usually above 80%, and on a number of practices, it's 100 percent.

The message? Organizations can do these practices if they choose to, if they want to. And as you'll see later, the practices that are important but have low actualization – in this case we took below 45% as are fertile ground to explore for gaining competitive advantage.

Just as an example, averaging all practices over all organizations in a particular grouping, the total population – again this is the... rounds 1 through 3, 44 organizations – [the] total population have an average actualization about 46%.

Results by industry

R&D organizations perform better. Life science organizations – mostly pharmaceutical companies but including one agro-science company – perform even better. And just as a point of reference, IT organizations tend to perform not so well relative, they're below the total population and well below R&D and life science.

Establishing the performance benchmark

But now if we say: "What if we averaged the top three actualizations over all 50 practices?", we get 88%. So it's really that top 3 actualization which is a different three companies for each practice. That's what establishes our performance benchmark.

So here's a graph, practice by practice, that shows you what's the average performance, and again this is based on the initial 44 organizations and what's the average the top three performers. And what you tend to see is the top three are about 40% higher on actualization than the average of everybody. So there's a very substantial gap between average performance and top performance.

And that becomes our best in class and again I remind you that actualization is the product of how frequently do we use it times how well do we execute it, each scored 0% to 100%.

Contribution vs. Actualization

This is a little complicated, but it shows all the practices with contribution – the importance metric – on the horizontal axis and actualization – our performance metric – on the vertical axis.

And then we first look at a quadrant where importance is above 5.4 average importance and actualization is above 50%. And we see in there H1, H2, H3, H4 = That's the PPM process category. We see G1, G3, G5 = that's the governance category and then we see you selection of other practices. So these are practices that are almost table stakes to do good portfolio management.  The average company is doing them pretty well. You can imagine that the top performers, the top three, are doing them very close to 100%, some of them probably over 100%.

But let's now go down look at the lower right quadrant, instead of the upper right quadrant. This is practices that people think are indeed important – they make a high contribution to achieving our objectives. But we don't feel that we're doing them very well. So what we see here: we see E1, E2, E3 = that's the Resource Management category Remember that E3 is "Do not overload the pipeline."  So you see that is on of the weakest of all the practices even though people know they should not be overloading the portfolio, they do it anyway. We also see F1, F2, F4. F is the strategy category so people are saying "Yes we know it's important to have a clear strategy and to communicate it well. Look it's almost 6. But we're not doing it nearly as well as we should, we're down around 37% And so on.

So, both the Resource Management category and the strategic information management category tend to be weaker categories needing improvement. Now, this may not be the case in your organization but this is what our average finding is. So, if you can do these much better than the average company, you have a chance to gain competitive advantage.

Summary of key findings

Just to summarize the key findings: Who does portfolio management well?  Well it was no surprise to me because it's so critical in that industry that pharma and healthcare was the best performing industry group. And in terms of functional findings, organizations executing mostly R&D projects score quite well. Organizations doing mostly IT projects as a comparator, do worse.

What's being done well? People use both quantitative and qualitative measures. They have well established and implemented criteria for decision-making. There's pretty good alignment with regular planning and control processes, things like budgeting. And the core categories of PPM process and PPM organization and governance are being done fairly well. But there are some issues in organization, as you'll see when we talk about improvement, that are a little on the weak side.

Main areas needing improvement? Again, you've seen some data for this. Insufficient integration of PPM and resource management. Portfolio not well aligned with strategy, or even strategy unclear.  People tended to feel they were doing nearly as good a job as they could with risk, how to consider risk both at the project level and the portfolio level. People didn't feel that their portfolio management was well aligned across different tiers of the company, or different functions in the company. So there may be centers of excellence, but the process may be different in different centers of excellence.

And finally: benefits management is frequently not executed, you know, once the project is handed off what's being done to make sure that the benefits anticipated get realized. In the results from all roads have been very similar on these points.