In Budget 2007, The Government of Canada invested $46 million over four years for the creation of the Business-Led Networks of Centres of Excellence (BL-NCE) program. The goal of the BL-NCE program is to fund large-scale collaborative business-led networks to enhance private sector innovation in order to deliver economic, social, and environmental benefits to Canadians, and to promote an Entrepreneurial Advantage as described in the Federal Government’s S&T Strategy, in five priority areas : (1) Environmental science and technologies; (2) Natural resources and energy; (3) Health and related life sciences and technologies; and (4) Information and communications technologies and (5) Management, business and finance.
The first competition concluded in October 2008 to selected four networks funded over 5 years (2008-9 to 2012-13) as follows:
As planned in the initial Program Terms and Conditions (2007) and the BL-NCE Integrated Results-Based Management and Accountability Framework and Risk-Based Audit Framework (RMAF-RBAF), the first evaluation, completed in 2011-2012, focused on the first three years of the program (2008-2011) and on questions related to relevance and performance (effectiveness, efficiency and economy) to inform program renewal and improve delivery. The next evaluation is planned after the end of the funding cycle to review relevance, performance and cost effectiveness and will be conducted jointly with the summative evaluation of the NCE program.
The NCE Steering committee commissioned Performance Management Network Inc. (PMN Inc). via a competitive process to design and conduct the evaluation. A final report addressing the main issues of this evaluation was submitted to NCE Steering in February 2012. The response from NCE Management to the evaluation recommendations and the proposed action plan are detailed below.
Overall comments:
NCE Management recognizes that this is an evaluation conducted early in the life of the BL-NCE program to provide information for program delivery improvement and to inform decisions about the future of the program. This evaluation was based on multiple sources of evidence (document review, interviews and case studies). However, as it is based on early results obtained within the first half of the funding cycle, it is understood that this evaluation can only discuss progress towards results obtained within the first years of the program by the networks. It is also worth noting that, as a comparative element, the BL networks were compared to new NCE five-year networks and NSERC Strategic Networks at the same life cycle stage (2-3 years of funding).
Overall, Management recognizes the amount of work invested in this evaluation by PMN Inc. for the detailed analysis and review of all source of evidence provided. The evaluation report provides a sound and accurate summary and analysis of the information available. NCE Management agrees with the overarching conclusion that the program is relevant and brings added-value and that the model should be maintained, but with some improvements in its delivery.
Detailed responses to each recommendation are provided below and an action plan is presented in the following table, although most some of these recommendations may only be implemented if the program is continued by government.
Recommendation 1
The BL-NCE program is showing early success and the model should therefore be maintained at the federal level. The BL-NCE program is addressing a continued need for private sector led collaborative research and development and making progress towards the achievement of expected outcomes. It is still too early to firmly conclude that the program will achieve its objectives to increase private sector investments in research in Canada, support the training of skilled researchers, and connect the resulting ideas and talent to businesses seeking to bring innovations to market, particularly given the early stage of the program as well as the limited number of funded networks. However, the findings of the evaluation support the validity and further funding of the program model. The findings also support the involvement of the federal government in funding of the program model as such funding enhances the scope and nature of the funded networks.
Management Response: Agreed
Management agrees with this recommendation. This evaluation was carried out after two years of funding representing the mid-point of the funding cycle. The evaluation shows that the program is showing positive signs of progress towards reaching the objectives and is still relevant. As described in the evaluation strategy included in the Terms and Conditions of the program and the initial RMAF-RBAF, a second evaluation is planned at the end of the first funding cycle (in two years) which should confirm if the program has reached its objectives. If the program is not continued, this evaluation will still be required, to update data collected in this evaluation and to provide comparison data for the NCE networks program evaluation.
Proposed Action
NCE Management will ask the NCE Secretariat to plan for the next program evaluation. Results obtained in the last two years of delivery by the BL networks will be included in this evaluation to assess the overall progress towards program objectives.
Recommendation 2
If renewed or extended, the NCE Secretariat should consider the following to enhance the program’s ongoing relevance and effectiveness. First, allow existing networks to re-apply in future program competitions as there will likely still be an ongoing need for federal government support to these networks to achieve program outcomes. Second, focus on steps to solicit applications for networks in priority areas not funded to date to improve the alignment of the program with priority areas and private-sector needs (i.e., in the two priority areas not yet funded). Third, provide more support for the development of network applications and the implementation of funded networks to help mitigate and/or lessen the challenges that have adversely affected network implementation and operation to date. In terms of support for network implementation, this could include identifying the types of expertise and resources required to implement a business-led network as well as providing additional assistance with the development of network agreements. With respect to the application process, additional emphasis could be placed on assessing the required expertise and resources in subsequent program competitions by revising the program’s assessment criteria and application requirements.
Management Response: Agreed
Management agrees that the extension of on-going networks may be useful to maximize the results of the networks. This has been a practice in the NCE program and may also be implemented in other programs delivered by the NCE Secretariat. If new funding is allocated for continuation of the program, Management will need to examine the allocation of funding to existing and new networks.
With regard to the priority areas, Management may need to examine which priority areas would be more relevant to government priority and private sector needs. One option may be to launch a new call for proposal in all five Budget 2007 priority areas or to focus only on the priority areas that were not covered in the last round of funding. Another may be to review and define new priority targets aligned to government priorities and private sector needs.
Management fully agrees with the recommendation to develop more support to guide implementation and operations of existing or new networks. Program documents and guidelines were developed at the beginning of the program and were built on the policies and processes associated with academic grants. As the BL-NCE program continues to adapt to the needs of the business-led grant recipients, the NCE Secretariat will further engage the Private Sector Advisory Board (PSAB) to revise the documents and guidelines should the program be continued.
Proposed Action
Evaluation Recommendation 3
The BL-NCE program’s expected outcomes and performance measurement strategy should be revisited. While the program theory appears appropriate, based on the nature and performance of the four networks funded to date, there is a need to revisit the extent to which and the timeframe in which some expected outcomes can be achieved. Further, findings from the evaluation indicate a need to revisit the program logic model and performance measurement strategy. The evaluation found that there is a need to establish a better linkage between the network level outcomes to program outcomes. Therefore, more work is needed to better demonstrate how the outcomes of individual networks are contributing to program outcomes. This should involve further refining the expected outcomes in the program logic model. This should also involve revisions to the performance measurement strategy as well as assessment of the quality and comprehensiveness of the data collected to improve the relevance, appropriateness and reliability of performance indicators used to measure both network and program performance. Revisions to the performance measurement strategy could be informed by a review of the performance data already collected as well as continued work with the four BL networks.
Management Response: Agreed
NCE Management recognizes the need to review the logic model and Performance Measurement Strategy (PMS). The logic model and the RMAF-RBAF were developed based on the NCE program model and at the start of the program. They will need to be revised, based on the information obtained in this evaluation to better report on the successes of the program and make linkages between the network’s and the program’s performance more evident.
Proposed Action
If the program is continued, the PMS including the logic model, expected outcomes, performance indicators, as well as the monitoring and reporting tools, will be revised, to include performance indicators common to all networks representing program expectations, and specific ones representing network’s own objectives. The NCE Secretariat will consult the PSAB, currently funded networks and other stakeholders to update the performance indicators and reporting tools.
Contacts
For further information on the Program Management Response, please contact Stephanie Michaud, Assistant Deputy-Director, BL-NCE Program. For information on the evaluation, please contact: Susan Morris, Director, Evaluation, NSERC.
NCE- BL-NCE Program Management Response (Summary)
Recommendations | Agree/ Disagree | Proposed Actions | Responsibility | Timeline |
---|---|---|---|---|
Recommendation 1 | ||||
The BL-NCE program is showing early success and the model should therefore be maintained at the federal level. |
Agree |
Whether the program is maintained or not, the NCE Management will ask the NCE Secretariat to plan for the next program evaluation at the same time as the NCE evaluation as planned. |
NCE |
Sept 2014 |
Recommendation 2 | ||||
If renewed or extended, the NCE Secretariat should consider the following |
Agree |
If the BL-NCE program is continued, NCE Management will review budget options to inform the decision of continued funding for existing networks vs. funding of new networks. Furthermore, the NCE Management will task the NCE Secretariat with engaging with government stakeholders regarding the identification of priority research sector funding areas. |
NCE Secretariat |
Nov 2013 |
Management will review and revise program documentation to ensure better support to its networks. | Dec 2013 | |||
Recommendation 3 | ||||
The BL-NCE program’s expected outcomes and performance measurement strategy should be revisited. |
Agree |
If the program is continued, the Performance Measurement System (PMS) including the logic model, expected outcomes, performance indicators, and the monitoring and reporting tools, will be revised, to include performance indicators common to all networks representing program expectations, and specific ones representing network’s own objectives, in consultation with the Private Sector Advisory Board (PSAB), currently funded networks and other stakeholders. |
NCE Secretariat |
Jan 2014 |