2006-2010 Statewide Comprehensive Plan for Mental Health Services
Chapter 7
Focusing on Performance Management - Promoting Positive Outcomes

View Adobe Acrobat Version  |  Download Adobe Acrobat Reader

The final phase of the balanced scorecard approach focuses on using the "balanced scorecard" for actually measuring and reporting on the outcomes experienced by individuals served within the New York State public mental health system, results of public mental health efforts undertaken by the agency, and critical indicators of organizational performance. The balanced scorecard is aimed at improving accountability and ultimately at supporting recovery, wellness and the transformation of the New York State public mental health system.

The previous chapters have framed the ongoing refinements to the Strategic Plan; provided a structure for examining environmental trends, challenges and opportunities; and permitted the systematic identification of strategic priorities to mirror the concerns of stakeholders and to take into account the environment in which the Office of Mental Health, (OMH) operates.

This chapter describes the new balanced scorecard methodology for gathering together data collected by OMH, and displaying it in a way that permits stakeholders to examine progress in strategic areas, monitor the performance, and use data to inform decision making. Historically, OMH has collected and reported results of performance through a variety of formats such as evaluation reports, Statewide Comprehensive Plans, progress reports, professional publications, technical reports and others. The balanced scorecard takes the best of the balanced scorecard strategic planning approach, previous and existing methods for reporting and presenting data graphically, cross-divisional collaboration and stakeholder partnerships, and advances in technology to meet three fundamental goals:

• To combine existing data sources into a secure integrated database and balanced scorecard platform for performance analysis, reporting and monitoring
• To disseminate the performance data online and through other mechanisms such as formal written reports to educate stakeholders and guide advocacy and awareness efforts
• To establish performance benchmarks to spur quality improvements, to inform management decision making and policy planning, to provide a basis for evaluating progress and making necessary adjustment to achieve performance targets, and to develop recommendations for future mental health care initiatives

Overall, an important goal is to enhance our processes across the agency's operational divisions by integrating knowledge from the areas of performance measurement, strategic planning, and quality improvement. The balanced scorecard provides the structure for identifying measures of performance and setting performance targets; utilizing technological innovations to automate data collection, analytic and reporting capabilities; translating OMH's scorecard approach throughout the organization; determining the degree to which results match expectations; and making adjustments to continually improve performance. To better understand terminology used in describing the balanced scorecard, please see Figure 7.1.

Figure 7.1

Commitment to Quality Processes

OMH recognizes that performance management is a fundamentally important tool for improving organizational capability, meeting broad organizational goals and objectives, and delivering high-quality policy advice and program administration for New York State's public mental health system.

In the 2005 Statewide Comprehensive Plan, OMH committed to expanding its performance improvement structures to foster cross-divisional review of performance data in relation to the OMH Strategic Plan Framework. The primary mechanism for this expansion has been the Performance Improvement Committee (PIC). Consisting of high-level managers from all parts of the agency, the Committee, also known as the PIC, has been charged with integrating the performance management and planning processes necessary to achieve Strategic Plan goals and objectives.

OMH has made significant progress this year in strengthening its performance improvement infrastructure and preparing for the work of the PIC. A notable accomplishment has been the deployment of "portal technology" for flexibly developing a web-based platform capable of producing reports using information contained in underlying databases. The initial results of these efforts will be unveiled in March 2006 and more information on the release of the scorecard is described further in this chapter. The PIC and the web-based balanced scorecard have been instrumental in enabling OMH to focus its attention on the collaborative, systematic monitoring of progress and encouraging reliance on the scorecard approach throughout all offices, centers and divisions within OMH.

Performance Improvement Committee

The PIC is at the heart of OMH's mission to transform the agency's various existing performance measurement processes into a comprehensive performance measurement system that provides accountability for results.

The Center for Information Technology and Evaluation Research has teamed up with the Division of Planning to lead the Committee's work and to provide guidance to the balanced scorecard technical development team. Drawing upon senior-level representation across the various departments and divisions of the agency, the PIC is charged with developing recommendations for review and approval by the Commissioner and the executive leaders within her Cabinet.

As components of the scorecard are developed, stakeholder input will be continuously solicited from the Commissioner's advisory committees including the Multicultural Advisory Committee (MAC), the Commissioner's Committee for Families, the Recipient Advisory Committee, the Mental Health Services Council and the Mental Health Planning Advisory Council (MHPAC). OMH has also built into the process solicitation of input from the stakeholder community through its annual public briefings, hearings and numerous other forums and opportunities.

Designing performance measurement systems is a highly complex endeavor that requires numerous decisions about technical issues, such as the selection of performance indicators, the measures and protocols for data collection, and the information systems/software design and development for data collection, management, analysis and reporting. Thus, the PIC's responsibilities include a diverse set of challenging tasks:

• Use input from stakeholders and recommend short- and long-term goals within the Strategic Plan.
• Identify and recommend specific management objectives, performance measures, performance indicators and performance targets to monitor progress toward goal achievement.
• Review data from the performance indicators, assess whether desired changes are occurring, determine whether established performance targets are being met, and routinely convey the results to the Cabinet.
• Identify and recommend management strategies and interventions most likely to prove beneficial in achieving individual goals.
• Review and make recommendations regarding proposed action steps and organizational unit work plans for these strategies and interventions.
• Monitor the implementation of specific work plans, review their subsequent impact on outcomes, and make recommendations to the Cabinet routinely.
• Review and recommend appropriate performance benchmarks and protocols for managing performance within the balanced scorecard framework.

The PIC has been given a broad advisory function to the OMH Executive Cabinet. In this regard, the PIC has assumed responsibility for assessing and monitoring the degree to which the OMH balanced scorecard accurately reflects the agency's mission, vision and values and making recommendations for optimal effectiveness.

Web-based OMH Balanced Scorecard

As detailed in Chapter 3, the balanced scorecard evolves from a dynamic, continual and comprehensive strategic performance management approach that encompasses mission, strategy and perspectives. Substantial efforts and resources have been dedicated to preparing for the launch of the web-based balanced scorecard, which will provide stakeholders with accurate and up-to-date information about progress made toward the agency's Strategic Plan goals.

Brief review of the infrastructure

OMH has invested several years in developing a multi-level, integrated, state-of-the-art information technology infrastructure that makes the balanced scorecard possible. A brief overview is presented here to provide the reader with a conceptual understanding of the relationship between operational data and the provision of useful performance measurement information. The overview will also indicate how recently developed information technology resources at OMH support this translation.

Simply put, the architecture of the balanced scorecard as depicted in Figure 7.2 shows, starting from the bottom of the graphical representation, operational data collected at the Local and State levels. The middle or integration layer is that part of the system that utilizes data sources from the operational level, extracting this data and reorganizing it in the central repository called the OMH Data Warehouse. The warehouse is designed to support the analysis and display of data in a manner that enhances policy decision making and performance monitoring. The portal technology is leveraged to use the data from this layer of the architecture to make it possible and practical to continuously produce current and "real time" web-based performance reports efficiently, as illustrated at the top layer of the structure. Figure 7.2 also shows the ability of the system to incorporate data from other data sets, for example, with those from the Center for Mental Health Services and the Centers for Disease Control and Prevention. This ability is particularly important because it permits OMH to compare its performance against national averages.

Figure 7.2

The balanced scorecard, or top presentation layer, has a number of components to inform viewers about the performance of the public mental health system. A work in progress, the balanced scorecard is to be implemented in March 2006 and updated at regular intervals using internal and stakeholder input. The initial release of the scorecard will contain a mix of indicators from data sources already available, integrated into the architecture, and relevant to the strategic goals and objectives. When new initiatives and programs are developed as a result of the strategic planning and budgeting process, indicators will be considered by the PIC and constructed to permit assessment of performance against the goals and objectives. The following description of components is provided to enable future users of the balanced scorecard to be familiar with major features of the system before its introduction, and thereby able to more readily view its contents and thoughtfully consider the performance information presented.

Balanced Scorecard features and content

The balanced scorecard will offer reports based on the latest data collected. Its pages have a common look and feel, facilitating a user's ability to analyze the information being reported. It presents the information in a consistent fashion, beginning with statewide data and permitting a review of data regionally. Performance indicators are tied to the Strategic Plan goals and objectives and are being constructed to show performance over time and against targets.

The initial availability of some features, such as the performance targets, may be impacted by the work of the PIC in its advisory function to the Commissioner as well as other factors. Because, as noted earlier, the system design is complex and dynamic, the system will evolve and mature over time, with new performance indicators being added to address new priorities and indicators being dropped as programs and initiatives phase out or change. Overall, the scorecard will contain a small, stable set of indicators to help OMH stay focused on what matters - person-centered recovery goals and promotion of public mental health.

A walk through the first web page you will see

To better understand the content, please see the following example, which uses one management objective being considered by the PIC for inclusion in the first release of the scorecard. As you enter the scorecard, you will see the following details at the top of your page (see Figure 7.3). This screen shot shows the scorecard's essential elements.

Figure 7.3

The scorecard has three tabs, and the content on the red-colored tab titled "Summary View (All Objectives)" is what you will see immediately when you call up the scorecard. Below "Statewide Results" to the left side of Figure 7.3 is an alphabetic listing of management objectives by goal. For the purpose of this illustration and ease of understanding, only one goal and one objective associated with it are being described. In the actual scorecard, however, you will see most if not all goals, with a number of management objectives under each.

Immediately above "Statewide Results," you will see three boxes or drop-down menus with textual descriptions to the left of each. These descriptions are "Geographic Region," "Population Focus," and "Strategic Goal Area." The drop-down menu next to each description permits the user to customize how the data are viewed. The geographic region lets the user choose from among any one of the five regions of the State as well as an overall statewide view of the information. The population focus permits a look at information in relation to children, adults or both. Finally, if a viewer wishes to concentrate on a specific goal area, this can be done by selecting the preferred strategic goal area from its drop-down menu.

As illustrated in Figure 7.3, this page is displaying statewide results for both children and adults and for every goal area. When viewing this page online, then, you should expect to see more goals represented as well as any number of management objectives under each.

Just below the large-type "Statewide Results," you will see a shortened description of the goal statement, "Continuous Quality Improvement," which is highlighted in dark blue ink to bring your immediate attention to the area of focus. Just below this shortened description is the number 4, signifying that this is Goal 4, and below this is a complete statement of the goal as it appears in the Strategic Plan.

The "Management Objective," is a quantifiable, desired outcome associated with a goal and one or more of the objectives under this goal in the Strategic Plan. In this case, you will view one management objective, to "increase the percentage of recipients who rate service quality as good to excellent," and an indication below this that the data show progress in outpatient mental health programs operated by OMH. This particular management objective is taken from the Mental Health Services Survey, which has been designed to better understand how individuals served by the public mental health system evaluate the quality of treatments and supports they receive and their own quality of life. The survey, which was pilot tested in 2004 in peer-run organizations, includes a series of questions in four key areas: access to, appropriateness of, outcomes of, and satisfaction with services.

If you allow your eye to move across the row labeled "Management Objectives," highlighted in light blue ink, you will first see the "Current Value," which is the most recent numerical average of the overall rating by recipients of service quality in the good to excellent range. For this particular indicator, there are no previous values, since the data reflect the only administration to date of this survey instrument. Once the survey is re-administered and new data become available, the current value will be added, the value of 92.7% that appears now will become the "Previous Value," and a change score will be computed and displayed as the "% Change." To the right of the "% Change," you will see a check mark that signifies the progress is in the desired direction by 10 percentage points or more.

For the purposes of demonstration, a "Target Value" appears next. Target values are important because they help OMH to keep the goals and aims in focus, motivate staff to do their best work, make explicit management expectations and serve as a mechanism for judging progress. What this target value indicates is that for this objective, an expectation was put in place to achieve a rating of 90%. Because this rating was exceeded, the last value in the row, "% of Target Achieved" shows attainment of 103%.

A look at content on "Detail View" tab

If you wish to see more detail on this management objective, you would place your cursor over the narrative description of the management objective and click on it. This would automatically take you to the "Detail View (Specific Objective)," which is the content represented by the second tab at the top of the page. If you look at that tab after you click on the narrative description of the management objective, you will see that it is now highlighted in red ink (Figure 7.4), signifying that you have been brought to this new page.

Figure 7.4

At the top of the page you will see the specific management objective stated and below this you will see the goal area and the actual wording of Goal 4. These serve to orient you to the page and provide the necessary information to allow you to look at the detail provided. A couple of differences will be noticeable from the previous page. One is that the "Current Value" of the "Statewide Results" provides the actual number of persons who answered the question (1,915) in relation to the number who responded that the quality of services was good to excellent (1,775). The ratio of 1,775/1,915 appears right under 92.7%. This provides the reader with more information about the size of the sample. Across the row, you can spot the remaining values, which are the same as those displayed on the previous page.

Another important difference between this page and the "Summary View (All Objectives)" page is that you are provided with a breakdown of the responses by region of the State. Under "Regional Results," data for each region are displayed in the same manner as in the "Statewide Results." The check marks indicate substantial change in the expected direction. Each management objective in this section of the scorecard provides this level of detail.

Information to help the reader understand the data

If you look to the top of the page depicted in Figure 7.4, you will observe the last and final tab labeled "About the Balanced Scorecard." When you click on this tab, you will be brought to a page with general information about the scorecard and its purpose (see Figure 7.5). Again, upon being brought to the page, you will notice that the tab is now red in color.

Figure 7.5

While this page contains a brief overview of the balanced scorecard, it is currently being developed further to provide additional text to aid the viewer in finding information, navigating the site easily, understanding the methods used to create the various indicators and their rationale for inclusion, and tips on getting help. It will also provide specific information about the program or initiative to which the measure is tied. Thus, at the time the balanced scorecard initially becomes available, the measure to "increase the percentage of recipients who rate service quality as good to excellent," would include substantive and pertinent information on the Mental Health Services Survey to foster greater understanding of the data being presented.

Because development and refinement of the scorecard presentation and content are continuously in progress, you may encounter new features not described here. In such instances, the scorecard will provide guidance on using these features and assist viewers in readily understanding and considering the information provided.

Chapter 8 highlights the initial introduction of the scorecard tool, content, and a time line for the addition of new information throughout 2006. It also outlines the processes that OMH will use to provide technical assistance in helping users navigate and understand the scorecard, obtain feedback on the utility of this reporting tool, and seek input into improving the scorecard in ways that benefit its intended purposes of fostering recovery and promoting wellness among New Yorkers.

Notes

1. Adapted from Carpinello S, Felton CJ, Pease EA, DeMasi M, & Donahue S. (1998). Designing a system for managing the performance of mental health managed care: An example from New York State's Prepaid Mental Health Plan. Journal of Behavioral Health Services & Research, 25(3), 269 - 278.

Comments or questions about the information on this page can be directed to the Office of Planning.