Outcomes Measurement:

Focus on Psychosocial Issues in Oncology

by Susan Davis-Ali, Ph.D.

Outcomes Measurement: The quantification or measure of clinical and functional outcomes during a specific period. This level is where most of the activity lies and typically involves a classic experimental model of administering pretests and post-tests. Outcomes measurements document treatment efficacy and cost-effectiveness.

Outcomes Monitoring: The serial or concurrent use of outcomes measures during the course of treatment. By periodically checking on a client's status during treatment, clinicians can monitor and modify the rate of progress against some standard of expected results. Interventions may be adjusted or tailored to improve an individual client's outcome. To be useful, providers need this kind of information during treatment.

Outcomes Management: The ultimate use of monitored data in a way that allows individuals and health care systems to learn from experience and make changes in the way services are provided and administered. This level provides all stakeholders with the advantages of continuous quality improvement.

Outcomes measurement, the process of collecting quality-related performance data, is increasingly becoming a staple of oncology programs as they try to demonstrate the value of their services. Pressured by managed care companies to contain costs and their own internal demands for information about the quality and effectiveness of their patient care, hospitals and cancer centers are seeking quality outcomes data to help them make informed decisions about resource allocation, institution-wide policy, and individual clinical decisions. The cost comparisons and socioeconomic evaluations involved in outcomes research give health care providers (as well as payers, pharmaceutical companies, and health care manufacturers) valuable information pertinent to reimbursement, use of medical and nonmedical resources, wages and productivity, and the impact of medical treatment on quality of life.⁶

At its most basic form, outcomes measurement is the process of measuring the results of an action. Outcomes measurement assumes that there is a baseline measure (X) prior to an action (Y), and that the outcome (Z) of the action is measured. The difference in scores between X and Z represents the change usually attributed to the action (Y). For example, one action in a psychosocial intervention might include sessions with an oncology social worker, and the outcome might be the cancer patient's level of depression, anxiety, or social support.

Definitions of outcomes are numerous. Outcomes may be defined as a complex construct composed of several independent dimensions, including mortality, symptomatology, social and occupational functioning, independent living skills, quality of life, use of support services, adverse clinical events or complications, relapse or hospital readmission, and satisfaction with treatment.^7,8

Len Sperry, M.D., Ph.D., of the Medical College of Wisconsin describes three levels of outcomes assessment, including outcomes measurement, outcomes monitoring, and outcomes management.⁹ According to Sperry, outcomes measurement is "the qualification or measure of clinical and functional outcomes during a specific period." Typical outcomes measurement uses a classic experimental model of administering a pretest and post-test. As such, it can document treatment efficacy and cost-effectiveness, but not how interventions could be adjusted or tailored to improve an individual client's outcomes. For that, Sperry believes, the next level is required—outcomes monitoring, the serial or concurrent use of outcomes measures during the course of treatment. The final level, outcomes management, provides all stakeholders with the advantages of continuous quality improvement. Sperry defines outcomes management as the ultimate use of monitored data in a way that allows individuals and health care systems to learn from experience and make changes in the way services are provided and administered.

The most basic element of any outcomes measurement project is data. Oncology program decision makers use data to transform their clinical knowledge from anecdote to science. Data are acquired through research, a process that moves from point A to point B in three phases—planning, implementation, and action.

PLANNING

Point A is the starting point for every outcomes study. Point A clarifies and articulates an oncology program's goals by defining what questions the study will attempt to answer. For example, when performing a psychosocial assessment of cancer patients, the following questions might apply:

• Which patients are experiencing psychological distress?
• Is the intervention effective in reducing a patient's psychological distress?
• What type of intervention is most effective?
• Does the type of cancer affect a patient's psychological adjustment to the illness?

Typically new questions will arise out of information learned in an outcomes study, thus the loop from Point B to Point A. For most researchers this is a never-ending process.

Once the study's initial questions are articulated, program goals must be clarified. The planning phase is the time to create the blueprint for the entire project. Planning a research project involves several steps:

Establish buy-in from the organization. Establishing buy-in at all levels throughout the institution is very important. Good communication must travel from the highest corporate levels to administration to the team of nurses, social workers, or administrative staff who will be overseeing the project. Everyone involved in the study should receive periodic feedback about how the study is progressing and any preliminary findings. Workshops, newsletters, and even information posted on the hospital bulletin board facilitate this kind of communication.

Determine which variables to measure. For example, the most common psychosocial variables might include:

• adjustment to illness
• work-related stress
• marriage and family disruption
• depression
• anxiety and phobias
• child and adolescent problems
• financial distress
• body image changes
• disability
• mourning and bereavement
• sexuality and fertility
• existential and spiritual concerns.¹⁰

Quality of life and psychological distress may also be included in this list.

Decide who will be assessed. Among the choices of whom to assess are patients, family members, providers, and administrators. Assess patients first; their perspective is most salient to their immediate care. If resources allow, include spouses and other family members in an assessment of their own distress and quality of life.

Determine frequency of assessment. How often will assessments be administered? Will patients complete pre- and post-tests only, or will they be retested on a regular basis? When will the baseline measure be established—at diagnosis, at the first treatment session, or at another time? There are no hard and fast rules about when to measure subjects, but oncology measurement planners should be consistent and follow an established protocol. Also, be sure to document any variations in the protocol.

Select assessment instruments. Will you use a standardized instrument, or will you customize your own? Standardized instruments have the psychometric characteristics (i.e., reliability and validity) necessary to ensure good, quality data. In addition, standardized instruments allow a hospital or cancer center to establish databases that can be shared with other institutions that use similar standardized measures.

Gotay and Stern identified those standardized instruments that have been used most frequently to measure psychological functioning in cancer patients.¹¹ These include seven scales that assess depression, anxiety, psychological symptoms, mood, and general psychosocial adjustment to illness:

• Center for Epidemiological Studies-Depression (CES-D)
• State-Trait Anxiety Inventory (STAI)
• Symptom Checklist-90-Revised (SCL-90-R)
• Brief Symptom InventoryÒ (BSI)
• Beck Depression Inventory (BDI)
• Profile of Mood States (POMS)
• Psychosocial Adjustment to Illness Scale—Self-Report (PAIS-SR)

Customized instruments can also be reliable and valid, but only with strong instrument design, item writing, and test development, which are not easy endeavors. Unless hospitals have access to someone highly trained in these areas, they are better off using a well-established instrument that ensures decisions based on a sound, reliable, valid measure.

Make initial decisions about recruitment of patients. Will all cancer patients at the center be invited to participate in the outcomes project? Will recruitment be limited to patients with a particular diagnosis? Will patients be asked directly by a health care provider to participate? Will they be sent a letter? Will fliers be posted asking for volunteers?

My recommendation is to design a protocol that requires all patients be assessed, especially if the assessment is being used as a screening tool to triage patients into two different psychosocial treatment categories, e.g., those who will receive intervention and those who will not.

From a clinical standpoint, assessing all patients rather than relying just on patient volunteers or clinician referrals is preferable for three reasons. First, distressed patients may be experiencing a variety of symptoms (e.g., fatigue, irritability, loneliness, anger) that make them less likely to volunteer for the very protocol that could benefit them. Second, distressed patients who do not exhibit overt symptoms of distress are often overlooked for mental health referral by medical providers. Finally, some patients who exhibit observable psychosocial symptoms are falsely labeled as clinically distressed by medical personnel, when in fact they are not.

By assessing only a segment of the patient population, clinicians are most assuredly missing a large number of patients who could greatly benefit from mental health treatment. With the ease, expense, and validity of many self-administered testing instruments, there is no reason to exclude any patient from screening for psychosocial distress. The relatively small cost and effort can have a potentially enormous impact on outcomes and quality of life.

Decide who is in charge of the daily operations of the project.

Be sure that this person is given the time he or she needs to manage the project. Hospital staff and administrators must appreciate both the importance and the demands of orchestrating these projects. Establish one person (an administrative assistant, an intern, a graduate student) to be responsible for the administration and collection of all the data. Ideally this person will champion the program and will work hard to keep it going.

Determine who will be informed of the results at the conclusion of the outcomes project, and provide ongoing, real-time feedback to clinicians as the project is in process. The final report generated at the end of the outcomes measurement project will undoubtedly look at patients at the aggregate (total group) level. While this type of information is generally used by managed care companies and hospital administrators, it is not the type of information most helpful to the health care provider who is treating individual patients. Real-time feedback means that clinicians receive immediate results about the assessment so that the information can be used to make clinical decisions and tailor a patient's treatment. To provide real-time data to clinicians, a system for scoring the instruments must be made part of the outcomes measurement program. Assessment instruments that are filed without being scored until the entire implementation phase is completed provide no value to the patient and provider.

Keep the assessments short and simple. Be sensitive to the health of the participants and do not create unnecessary "respondent burden."¹² Gather only the information needed from the participants and use medical records or family members to gather demographic information, such as insurance status or previous medical history.

Communicate. Network with colleagues to find out what kinds of outcomes measures other institutions are using. Search the Internet, attend conferences, and read journals to learn the latest trends in outcomes measurement. Time, money, and frustration can be saved by emulating the leaders in your field. Form outcomes research groups and share ideas, expenses, and data. Outcomes measurement, after all, is all about collaboration.

The planning phase is the foundation of the entire outcomes measurement study. The amount of time spent in the planning stage will vary with each project. Some planning phases may last as long as two years, others only months. However, it is important not to rush the process. In their eagerness to begin a research study, many individuals rush through the planning stage, only to find that at the end of the project that there is a particular piece of information that was not collected.

IMPLEMENTATION

During the implementation phase, data collection commences. The process of data collection involves distributing forms (e.g., assessment instruments), collecting completed forms, scoring forms, providing real-time feedback to clinicians, and setting up the database. This process remains the same whether one is conducting a research study, providing clinical care, or both.

Implementation is the phase with the most potential for sloppiness and error. Enthusiasm can wane. Even under the most ideal circumstances, a 10 percent attrition rate per year in an outcomes measurement project can be expected.¹³ If one does not actively work to keep subjects, clinicians, and project administrators involved in the project, it will fizzle out. Even the most enthusiastic intentions cannot compete with a normal attrition rate. Plan ahead for declining compliance and interest in the project. A plan must be developed to account for patients who are unable to complete all parts of the outcomes assessment due to morbidity, mortality, relocation, and other issues.

Every attempt should be made at a patient's discharge to gather as much information possible in the event the patient does not complete the post-test. Some attrition will be unavoidable, but researchers will need to capture as much information possible about all patients to ensure that the outcomes measures do not reflect only those patients who are able to complete the post-test. If information about patients who fail to complete the entire project is not captured, outcomes measures may show that patients were getting better, when in fact only the healthy patients were left to complete the final assessment.

ACTION

The action phase begins with data analysis, which must be performed correctly if administrators are to make decisions based on study findings. Consult a statistical expert if you have not had formal training in statistics or are not confident in your abilities. A local university or community college can help in locating a statistician. Misinterpreting data findings can result in improper analysis and erroneous conclusions.

Data that are analyzed correctly can help justify a variety of actions to help an institution perform more efficiently, such as purchasing additional or reallocating resources, instituting large-scale policy and procedure changes, and educating staff about the strengths and weaknesses of the oncology program. Outcomes data will also facilitate information sharing for benchmarking studies with other institutions.

An outcomes measurement system is an ongoing process of asking questions, finding answers, taking action, and asking more questions. As changes are implemented, other questions are likely to arise, which may result in an entirely new research process. When this cycle becomes a routine part of a hospital's operations, the outcomes measurement project is on the right track.

An optional step in the standard research model is a small-scale pilot study. Somewhat like a dress rehearsal for the larger proposed study, pilot studies are used to fine-tune and test any logistical issues that may need clarification. They are relatively inexpensive and quick. Results are a good way to show the value of the study and demonstrate effectiveness on a small scale in order to get funding for the larger project.

CASE HISTORY

The following hypothetical case history illustrates how one psychological screening instrument, the Brief Symptom InventoryÔ (BSIâ , see box for explanation), can be used for outcomes measurement.

General Cancer Center is a mid-sized community program with thirty-four dedicated cancer unit beds and 900 new analytic cancer patients per year. Like all cancer centers today, General must decrease costs while maintaining quality of care if it is to remain competitive. Administration is aware that distressed patients have higher health care costs than patients without psychological distress¹⁴ and that screening for patients in distress may allow them to cut costs and make appropriate referrals to social services, which may improve the quality of life of their patients.

Staff decide to use a standard instrument and select the BSI, a paper-and-pencil, self-administered test that takes five to seven minutes to complete. They write a protocol that states all new patients will be given a BSI at their initial visit to establish a baseline score for outcomes measurement and to screen for psychological distress. Distressed patients will be referred to the social work department, which will see each patient within a week of referral. All patients will be given a second BSI twelve weeks later.

Sarah is a 40-year-old woman who was recently diagnosed with breast cancer at General Cancer Center. She is given the BSI during one of her initial visits with her oncologist, and her scores are reported on the pretest file report.

Sarah's GSI score is 64, and she is flagged for distress. The oncologist discusses the findings with Sarah and refers her to the social work department.

Before meeting with Sarah, the social worker reviews her BSI profile. The social worker examines the scores on the nine subscales to identify which areas appear to be most problematic. Her profile report reveals that two subscales are elevated above the cut-off score of T=63: depression (DEP; T=72) and Hostility (HOS; T=71). Her scores on the Somatization (SOM), Obsessive-Compulsive (O-C), Interpersonal Sensitivity (I-S), Anxiety (ANX), Phobic Anxiety (PHOB), Paranoid Ideation (PAR), and Psychotism (PSY) are all below a T score of 63. From the outset of their initial visit, Sarah and her social worker work on strategies for reducing her depression and hostility.

Sarah is referred to intensive counseling for six to eight sessions and begins implementing some relaxation exercises. Twelve weeks after her initial assessment, Sarah is given the BSI test again. Her post-test scores are then compared to her pretest scores, and significant changes are documented. The scores show a decrease in Sarah's Global Severity Index score, a slight decrease in her Depression scale score, and a significant decrease in her Hostility scale score. Previously nonelevated scores remain below a T score of 63.

Sarah's scores are combined into a database at General, which includes other information such as a patient's age, diagnosis, rehospitalization, and insurance. Aggregate (i.e., group level) analysis is then conducted to compare the GSI scores of distressed patients and nondistressed patients. Nondistressed patients at General have a mean pre-test GSI score of T=58, compared to the mean GSI score for distressed patients of T=70. Twelve weeks later, the mean post-test GSI score for distressed patients drops to T=61 and the mean score for nondistressed patients remains fairly constant at T=59.

The staff at General quantify their results and provide real-time feedback to clinicians who are able to use the information to begin helping Sarah immediately. By implementing the BSI at General, clinicians are able to provide early intervention to other distressed patients. The aggregate data are presented to the managed care company to show the benefits of implementing a screening and outcomes program at the center. General shares its data with other cancer centers using the BSI test to see how its patients compare with cancer patients across the country. It is a win-win situation for the patient, the providers, the hospital, and the managed care company.

Ideally, each newly diagnosed cancer patient would be seen by a social worker or a psychologist to discuss issues related to psychosocial adjustment to illness. With the time constraints and increased patient loads placed on most health care providers today, such intervention is not routinely possible. Nevertheless, the need to "red flag" patients who are experiencing distress is paramount. The BSI test is an example of one assessment tool that allows clinicians to screen for distress and gather outcomes data at the same time.

Hospitals and cancer centers should begin with the largest high-quality outcomes assessment program that they can afford, implement, and manage. Begin by dividing an ambitious project into manageable parts. A small amount of good data is better than massive quantities of bad data. If you are not comfortable implementing a large-scale outcomes program at the outset, begin with a small pilot project of twenty-five to thirty patients. Or, if budget constraints prevent a program from automating and fully integrating all its data into one database, staff can at least begin to gather quantifiable information about patients and document interventions and outcomes. Standardized assessments will help them compare their results with those from other cancer centers.

C. Everett Koop once stated, "Anecdotes do not make good scientific material."¹⁵ Using outcomes measurement will document an oncology program's cost-effectiveness and its impact on patients' psychological distress, adjustment to illness, quality of life, and overall satisfaction. These data will help oncology programs survive and thrive in the 21^st century.

REFERENCES