Quality in Women's Health:
July 1998 - Number Eight
By Julianna S. Gonen, PhD
Managed health care has introduced a degree of accountability into the delivery of care that was lacking in the past. While there has been considerable scrutiny of the quality of care delivered in managed care plans, in part due to concerns that the incentives of managed care might compromise quality, there was little to no such evaluation of the quality of care delivered in the fee-for-service system. In this final issue of Insights, we address how quality in women's health is being defined and measured as performance measurement and accreditation programs continue to expand and receive greater emphasis. Throughout this series, several essential areas of women's health have been identified, and we now seek to determine the extent to which quality is being measured in each. These areas include comprehensive primary care, reproductive health care (including the prevention of unintended pregnancies and sexually transmitted diseases), mental health care, care for chronic conditions, and the unique health needs of low-income women insured through Medicaid. Following a discussion of general quality measurement issues, this paper reviews current mechanisms of measuring quality in managed care, with a particular focus on the extent to which they assess how well plans are meeting women's health care needs. Initiatives under development for measuring womenıs health quality within managed care are also discussed, followed by suggestions for areas on which to focus such efforts.
Defining and Assessing Quality
The Institute of Medicine defines quality as "the extent to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge."1 But while there is virtually universal agreement that quality is a desired end, there are many competing notions about how to measure quality and little systematic data on how good the quality of care is in the United States.
The Eye of the Beholder? A critical question in the debate over defining quality concerns whose perspective is reflected in any definition or assessment tool. Should quality be defined by "experts" using relatively objective, scientific criteria, or by consumers, the end users of health care services? The argument in favor of expert opinion holds that consumers are not equipped to assess the clinical quality of services because they lack the requisite knowledge and training. Proponents of the consumer perspective argue that patient satisfaction is as important as technical quality measured by a third party, if not more so. Some consumer advocates even argue that lay people can indeed distinguish technical quality and that their assessments should be considered along with those of experts. Current quality assessment initiatives are increasingly moving to integrate the two approaches -- each of which provides different and equally valuable perspectives -- by adding consumer satisfaction surveys to objective measurement sets and increasing the sophistication and standardization of these survey tools.
In assessing health plan quality several aspects of care may be evaluated: the structure of the delivery system, the processes by which care is delivered, and the outcomes of that care (which include clinical and functional status as well as patient satisfaction with the experience of care).2 Examples of structural features that may have a bearing on the quality of care include the percentage of network physicians who are board certified and the ratio of providers to the enrolled population. Delivery system structure is generally evaluated through an accreditation process, such as the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) program for hospitals and the National Committee for Quality Assurance (NCQA) program for managed care organizations.
Processes of care are assessed through performance measures. These measures typically report the proportion of the relevant population that received a particular service or treatment, such as cancer and cholesterol level screenings or immunizations. These measures are tabulated using administrative, laboratory and pharmacy data, and/or medical record abstraction. Administrative data, generally taken from claim and encounter forms and pharmacy records, have the advantage of being fairly uniform (although coding practices do vary) and relatively straightforward to gather and tabulate since they are automated; the disadvantages are that they only capture a portion of what occurs in health care delivery and miss services that are not coded and/or reimbursed. Abstracting medical records provides extensive detail on processes of care and clinical status but requires a separate and quite labor-intensive collection process. There is also considerable variation among providers in recording patient information in charts.3
Outcomes measurement presents one of the most significant challenges in health care today, not only for managed care plans but for all components of the health care system, from individual providers to hospitals to integrated delivery systems. Outcomes of interest might focus on clinical status (elimination or amelioration of a disease or illness) or might focus on functional status or patient satisfaction. A consistent criticism of current quality assessment programs has been that they focus predominantly upon technical processes of care rather than the interpersonal aspects of care processes or the outcomes of care. This is in part a reflection of the state of the science of outcomes research, the goals of which are to help understand the relative contributions of various factors (genetics, lifestyle, access to care, treatment decisions, professional practices, etc.) to improved health, and to provide evidence on which to base clinical practice.4 But it is easier to measure things that are easy to define, identify and quantify; hence the emphasis on receipt of services in current measurement sets. Defining and measuring health outcomes is considerably more complicated, which is in large part why such assessments, with the exception of patient satisfaction surveys, are not yet widely used. Survey data are useful in assessing patients' level of satisfaction with the experience of care but also require a separate collection activity. Presently there is wide variation among survey instruments in use, which compromises the ability to make comparisons among plans. In addition, many factors outside of the control of a health plan will affect health outcomes. There is a need for methods that account for the proportion of variation in outcomes that can be attributed to the health plan and that adjust for severity and other factors that may affect outcomes among enrollees. Without such adjustments, variations in outcomes may reflect differences in enrolled populations rather than differences in the quality of care delivered.
Uses of Quality Information. Why should quality be measured at all? Who uses this information? The potential end users of quality ratings of health plans include purchasers of health care (both private and public), consumers (current and potential health plan enrollees), and health plans themselves. Purchasers may use accreditation status and performance measures to determine the plans with which they wish to contract. Potential enrollees may use consumer-oriented "report cards" on health plans to select the one to join. There is in fact evidence that creating consumer reports may also have a secondary or "by-product" effect of leading to positive changes on the part of the clinicians and health care delivery organizations being evaluated.5 Plans themselves also use quality indicators to identify areas for internal quality improvement efforts. Having standards for the quality of care that health plans deliver and holding them accountable to those standards is meant to improve the value of services being offered to purchasers and consumers.
But the extent to which this is actually happening is not fully known. Many observers assert that it is still price that drives purchaser behavior, with quality concerns coming in second at best. Consumer-friendly quality information has been slow to proliferate, making it difficult for potential enrollees to find objective criteria to aid their enrollment decisions. As women are the primary users of health care services and also the main decision makers within their families regarding which plans to join and when to seek services, the availability of user-friendly quality information is especially critical for women.
Challenges in Quality Measurement. Significant methodological issues present challenges to health plan quality measurement. With the health care system in a state of rapid evolution, it becomes difficult to determine which piece of a large delivery system to evaluate. Health plans now take the form of rather complex delivery systems or are in fact parts of larger such systems. While programs like the accreditation standards and HEDIS measures of NCQA assess services at the health plan level, there is increasing attention to the need to perform similar assessments at the individual provider level, and also at other discrete care sites such as clinics and medical group offices.
One of the barriers to comprehensive quality measurement in health plans is the cost of obtaining the necessary data and analyzing it to produce meaningful measures. Managed care organizations (MCOs) often claim that while they support accountability, the ever-increasing demands for data and measurements are proving prohibitively expensive. This is particularly the case for more loosely organized health plans, which now constitute the majority of the market. When a plan contracts with multiple physicians, hospitals, and other delivery sites, it becomes difficult to collect full and accurate data on all patient encounters from all of these sites. And providers themselves, who often contract with multiple plans, are also straining under the burdens of having to report more and more data to each health plan in which they participate.
Programs for evaluating quality are addressing these issues to some extent. A new program of the American Medical Association entitled the American Medical Accreditation Program (AMAP) aims to alleviate some of the burdens on individual physicians by providing credentialling services that would eliminate the need for multiple reviews by each of the many health plans with which the physician contracts. And in May 1998 the Performance Measurement Coordinating Council was formed by the AMAP, NCQA, and the JCAHO, the main credentialling body for hospitals. The primary aim of this new council is to establish standards in the field of performance measurement so that providers and plans will not have to re-measure data for multiple outside agencies. While the effort will address some of the issues of the high costs associated with meeting several different sets of performance standards, it will also attempt to address the issue of the level of analysis in quality measurement, as it aims to develop consistent measures that work across sector boundaries, whether physicians or hospitals or health plans. An additional goal of the council will be to develop risk adjustment methodologies. As described by JCAHO president Dennis O'Leary, MD, "as it gets harder and harder to separate the hospital from the health plan from the doctor, consistent means are needed to measure their output."6
Other methodological challenges involve determining how in fact to define and capture the desired information. While it may be relatively straightforward to measure how many members of an enrolled population received a service with a recognizable CPT code during a defined time interval, other less precise but equally important aspects of care do not lend themselves as easily to quantification and measurement. For example, there is at the moment considerable emphasis on physician counseling for everything from preventing sexually transmitted diseases (STDs) to whether to initiate hormone replacement therapy. But there is as yet no direct way to capture whether counseling occurred during a physician encounter, and even less ability to determine the quality or effectiveness of that counseling.
Current Quality Assessment Mechanisms
With the growing emphasis on accountability in health care delivery and the need to measure the quality of care being delivered in managed care, quality assessment initiatives have proliferated in recent years. These efforts range from internal MCO quality improvement programs to external private-sector and government-sponsored evaluation programs. Until now, however, there has been no standard definition of quality in women's health care, and the quality measures that do exist have focused on women-specific services that are easy to measure with administrative data (for example, receipt of pap tests, mammography screening, and early prenatal care). There has been little effort to define quality in women's overall health care or to identify the structure, process, and outcomes dimensions of quality that are of particular relevance to womenıs health. There are, however, signs that this is changing, as health care systems come to terms with the prevalence of chronic disease (which particularly affects women), as there is increasing recognition of the patient's central role in decision making and the resulting effect on health outcomes, and as there is growing realization of the limited generalizability from randomized clinical trials which do not account for the organization, delivery and financing of treatment. As stated in a recent report, "as we enroll more women in clinical trials and pay more attention to their special health needs, it will become increasingly important to understand when differences in health outcomes between men and women are gender-based and when they are service-based."4
What follows is a review of the major current quality assessment initiatives, with a specific focus on their attention, or lack of attention, to womenıs health quality.
Private Sector Initiatives. The National Committee for Quality Assurance (NCQA) is a private, nonprofit organization based in Washington, DC, that accredits managed care plans and also administers the most widely used set of quality measures, the Health Plan Employer Data and Information Set (HEDIS). Established in 1979 by the managed care industry, NCQA became autonomous in 1990 in order to attain greater independence from the industry it was evaluating. NCQA views its two program areas -- accreditation and performance measurement -- as complementary strategies for evaluating health plans, as the former assesses the strength of the core systems upon which quality health services depend, while the latter (HEDIS) measures the actual care delivered. Although the two programs at present are separate and distinct, beginning in 1999 performance measurement will be integrated into the accreditation process.
HEDIS 3.0, the current iteration of the measurement set, is divided into several sections, or domains. Within each domain are the current reporting set measures, as well as several "testing set" measures whose implementation will depend upon their feasibility being favorably evaluated. The Effectiveness of Care domain contains measures such as eye exams for people with diabetes and childhood and adolescent immunization status. Measures pertaining specifically to women's health7 in this domain include cervical cancer screening, breast cancer screening, initiation of prenatal care in the first trimester, number of low birth-weight babies, and check-ups after delivery. Women's health measures in the testing set for this domain are for follow-up after abnormal pap tests and mammograms, and the rate of screening for chlamydia.
The only women's health-specific measure in the Access/ Availability of Care domain is of the number of low birth-weight deliveries at facilities for high-risk deliveries and neonates. The Satisfaction with the Experience of Care domain has in the testing set a measure of satisfaction with breast cancer treatment. In the Use of Services domain, the reporting set contains measures of cesarean delivery and vaginal birth after cesarean (VBAC) rates, discharge and average length of stay for females in maternity care, births and average length of stay for newborns, and the frequency of ongoing prenatal care. The domain entitled Informed Health Care Choices contains in the testing set a measure of counseling women about hormone replacement therapy. (See the discussion below of NCQA's Women's Health Measurement Advisory Panel modifications to this measure.) And finally, in HEDIS's Health Plan Descriptive Information domain there are indicators for weeks of pregnancy at time of enrollment and availability of family planning services.
NCQA is now gathering HEDIS data from health plans and compiling it into a national database labeled Quality Compass. In 1997 this database contained data on 329 managed care plans, including standardized consumer satisfaction results, providing an emerging picture of the quality of care being delivered in MCOs.8 For example, these data revealed a mean mammography rate of 70.36, with the lowest reported rate being 27.70 and the highest 89.00. For pap tests, the mean was similar -- 70.44 -- and reported rates ranged from 24.20 to 100.9 The rates of women receiving prenatal care in the first trimester were higher, with a mean of 84.48 and a range from 40.50 to 99.80.
This year (1998), the federal Health Care Financing Administration (HCFA), which runs the Medicare program, is requiring all Medicare HMOs to submit audited HEDIS data to NCQA by July 1. The auditing requirement is a change from previous years, and HCFA plans to release this yearıs data to the public.
In 1997, NCQA convened the Women's Health Measurement Advisory Panel (Womenıs Health MAP) to recommend additional measures for development. The Women's Health MAP is developing a measure of counseling patients on options for management of menopausal hormonal changes and measures related to the prevention of unintended pregnancy. Other activities of the MAP include prioritizing women's health conditions for future measurement development, interacting with other NCQA MAPs to ensure that women's health concerns are addressed, and exploring gender differences in patient satisfaction survey items.
Public Sector Efforts. The federal Agency for Health Care Policy and Research (AHCPR) is also conducting several projects in the area of quality assessment, from both the "expert" and the consumer perspectives. On the clinical side, projects include the Computerized Needs-Oriented Quality Measurement Evaluation System (conquest), a computer tool to help organize and evaluate performance measures. AHCPR has also awarded cooperative agreements, labeled q-span, for the development and testing of new measures. The agency's main initiative on the consumer side is the Consumer Assessment of Health Plans (CAHPS), a 5-year project to develop surveys for the assessment of consumer perceptions of health plan quality. The CAHPS survey goes beyond traditional consumer satisfaction surveys that report on overall satisfaction and measures consumer experience with specific aspects of their own health plans. It is also designed to assess experience across different populations and care delivery systems. A joint team from the CAHPS project and NCQA is developing a new tool, merging the NCQA member satisfaction survey and the CAHPS instrument, for inclusion in HEDIS 1999.10
The Health Care Financing Administration also administers the federal portion of the Medicaid program, and in 1993 the agency implemented the Quality Assessment and Reform Initiative (QARI), combining the managed care plan accreditation standards of NCQA with federal quality requirements to begin to assess the quality of care being delivered in Medicaid managed care plans. Through QARI, HCFA directed states to monitor Medicaid MCOs' internal quality assurance programs and to perform external quality reviews or focused studies. A 3-year demonstration project of the QARI system in three states from 1993 to 1995 found that larger plans were able to meet QARI requirements in the process of preparing for NCQA reviews, while smaller plans had difficulty meeting the QARI quality assurance program standards. The statewide data generated through QARI contributed more to a broader, external discussion of quality of care for Medicaid beneficiaries than to internal, plan-specific quality improvement efforts.11
New Initiatives. One criticism of NCQA's HEDIS has been its focus on processes of care rather than on outcomes of care. Partly in response to this, a new coalition was formed to develop outcomes-based measures of health plan quality. The Foundation for Accountability (FAcct) is an alliance of corporate and government purchasers, consumer organizations, providers, researchers, delivery systems and managers. To develop measures, FAcct "conducts focus groups and other research to understand the aspects of quality that are important to consumers. FAcct combines these patient expectations with the best available clinical knowledge and scientific research to create measures that hold the health system accountable to high-quality care."12 The seven clinical areas in which measures are being developed and tested include adult asthma, breast cancer, diabetes, major depressive disorder, health status, health risks, and consumer satisfaction.
The FAcct breast cancer set contains several components. Those falling under "Steps to Good Care" include proportion of women age 52-69 having a mammogram in a two-year period, proportion whose cancer was detected at Stage 0 or I, proportion of Stage I and II patients receiving adequate information about radiation treatment and undergoing breast conserving surgery, and proportion of those undergoing such surgery who subsequently receive radiation treatment. The set also contains a satisfaction level score, an assessment of patients' quality of life after living with breast cancer, and the probability of disease-free survival for a group of patients.13
While independent organizations such as NCQA and FAcct have taken the lead in accrediting managed care plans and measuring the quality of care delivered, other initiatives have emerged from entities with whom health plans contract -- purchasers and labor organizations. The Pacific Business Group on Health (PBGH) developed the California Cooperative Healthcare Reporting Initiative several years ago to gather performance indicators on health plans operating in California in order to aid their contracting decisions. And the American Federation of Teachers (AFT) developed its own health plan standards, based on NCQAıs accreditation standards, and is in the process of supplementing them with standards specifically designed to assess women's health quality.
The Rand Corporation, with funding from HCFA, is also developing quality assessment systems for pediatric and adult women's health for use in managed care plans. The projectıs expert panels came up with 340 indicators across 20 clinical areas in adult womenıs health (age 17-50), applicable to either commercial or Medicaid health plan enrollees. The clinical areas, selected by reviewing the 1991 National Ambulatory Medical Care Survey as well as topics covered by guidelines from AHCPR, include alcohol dependence, breast mass, depression, family planning, and upper respiratory tract infections, among others.14 This initiative is distinguished from others in terms of both the number of indicators included and its mechanisms such as aggregate scoring, which allows incorporation of indicators across clinical topics that would otherwise apply to too small a proportion of enrollees to be measured. The Rand system uses both medical records and administrative data but relies heavily on the former.15
Women's Health Needs and Quality Measurement
Throughout the Jacobs Institute's project Defining Issues and Monitoring Trends, numerous important areas of women's health have been identified and explored to determine how managed care is affecting women's receipt of needed services. These priority areas can yield goals for continuing efforts to develop meaningful measures of women's health care quality. Although identifying and describing discrete measures is beyond the scope of this paper, several critical areas upon which to focus future efforts in measuring women's health quality are outlined below. Many measures could potentially be derived, taking all of the current forms -- structure, process and outcomes. This is not intended to be a comprehensive listing of all areas in which women's health quality can and should be measured, but rather is based on the areas chosen for exploration in this project.
In the area of primary care, women have been shown to receive the most preventive services when they have access to both generalist and ob-gyn physicians. Thus the availability of a range of primary care providers, or even of interdisciplinary primary care teams, and also the extent to which MCOs are defining gender-specific primary care competency and requiring that their primary care providers have training in women's health, are key determinants of quality for women's health.
Receipt of preventive services is perhaps the most easily identifiable way of measuring primary care, and existing measurement sets do evaluate MCOs using process measures of the proportion of women in relevant age groupings who receive services such as pap tests and mammograms. But a discrete set of clinical preventive services does not constitute the full range of comprehensive women's primary care; measures should be developed for the comprehensiveness of preventive services (appropriate to the womanıs age and risk status) and for other process features, including the coordination of general and reproductive components of care and the amount and quality of physician-patient communication.16 As low-income women have been found to receive fewer preventive services than middle- and upper-income women, Medicaid MCOs should be giving especially high priority to increasing the proportion of women receiving basic preventive care. Because of the historical fragmentation of women's health care services and the variations in delivery of women's primary care, it is also essential to measure outcomes of well-woman and preventive care. Examples of such outcomes might include successfully protecting women from unintended pregnancy or enhancing their ability to engage in self-care. In addition, measures of outcomes from the perspective of the patient, such as satisfaction, functional status, and quality-of-life measures, should also take gender differences into account. Standard enrollee satisfaction questionnaires may not include measures that are particularly important to women, such as the provider's attention to the psychosocial components of care or the provider not talking down to the patient.16
Quality of care in the area of women's reproductive health needs requires incorporating prevention of unintended pregnancy and STDs into MCOs' overall prevention paradigm. The existence of contracts between MCOs and traditional community-based family planning providers would provide an indication of the plans' commitment to allowing their enrollees full access to family planning services in the settings most comfortable to them. The use of adolescent health specialists to meet the unique health care needs of adolescent enrollees would enhance quality of care for young women.
Counseling about health risks is also an important component of both primary and reproductive health care but is less readily measurable. It includes counseling regarding risky sexual practices and the prevention of unintended pregnancy and STDs. The inclusion of comprehensive sexual history taking and screening and treatment for STDs must be standardized among MCOs' primary care providers. STD screening should be included among the preventive services that are now commonly measured as indicators of health plans' processes of care, and the prevalence of STDs within a health plan population should also be assessed as are other important health conditions.
There is also a need for better integration of mental health competency in primary care; this is of particular importance to women, who suffer disproportionately from depression. Women's mental health needs must be given consideration equal to that given to their physical health needs. An area of particular concern in mental health treatment is the role of "carve-out" programs that separate mental health treatment from the delivery of other care. For those MCOs using carve-outs for mental health, it is important to assess how well this treatment is monitored by and integrated with women's general primary care. Mental health competency must be included within primary care training. Quality may be further assessed by the availability of both psychotherapy and pharmacotherapy in treating mental illnesses such as depression, since a combination of the two is often found most effective. Screening for depression is a critical service that must be conducted more frequently, particularly given the high prevalence of depression among women.
Traditional community clinics and providers have long played critical roles in providing care to the underserved, and there is concern that their viability may be threatened as Medicaid programs roll their beneficiaries into managed care plans. Those plans that elect to serve Medicaid enrollees could be evaluated on their incorporation of traditional community providers into their networks and on the availability of enabling services such as transportation and language interpretation. The traditional provider issue presents a prime example of the tension that sometimes emerges between objective quality criteria and more subjective patient-based assessments. Traditional providers frequently lack the experience and credentials that would earn them high marks on objective quality measures, but their familiarity with the health and other problems facing their communities, in addition to patients' greater comfort level with them, would often render such providers preferable from the consumers' perspective.
The integration of specialty with primary care is critical for women with chronic conditions; when necessary and appropriate, women with chronic conditions should be able to have a specialist physician serve as a primary care provider. At the same time, availability of non-physician providers for regular routine care also serves to enhance quality of care for this population. Coordination across multiple care sites is especially important for those with chronic illness, and MCOs possess the potential to implement improvements in this area. Strong linkages to other social services needed by those with chronic conditions, as well as patient education around self-care, constitute additional structural features upon which plans could be evaluated.
Quality will also improve as plans take into account any known gender differences in the effectiveness of certain medications when making drug formulary decisions, so that they do not include only pharmaceutical agents known to be effective in treating men. Similarly, gender analysis should be incorporated when developing clinical practice guidelines, so that rather than calling for unisex treatment protocols, they acknowledge and adjust for differences in how conditions manifest differently in women and men and thus require different courses of treatment.
Remaining Issues and Questions
The debate over how to define and measure quality health care will not be resolved imminently, but it is critical for women's health needs to become part of this dialogue as systems are evolving. From the foregoing review of the current state of quality assessment initiatives, both generally and specifically in women's health, several issues emerge for further consideration:
1. Lohr K (ed.) Medicare: A Strategy for Quality Assurance, Vol. 1. Washington, DC: National Academy Press, 1990.
2. Donabedian A. Explorations in Quality Assessment and Monitoring, Volume 1: The Definition of Quality and Approaches to Its Assessment. Ann Arbor: Health Administration Press, 1980.
3. McGlynn EA. Quality assessment of reproductive health services. West J Med 1995;163[suppl]:19-27.
4. Toward a Women's Health Outcomes Research Agenda. Washington, DC: Society for the Advancement of Womenıs Health Research, October 1997.
5. Longo DR et al. Consumer reports in health care - do they make a difference in patient care? JAMA 278:19;1997.
6. Moore DJ. Standardizing standards - accrediting agencies form council to end confusion. Modern Healthcare (May 25, 1998).
7. For purposes of this discussion, measures termed "women's health measures" are those which exclusively or largely apply to women, and not to measures such as continuation of depression treatment (testing set) which would disproportionately pertain to women but applies to all enrollees.
8. Presentation by Margaret O'Kane, NCQA President, Alliance for Health Reform briefing, "Managed Care Plans: What Impact on Quality?," February 13, 1998.
9. The mammography numbers refer to the percentage of female plan enrollees aged 52-69 who had received mammography screening during the previous 2 years. The cervical cancer screening measure counts the number of enrolled women aged 21 to 64 who had received one or more Pap tests during the past 3 years.
10. See http://www.AHCPR.gov for additional information on the CAHPS project.
11. Felt-Lisk S and St. Peter R. The Quality Assurance Reform Initiative (QARI) Demonstration for Medicaid Managed Care Final Evaluation Report. Washington, DC: Mathematica Policy Research, 1996.
14. The full list of clinical topics is acne, alcohol dependence, allergic rhinitis, asthma, breast mass, cesarean delivery, cigarette use counseling, depression, diabetes mellitus, family planning and contraception, headache, hypertension, acute low back pain, medication allergies, prenatal care, preterm labor, preventive care, upper respiratory tract infections, urinary tract infections, vaginitis, and sexually transmitted diseases.
15. Schuster MA et al. Development of a quality of care measurement system for children and adolescents. Arch Pediatr Adolesc Med 151:1997.
16. Weisman C. Women's Health Care - Activist Traditions and Institutional Change. Baltimore: Johns Hopkins University Press, 1998.