Dissertation Defense Max Masnick on November 3, 2015

Thank you for coming today.

My dissertation title is a mouth-full, so I’ll go through it piece-by-piece to explain the background of my research, and then I’ll talk about my specific aims, results, and conclusions.

I’ll start with “hospital quality measures”.

Quite simply, hospital quality measures are a way to quantify hospital quality.

This is kind of abstract, so I’ll go through some examples of the different kinds of quality measures to make it more concrete:

• Outcome
  • Outcome measures relate directly to quality (patient survival and quality of life)
  • Examples include 30 day readmission, infection, and death rates.
  • These are often risk-adjusted to account for patient factors affecting outcomes outside the hospital’s control
• Process
  • These relate to following standards of care
  • For example, the percentage of heart attack patients given aspirin at discharge is a process measure.
  • Process measures are further removed from quality (patient survival and quality of life), but they are related because theoretically following evidence-based standards of care should by definition improve patient outcomes.
• Structural
  • Characteristics of hospitals
  • Examples: use of electronic medical records or the percentage of board-certified physicians.
  • Good performance on structural measures means hospitals have the capacity to have good process and outcome measures.

Now I’ll briefly talk about public reporting.

• A number of quality measures for hospitals are reported to the public
• The goals of public reporting are
  • Encouraging hospitals to improve their performance on these measures
  • Reducing costs through avoiding preventable adverse events
  • Providing information for the public to make decisions about which hospitals they use
• History
  • Public reporting began ad-hoc in the 1980s
  • The first systematic, national reporting tied to a financial incentive was in 2003 with the Medicare Modernization Act
  • Public reporting of hospital quality measures has been a part of all the major national health care legislation since then, including the Affordable Care Act in 2010

(Note: I’m going to be talking about CMS Hospital Compare and the data published on it a lot in this presentation.)

HAIs are one of these quality measures.

• More specifically, HAIs are an outcome measure reported on CMS Hospital Compare
• HAIs are defined as infections that are a result of a hospital stay, so not something that was already present when a patient was admitted

• There are 5 types of HAIs reported on CMS Hospital Compare
  • CAUTIs are urinary tract infections occurring with a Foley catheter
  • CLABSIs are blood stream infections in patients with central lines
  • SSIs are infections at the site of a surgical procedure
  • You’ve all probably heard of MRSA before. One of the HAIs are blood stream infections positive for MRSA.
  • And finally, CDI is a bacterial infection of the GI tract that is often related to antibiotic treatment for a different infection.

• HAIs are relatively common, with more than 700,000 in 2011.
  • This means that 1 in 25 inpatients was affected by an HAI
  • The costs of HAIs are estimated at up to $45 billion in direct medical costs
• Many HAIs are also considered preventable
  • One study estimated that up to 70% of CAUTI and CLABSI are preventable

• We’re interested in improving understanding of HAIs because, as I just mentioned, they are common, often preventable, and costly in terms of mortality, morbidity, and medical costs
• Reporting of HAIs is expensive and time-consuming, so we want to make the most of the data that’s collected
• HAIs have some difficult data presentation challenges, which I’ll talk more about in a minute

The last part is “Assessing and Improving Patient Understanding”, which I’ll talk about now.

First, I’ll quickly define these terms.

In order to see how this applies to HAI data on CMS Hospital Compare, I want to show you what’s actually on the Hospital Compare website

There are two tables for presenting HAI data on the Hospital Compare website. The “simple table” is the default for presenting HAI data.

The “complex table” provides more of the underlying data. It is available through a “more details” button.

We’re going to be coming back to these tables throughout the presentation, so please refer to the handout if you don’t remember the details about them.

This is the “simple table”…

I’ve made a cleaner version of this table that’s easier to read. It shows the same HAI information as the screenshot. (Note that I also switched the rows and columns so it matches up with the “complex table”.)

• This is the first, default table you see when you look up HAIs on Hospital Compare
• Each row here is a hospital
• The one column of data is the written “evaluation,” which is a summary of the 95% CI for the standardized infection ratio. Basically, if the 95% CI is <1, the evaluation is “better than…”; if it crosses 1 it’s “the same as…”, and if it’s >1 then it’s “worse than…”. The handout has a diagram showing this.

This is the complex table. Again, this is available by clicking a “more details” button from the simple table.

Again, here’s the cleaner version of this table that’s easier to read.

I’m not going to spend any time on this slide because I go through all the columns on the next slide, but you can this table in the handout.

The “complex table” actually shows all the data that are publicly available. This slide is essentially listing out the columns in the “complex table” on the previous slide.

• HAIs are reported by hospitals as a number of infections (#1) and patient-time (#2)
  • So for CAUTI, this would be the number of UTIs in patients with catheters and the number of catheter-days
• An infection rate can be calculated by dividing #1 by #2
• CMS provides a predicted number of infections, which is based on characteristics of the hospital (#4). For example, CAUTI risk is based on the location reporting the infection, eg, surgical ICU.
• Risk adjustment is done by looking at the ratio of actual infections (#1) to predicted infections (#4). This is called the standardized infection ratio or SIR.
• And 95% confidence intervals are provided for the SIR.

Again, SIRs are the way that HAI measures are risk-adjusted. This is important because there are factors outside of a hospital’s control that affect HAI rates, like how sick the patient population they serve is. If you’re talking about comparing hospitals based on HAI data, risk-adjustment is important.

The CDC says that shifting base distortion is unlikely to be substantial with HAI data.

There is only one paper investigating this, and they show that it can happen in hypothetical data. But without the raw data for hospitals nationwide it’s not possible to determine how realistic their hypothetical examples are.

Because of the limited data that is reported by hospitals for HAIs, there really isn’t a better option for risk adjustment than indirect standardization. The data that are reported is unlikely to change, and public reporting is mandated by law. So we have to work within this limitation.

I think that more importantly, these data are publicly available and with the way they are currently presented it’s much more likely that incorrect comparisons of hospitals would come from mistakes in interpreting the data rather than potential shifting-base distortion.

Our primary goal is to address the major issues in basic understanding of the data, but we keep the potential of shifting-base distortion in mind throughout this research.

Remember, we were talking about improving patient understanding.

So why do we think people don’t understand these data?

In the interest of time, I’m only talking about a few reasons why we think this.

(There are lots of other reasons why these data are difficult to understand, and examples of how CMS Hospital Compare compounds these issues with poor design.)

With that said, let’s look at the complex table again.

Consider the SIR column.

This column is arguably the most important numerical column here because it’s the only column that takes the infection rate and risk-adjustment into account.

However, this is difficult to understand for several reasons…

Even if the viewer understands the SIR, it’s not clear which column or columns to focus on.

More generally, we know interpreting data like this is difficult for a large portion of the general population.

55% of Americans have basic or below-basic quantitative literacy, which includes some of the skills needed to interpret data like the “complex table”.

The upshot of this is…

[see slide]

The overall premise of my dissertation is to address the issues that I just talked about with presenting HAI data to the public.

I did this in three aims:

Aim 1 assessed understanding of HAI data on CMS Hospital Compare. Just looking at the way the data are presented on this website, we were fairly confident that understanding would be low, but of course we needed to test this experimentally.

Aim 2 looked at the national HAI data, with two goals in mind. The first was to see if it actually does distinguish among hospitals, assuming the viewer understands it perfectly. For example, if most hospitals report 0 infections, that is really good to know as a patient but it doesn’t help you choose a hospital.

The second goal was to see if the tables on CMS Hospital Compare are suitable for presenting the actual HAI data reported by hospitals, given characteristics of these data. This is really abstract, but I’ll make it more concrete when I talk about Aim 2 in detail.

Aim 3 involved designing a new way of presenting HAI data to the public, and then testing that against CMS Hospital Compare in a randomized controlled trial.

Again, the goal of Aim 1 was to assess understanding of HAI data on the current CMS Hospital Compare website.

This is the part of the survey where we asked participants to compare hospitals.

Study population was fairly diverse, with a range of incomes and education.

We asked about health experience because we thought that this might have some effect on ability to understand the data.

Our study population had a lot of experience with the healthcare system, with almost 90% of the sample hospitalize more than two times. One third of participants had been healthcare workers or were currently.

We also asked participants if they had used a hospital comparison website previously, and if the hospital comparison website would have helped in their decision to come to UMMC

Few had used a hospital comparison website before.

However, it is surprising that so many participants (36%) said a website for comparing hospitals would be helpful. This includes a large portion of participants who did not have any choice in hospital, such as transplant patients, who generally always go back to where their transplant was done. We expect this percentage would be even higher in a population of patients who did have a choice among hospitals.

Results from the hospital comparison portion of the survey:

This figure shows the mean percent correct across all n=110 participants split up by task. Remember that the tasks are 3 questions each: Task 1 was 3 questions comparing 2 hospitals with the “simple table”. Tasks 2, 3, and 4 were also 3 questions each, comparing 2 hospitals with the “complex table”. Task 2 was designed to be easier than 3, and 3 easier than 4.

For example, participants got 72% of the Task 1 questions correct, with a 95% CI from about 65 to 80.

We see scores decreasing as tasks get more difficult.

Recall, Task 1 used the “simple table”, while the other tasks used the “complex table”.

Thus, participants performed at a D level or worse when using the complex table.

One obvious critique of this work was that our study population was too sick to answer the questions, though our observations during interviews did not support this.

To address this, we did a sub-analysis including only participants who got >66% correct for the first (easy) task.

We see the same pattern of decreasing scores as the questions get harder, and the averages for each task aren’t substantially different from those with the full sample.

We also did a sub-analysis with just participants who were ever health care workers and participants who cared for a frequently hospitalized person.

The purpose of this was to see if experience with the health care system was associated with performance. It does not appear to be strongly associated.

Additionally, hypothetical HAI data in survey is less complex than real world. → For example, the survey questions only had 2 hospitals, while actual searches could have many more.

Thus, we expect even poorer performance for understanding data from actual searches on CMS Hospital Compare.

Strengths of Aim 1 include the large sample size (n=110), random selection of participants, and the method for presenting participants with HAI data on an iPad to ensure that the formatting of the HAI data were identical to how they appear on CMS Hospital Compare.

The primary limitation is that the study population may not represent members of the general public. However, our population was fairly diverse. Additionally, we did a sub-analysis including only those who got at least 2/3 correct for the easiest questions in the survey. These results were nearly identical to those for the entire sample, which indicates our sample did not consist of patients too sick to complete the survey.

Again Aim 2 looked at the national HAI data, with two goals in mind.

The first was to see if it actually does distinguish among hospitals, assuming the viewer understands it perfectly. For example, if most hospitals report 0 infections, that is really good to know as a patient but it doesn’t help you choose a hospital.

The second goal was to see if characteristics of the national data match up with the way its presented on CMS Hospital Compare.

First I’m going to talk about how we did the analysis for this aim.

We’re interested in the characteristics of the HAI data on actual searches on the hospital comparison website.

So for example, let’s say I want to search to compare the hospitals here in Baltimore, and then ask questions about these hospitals such as: Do all the hospitals here report data for the 6 HAIs on CMS Hospital Compare? Do all the hospitals here have zero infections?

But we want to do this nationwide, so we say all the hospitals in Baltimore are one group of hospitals, and then we find all the other groups of hospitals in the country, asked the same questions for all of them, and then aggregate the results.

The first method for grouping hospitals is with simulated searches. This starts with identifying all the “urban areas” (towns or cities with >2,500 people). These are represented by colored regions on the map (the geographically smaller ones look like dots).

We then look for the closest 10 hospitals within 100 miles of the urban area. This is a group of hospitals.

(The purple arrow is pointing at an urban area, which is in the center of the search circle.)

We then look at the HAI data for these hospitals.

And we repeat this for groups of hospitals from all the urban areas in the country.

The second method for grouping hospitals is with HRRs. These are geographic areas based on “where patients were referred for major cardiovascular surgical procedures and for neurosurgery.” They are defined by the Dartmouth Atlas, and have been used before in this type of analysis.

In the map, each colored region is a HRR.

Each HRR has a number of hospitals inside of it (between 2 and 83).

Let’s say the HRR we are in now includes all the hospitals in Baltimore and DC. We would consider all the hospitals in this HRR to be a group of hospitals.

We use these two ways of grouping hospitals to answer 4 analysis questions, which I’ll go through one by one. We answered these questions first with simulated searches and then separately with HRRs.

These percentages are from the simulated search groups of hosptials. The HRR numbers were similar so I won’t show them in the interest of time.

We see that 78% of groups of hospitals in our analysis had at least half their hospitals reported CDI data.

Takeaways:

• If you are searching for a hospital, C. difficile is likely to be reported by the majority of hospitals in your area.
• You may have less luck with the other types of HAIs; it will depend on where you are located.

Why do hospitals not report data? It’s not required so some just don’t do it. Others are too small so CMS lists them as not reporting the data.

For Question 2, we were trying to see for the hospitals that do report HAI data, if these data actually differentiate between hospitals.

Again, we did this analysis with both the simulated search method for grouping hospitals and the HRR method.

These are from the simulated search method. The HRR method has similar numbers, which I’m not presenting in the interest of time.

Takeaways:

• If you are searching for a hospital, there is a 50% chance that at least one pair of hospitals in your search results will have very different performance with C. difficile.
• There is a 32% chance of this for CAUTI.
• It is much less likely for the other HAIs.

For Question 3, we are looking at how often the CMS Hospital Compare “simple table” is misleading when presenting actual HAI data.

Recall, the “simple table” only shows the written “evaluation” for the hospitals (only the black text in the table on the slide). If the 95% CIs for a pair of hospitals are both on the same side of 1 but do not overlap, the hospitals could perform quite differently despite looking identical in this table.

Focused on the bold numbers, which show the % of groups of hospitals where at least one pair of hospitals in the group fit the criteria shown in the previous slide.

Ex: for C. difficile, 15% of groups of hospitals will have at least one pair of hospitals that appear to have identical performance in the “simple table”, but in fact have differing performance numerically. With simulated searches, this is 4% for C. difficile.

Takeaways:

• Results differ somewhat by method of grouping hospitals.
• Not unexpected, because groups of hospitals from simulated searches include only 10 hospitals, while groups of hospitals from HRRs can include many more.
• Thus, the relevance of these data for search results depends on how you are searching. It is less important if you are only looking at the 10 closest hospitals to your location. But if you are comparing across the hospitals in your region, this may occur more frequently.
• Data presentation methods should never be misleading, even 5% or 10% of the time. So the “simple table” should be changed to avoid this problem.

Question 4 was based off of an observation from Aim 1: some people only look at the first column in the “complex table”. We wanted to know how often this would lead to incorrect conclusions when looking at the actual HAI data.

Again, focus on the bold numbers.

Takeaways:

• Similar results for both methods of grouping hospitals.
• Looking at just the first column is often misleading for C. difficile data (>90%), and is ≥50% for CAUTI and CLABSI
• Combined with our Aim 1 observations, this strongly suggests that the “complex table” should not have the # of infections in the left-most column.

How is “misleading” defined? One or both of the following scenarios must occur:

Scenario 1:

a) The evaluation column is different for the two hospitals, AND

b) The hospital with the better evaluation column has a greater number in the # infections column

Scenario 2:

a) The evaluation column is the same for the two hospitals, AND

b) One of the hospitals has > 10 infections, AND

c) One hospital has at least 50% fewer infections than the other

Strengths of Aim 2 include using 2 methods for geographically grouping hospitals. Additionally, both methods yielded fairly similar results. We used nationwide data and included all the HAIs in the CMS dataset in analysis.

The primary limitation of this aim is that while we think the geographic grouping methods are good because they both had fairly similar results, they can’t be a perfect proxy for the hospitals that people actually are comparing.

Question 2 is limited because the only risk-adjusted measure available for HAIs is SIRs, and we’ve already discussed the issues with SIRs and shifting-base distortion.

Moving on to Aim 3…

In this aim, we developed a new method for presenting HAI data, and then used a randomized controlled trial to compare understanding between the new method and CMS Hospital Compare.

First, we defined a systematic design process. This was codified in a “design document” that described the goals of designing the data presentation method from a high level down to implementation details.

Next, we reviewed references on effective techniques for data presentation. (More at http://bit.ly/datatips.)

We then created an initial implementation of the new data presentation method based on the design document and a review of relevant data presentation literature.

Finally, we conducted user testing to iteratively refine the initial implementation of the new data presentation method.

This is what we started with for the initial implementation. I’m not going to spend any time on this because the final version was so different, but if you’re interested in the progression between this and the final version, there’s draft manuscript about this linked up at the URL on the handout.

We arrived at the final version after many iterations and one-on-one user testing with n=14 participants at UMBC.

This is the final version of the new design. Features include:

• “5 dot” method for visualizing SIRs
• Written description of SIR 95% CI: “high” / “medium” / “low”
• Color used as shorthand to signify green=good and red=bad, but color is not necessary for understanding the figure

The cutoffs used for the “5 dots” and the 95% CI summaries are based on the standard deviation and IRQ for national HAI data. These cutoffs could be easily modified if improved cutoffs can be determined without affecting the understandability of the data.

See the Aim 3 survey for details.

Detailed information on the HAI data are available on demand.

See the Aim 3 survey for details.

We then took this new way of presenting HAI data and compared it to the “complex table” method on CMS Hospital Compare.

[See slide…]

And like Aim 1, units like ICUs, obstetrics, psychiatry were excluded to avoid disruption in care. Also, patients were excluded if they (1) were not able to be found after multiple attempts, (2) were unable to participate for mental or physical reasons, or (3) couldn’t read or speak English.

The Aim 3 survey was identical to the survey for Aim 1 except for the hosptial comparison tasks, for which participants were randomized into one of the two arms.

There were 12 hospital comparison tasks. These questions used realistic data for two hypothetical hospitals.

One participant was interrupted in the middle of the survey, so they could not complete it. The demographics questions are at the end, so we don’t have any of this information for them. They are included in the analysis still under intention-to-treat.

There are two important take-aways from this slide.

First, the study population was fairly diverse, with a range of incomes and education.

Second, there were no statistically significant differences between study arms.

We asked several questions to gauge experience with the health care system. We asked the same questions in Aim 1.

Again, there are no statistically significant differences between the study arms.

We also asked whether participants had used a website for comparing hospitals before, and if such a website would have helped with their decision to come to UMMC.

In Aim 1, 6% answered “yes” to the first question, and 36% answered “yes” to the second question. This is fairly similar to what we see here.

Again, there are no statistically significant differences between the study arms.

Drum roll…

The average score in the experimental arm was 56%, compared with 32% in the control arm, with a highly significant p value.

A 24% difference is a greater than what we considered practically relevant (15 to 20%).

Excluding the participant who did not complete the survey does not change the results.

It took participants bout the same amount of time to answer the questions in both study arms.

The strengths of this aim are the randomized controlled trial design. Qualitative study designs like focus groups are the status quo for this kind of research, and our study design is much more rigorous. The study population was also fairly diverse.

The primary limitation is generalizability, just like with Aim 1.

The next steps with this work are to do more testing of the new presentation method and make further improvements.

The long-term goal is to actually change CMS Hospital Compare. I’ve created http://haicompare.org, which demonstrates the new presentation method using the actual Hospital Compare dataset on a live website. You can go there and try it out yourself if you want.

I’m also applying for a job at the U.S. Digital Service, which is the government agency that is tasked with improving all the crappy federal government websites. If I worked there, I might actually be able to make changes to CMS Hospital Compare myself!

Genomic Epidemiology and Clinical Outcomes

Also I would like to thank all the other epi faculty and students I’ve known here.

1. Magill 2014
2. Scott 2009
3. Umscheid 2011
4. Birnbaum 2011
5. Kutner 2007
6. Safavi 2014