In my days doing epidemiological modeling, I always heard the phrase “Models are always wrong, but some are useful” (attributed to George Box). And, I’m struck by the potential utility of this mindset when looking at the various population health data reports, summaries and rankings published with regularity.
Over the past few weeks, there have been some big analysis published by City Health Dashboard and County Health Rankings – both funded by Robert Wood Johnson Foundation and both included data for the relatively small city (and county) where I now live.
Like most people, the first thing I do when I see a headline of best or worst cities is to look up where I live, to see where it falls in the list. As I wrote about in March, Wilmington, NC is doing terribly when it comes to well-being. But, I’m optimistic because the New Hanover county ranks 12th in the state (out of 100 counties in North Carolina) for health outcomes. However, as I dug more into the actual metrics and data, I ended up with more questions than answers about the current state of health in my neighborhood.
What results were the same
There was some alignment about what is going well here, especially in comparison to other cities and counties. Specifically, Wilmington/New Hanover county is in a good place* when it comes to the following:
- Physical activity rates are higher/better than average (in alignment with Well-Being Index results)
- Lower teen birth rate than average
- High levels of preventive care, dental care, and primary care, with a large provider to resident ratio
Wilmington/New Hanover county is not in a good place* when it comes to the following:
- Child poverty rates are higher/worse than average
- Housing costs are higher/worse than average
- Income inequality is worse than average
- Segregation is worse than average
- Violent crime is higher than average
What results were different
Where the real confusion set in was around areas where these two recent analyses conflicted, so I dug into those a bit more.
- High school graduation rate – Depending on which website you look at, you get a different story about how well the area is doing in getting high school students to graduate. I recognize that there is a difference between the county and the city as far as populations included and school districts, which may play into this metric. So, is Wilmington doing slightly better than average cities while New Hanover county is doing worse than both state and national averages?
According to the City Health results “84.3% of Wilmington’s public school students graduated high school on time, compared to an average of 83.7% across the Dashboard’s 500 cities.” (for timing of data, text says “year varies by state).
According to the County Health Rankings, 82% of 9th grade students graduate in 4 years (using the 2014-2015 data), while the state average is 86%. As reference, national average for the 2015-2016 school year is 84.1%. And, when you look at the chart above, the concluding line at the bottom is “New Hanover County is getting better for this measure” – can we really say that with authority? It looks like the rate has been relatively stable for the last 4 years of data after a marked increase over the 2010-11 value.
2. Premature death – While these results are using data that is over a slightly different time period, the resulting insights are opposite. Are we doing better or worse than other communities at a population level, and what about health inequality by race?
According to the City Health results “Wilmington had 8700 years of potential life lost per 100,000 population, compared to an average of 7407 across the Dashboard’s 500 cities” (data from 2013-15).
According to the County Health Rankings, New Hanover County has 6,700 years of potential life lost before age 75 per 100,000 population (age-adjusted, data from 2014-16), while the average for North Carolina is 7,300. However, when you dig in for more details, the years of life loss rate is much different by race within the county: 11,800 for Blacks, 3,800 for Hispanics, and 6,000 for Whites. So, we are doing worse than average for some parts of the population, and that unfortunately isn’t part of the ranking system.
3. Food Access – Access to healthy food seems to be a challenge within Wilmington as compared to other cities, while New Hanover county is about average when looking at food access among low-income people within North Carolina. Again, is healthy food access an issue or not?
According to the City Health results “69% of Wilmington’s residents had limited access to healthy food, compared to an average of 61.9% across the Dashboard’s 500 cities” (2015 data).
According to the County Health Rankings, 8% of population who are low-income and do not live close to a grocery store in New Hanover county, as compared to the state average of 7%.
Why does it matter
Whether local health departments like it or not, any type of ranking or comparison among communities leads to opinions and questions. The opinions can be on if the data are fair (or not), what should (or should not) be done in response, and what to prioritize in the future. The questions can be on the data sources, accuracy, or methods. And, while the two analyses I looked at for this article were done for different purposes, using different geographic units, and with likely different goals, it is not unreasonable to think that a busy politician or community member may only look at one such report or headline when pressuring the local health department. And, there are likely additional similar outputs that I could have found that may have given more results. So, are tools like these enabling and informing conversation towards improved public health, or are they causing more noise in an already overwhelmed world of data (especially for smaller jurisdictions)?
And, given the time lag between data collections and results, are these reports too stale by the time they are produced? Here in New Hanover county, the two major public health issues that are front of mind are the opioid crisis (where the Cape Fear region remains on the top of lists of opioid deaths and overdoses) and contaminated water (from Gen-X and fluoride). Neither of those current issues are part of these assessments, but need to stay at the top of priority lists for health officials, politicians, and the public.
*Analysis note: When looking at the City Health data, if the difference between the 500 city average was <1%, I classified it as neutral. When looking at the County Health Rankings, I used the “areas of strength” and “areas to explore” to classify what was better or worse than average.