I talk a lot about data bias and methodology. My clients understand that how we ask the question often determines the answer. More specifically, how we select and weight data, filter candidates and build the model can lead to very different predictions and rankings.
With so much data available, what you data features you select to answer those questions is just as important. Often there are no right answers. Maybe not even best answers.
I’m fairly wired into the “Best Places for X” publishing phenomenon. It drives users and engagement. Period. I track stories to see what’s doing well and what my clients’ competitors are up to. And guess what? Publications cover the same topics all the time.
And sometimes their Top 10 lists are completely different.
Spring season and college graduations are upon us, so Best Cities for Jobs are a hot topic at the moment. Two recent lists from GlassDoor.com and Forbes.com perfectly illustrate how intent and content selection determine the results.
Two studies. Two very different lists.
OK, the Bay Area looks pretty good. And oddly Austin is #6 on both lists. Other than that only Raleigh is common to both lists. How can that be? Isn’t this a story about jobs? How can there be so much disagreement on the best places to find them?
Glassdoor’s Best Cities for Jobs
- San Francisco, CA
- San Jose, CA
- Orlando, FL
- Nashville, TN
- Dallas, TX
- Austin, TX
- Denver, CO
- Charlotte, NC
- Raleigh, NC
- Portland, OR
Forbes’ Best Cities for Jobs
- San Jose, CA
- San Francisco, CA
- Seattle, WA
- Boston, MA
- Washington, DC
- Austin, TX
- Salt Lake City, UT
- Raleigh-Durham, NC
- Minneapolis-St. Paul, MN
- Oklahoma City, OK
Why so different?
Its the methodology.
Starting with the question they ask, the data selected, and how its modeled to rankings. The publications have different intent. Forbes is reporting facts and hard news focused only on one thing: employment growth over time. GlassDoor is promoting their own internal job satisfaction and jobs data. On top of that, there are other critical differences.
Filters – what do they even mean by “city”?
Turns out Forbes is using Metropolitan Statistical Areas (MSA’s). Glassdoor is using, well, Glassdoor cities. They tell us these are the 50 largest metros but don’t disclose whether they are MSA’s, but it seems likely.
What’s the impact: same city definitions should encourage similar results.
Forbes: MSA’s with 450,000+ employed
Glassdoor: Cities (presumably MSA’s)
Feature Selection – what data are they using?
What does it mean to be great for jobs? Ultimately, thats the question we’re answering here.
Forbes takes a straight forward approach focused on near term, mid-term and long term job growth. That’s it.
Glassdoor uses a different, but presumably related metric: how easy it is to get a job. To find that, they look at the number of jobs openings adjusted for city size. But there methodology is more complex, adding “soft” factors such as job satisfaction and work life balance. Also included is cost of living.* So, not just jobs. Maybe there’s even a good recommendation here for Talia Jane!
What’s the impact: different input features are going to create different output.
Forbes: singular focus on job growth. Intent is clearly an employment driven ranking.
Glassdoor: focus on other factors exposes intent to identify great places to work where jobs are available. This is a fundamentally different approach geared toward satisfaction.
Weights and Methods
Forbes uses a somewhat complex point system geared to evaluate job growth in three ranges (short, mid and long term).
Glassdoor weights each of their four factors equally.
What’s the impact: since the two publications are using different features, the way they weight them can’t be compared.
So which is better?
Personally, I like the Glassdoor approach. But that’s more a comment on their intent to identify great places to find a job and live – which is a more engaging topic. But neither story is a better data story…they are simply different.
And that’s really the point. There are many possible Best for Jobs stories. Picking the best one for your brand will be the right choice. And that’s something we can help you do not only by providing the content and analysis but also with crafting the story intent and selecting features, weights and filters. Find out how we can help you generate buzz, traffic and business!
* a note about Cost of Living: Glassdoor defines it as the ratio between median base salary and median home value. This is an example within an example about feature selection. There are many ways to define cost of living. Home values are a part of the picture, but so are market goods costs, average rental costs, transportation expenses and taxes (sales, state and local income tax, etc.). To get a complete picture, you need to look at all of this and more.