Check out the Nonprofit Data Scorecard here.

A big part of the value we see coming from the Nonprofit Data Scorecard is in its transparency and responsiveness to the public. Early on in the Scorecard’s development, we decided that it was crucial to be as rigorous and systematic as we could in our data collection and rankings for the end product to have any value to users.

So, with these intentions, it’s time to say a few words about methodology. How did we put the Scorecard together? How did we arrive at the ranking categories – and assign those rankings? What was the method to our madness?

Gathering the Data

Our first task was finding where the best free and public data lived online. After some initial rooting around, it became clear that most state governments maintain some kind of database of the nonprofits operating in their borders, usually under the guise “Charitable Statistics.” State Attorney General’s offices and Secretary of State’s offices were the homes of these data sets, and we eventually began to use this directory provided by the National Association of State Charity Officials as a go-to resource.

Take the case of Illinois. The charitable statistics for that state are hosted on the Attorney General’s Office website, where you can search their database with a fairly user-friendly interface – searchable using basic information like organization name, address and registration number. If I search “Sierra Club” for instance, I’m brought to brief but clear data about that organization’s location, registration number and recent financial history.

Illinois clocks in with a final grade of “C” – 3 stars out of 5 – which means that the dataset Illinois makes available online was mediocre in terms of the level of detail, how up-to-date the information in it is, and how user-friendly the dataset is. This means that for some states the hunt for data was easier, cleaner and faster, and for some states it was tedious, messy and slow: sometimes with searchable interfaces, sometimes without, sometimes with no data at all, and sometimes with data squirreled away in hard-to-read PDF’s.

The Ranking Process

The Scorecard’s usefulness hinges on the relevancy of its ranking criteria. The ranking categories were inspired to some extent by what is generally considered the standard in nonprofit data sets – those offered, for a stiff fee, by GuideStar and other reporting outlets. But even GuideStar, whose data set is prohibitively expensive, is not necessarily that much more timely or complete than what is maintained by state governments.

Each of the 7 categories – Relevancy, Searchability, Downloadability, Timeliness, Historical Trending, Comprehensiveness and Accuracy – were tuned to the best of what GuideStar offers, and to the ease of use for folks like us who don’t have the budget to buy access to data or the time to fish around in messy, obscure state PDF files.

I asked Mary Catherine Plunkett, who played a key role in developing the ranking criteria, to say a few words about each. Here are some of her thoughts on rationale:

On Searchability: “A key to data transparency is providing a searchable user interface that allows you to search by multiple data fields. For example, an interface where you need to have an ID number to search is not very user friendly, while a search engine that allows keywords or partial information is much more useful. There’s a reason so many people use Google!”

On Downloadability: “A critical trait for using a dataset for further analysis, and most states lagged on this measure. You could search for “Sierra Club”, but you couldn’t pull down the entire dataset for additional analysis, mashups, and remixes. This is an area where in which most states can improve.”

On Timeliness and Historical Trending: “Having the ability to look at changes in data over time instead of a “snapshot” is required to uncover trends such as non-profit closures and openings. A regularly updated dataset is also required to prevent the “garbage in, garbage out” issue that can occur with the analysis of a low quality dataset.”

On Comprehensiveness and Accuracy: “These criteria also fall into the “garbage in, garbage out” camp. You want a full and accurate picture of the real world to be reflected in your dataset. We used spot checks with third-party, high-use websites like Idealist.org and WiserEarth.org to sample the comprehensiveness and accuracy of the data from each state.”

Nonprofit data from all 50 states was given a score between 1 and 5 for each category, and then aggregated into a final score. The average final score for all 50 states was 2, and the final score for federal nonprofit data housed within irs.gov was 3.

So there you have it! The legwork of finding and combing through state datasets, developing ranking criteria, and assigning ranks to each state was done, graciously, by our all-volunteer team. Special thanks goes out to Mary Catherine Plunkett, the primary creator of the Scorecard’s ranking criteria, to SVT Group who advised on the criteria, to Julia Tran and Tracy Greene of SVT Group, and to yours truly (Sara insists I include myself), whose time, enthusiasm and meticulous attention to detail made all of this possible.

Thank you – and please review, critique, improve, remix and repurpose!