Check out the Nonprofit Data Scorecard here.

Collaboration and team work are the engines that have made the Nonprofit Data Scorecard a reality. The energies of our small team of volunteers have begun to chip away at a few of the big questions facing the nonprofit world: How has the recession impacted NPO’s? Who are the holders of the data required for answering that question? How can that data be improved?

We’ve put together what we think is a robust and helpful Scorecard for ranking state nonprofit data sets and getting at some of those questions, but we hope that’s just the beginning of a longer, more participatory conversation about data availability. Because we’re humans – and just a small group with limited resources – we know that the best way to sharpen our work is to engage with you, the public, to question, contest, comment on and revise the data behind the Scorecard. So consider this an open source invitation – an invitation to take community ownership of the spirit behind the Scorecard, and to sharpen and improve the work we’ve presented thus far.

Key to the vision of Nonprofitmapping.org is a notion of active and total transparency, backed by a functional crowdsourcing strategy. So, in that spirit, we’ve shared with you the methodology behind the Scorecard’s creation, the entire data set from which it is sourced, and the tools we’ve used to present that data geospatially. We encourage you to check all that out, and leave constructive critiques, challenges, comments, and new information where warranted. Here’s how:

*Read our blog series on the Nonprofit Data Scorecard, and leave your comments at the bottom of posts.

*Visit the Scorecard page and click the “view data” button under the map. Have a look at the results for your state. If you feel you’ve got better information about a particular category -say, where to find better nonprofit statistics in that state – let us know! Click the “improve data” button below the map to send your feedback to the team for review.

*Share! On the Scorecard page, click the “embed” button to copy/paste the code behind the interactive map into your website or blog. If you’re a journalist or blogger, maybe you’ll find it a  useful springboard to a larger discussion about government data. Or if you’re a state government employee, maybe you’d like to compare the quality of your state’s data set to others. And if you’re in the philanthropy or nonprofit community, maybe you’ll find the map useful for strategically connecting projects and funders.

*Mashup! We’ve provided all of our raw data in downloadable, spreadsheet format for folks who’d like to take it with them, combine it with other data, or churn out their own visualization. For tips on free and easy-to-use visualization tools, see this post on ManyEyes and this one on Fusion Tables.

We’d like to see the Scorecard become a community-built tool for changing the culture of nonprofit data. Help it happen by getting involved! Don’t hesitate to drop us an email at nonprofitmap@gmail.com with further questions – and don’t spare us your critiques!

Check out the Nonprofit Data Scorecard here.

A big part of the value we see coming from the Nonprofit Data Scorecard is in its transparency and responsiveness to the public. Early on in the Scorecard’s development, we decided that it was crucial to be as rigorous and systematic as we could in our data collection and rankings for the end product to have any value to users.

So, with these intentions, it’s time to say a few words about methodology. How did we put the Scorecard together? How did we arrive at the ranking categories – and assign those rankings? What was the method to our madness?

Gathering the Data

Our first task was finding where the best free and public data lived online. After some initial rooting around, it became clear that most state governments maintain some kind of database of the nonprofits operating in their borders, usually under the guise “Charitable Statistics.” State Attorney General’s offices and Secretary of State’s offices were the homes of these data sets, and we eventually began to use this directory provided by the National Association of State Charity Officials as a go-to resource.

Take the case of Illinois. The charitable statistics for that state are hosted on the Attorney General’s Office website, where you can search their database with a fairly user-friendly interface – searchable using basic information like organization name, address and registration number. If I search “Sierra Club” for instance, I’m brought to brief but clear data about that organization’s location, registration number and recent financial history.

Illinois clocks in with a final grade of “C” – 3 stars out of 5 – which means that the dataset Illinois makes available online was mediocre in terms of the level of detail, how up-to-date the information in it is, and how user-friendly the dataset is. This means that for some states the hunt for data was easier, cleaner and faster, and for some states it was tedious, messy and slow: sometimes with searchable interfaces, sometimes without, sometimes with no data at all, and sometimes with data squirreled away in hard-to-read PDF’s.

The Ranking Process

The Scorecard’s usefulness hinges on the relevancy of its ranking criteria. The ranking categories were inspired to some extent by what is generally considered the standard in nonprofit data sets – those offered, for a stiff fee, by GuideStar and other reporting outlets. But even GuideStar, whose data set is prohibitively expensive, is not necessarily that much more timely or complete than what is maintained by state governments.

Each of the 7 categories – Relevancy, Searchability, Downloadability, Timeliness, Historical Trending, Comprehensiveness and Accuracy – were tuned to the best of what GuideStar offers, and to the ease of use for folks like us who don’t have the budget to buy access to data or the time to fish around in messy, obscure state PDF files.

I asked Mary Catherine Plunkett, who played a key role in developing the ranking criteria, to say a few words about each. Here are some of her thoughts on rationale:

On Searchability: “A key to data transparency is providing a searchable user interface that allows you to search by multiple data fields. For example, an interface where you need to have an ID number to search is not very user friendly, while a search engine that allows keywords or partial information is much more useful. There’s a reason so many people use Google!”

On Downloadability: “A critical trait for using a dataset for further analysis, and most states lagged on this measure. You could search for “Sierra Club”, but you couldn’t pull down the entire dataset for additional analysis, mashups, and remixes. This is an area where in which most states can improve.”

On Timeliness and Historical Trending: “Having the ability to look at changes in data over time instead of a “snapshot” is required to uncover trends such as non-profit closures and openings. A regularly updated dataset is also required to prevent the “garbage in, garbage out” issue that can occur with the analysis of a low quality dataset.”

On Comprehensiveness and Accuracy: “These criteria also fall into the “garbage in, garbage out” camp. You want a full and accurate picture of the real world to be reflected in your dataset. We used spot checks with third-party, high-use websites like Idealist.org and WiserEarth.org to sample the comprehensiveness and accuracy of the data from each state.”

Nonprofit data from all 50 states was given a score between 1 and 5 for each category, and then aggregated into a final score. The average final score for all 50 states was 2, and the final score for federal nonprofit data housed within irs.gov was 3.

So there you have it! The legwork of finding and combing through state datasets, developing ranking criteria, and assigning ranks to each state was done, graciously, by our all-volunteer team. Special thanks goes out to Mary Catherine Plunkett, the primary creator of the Scorecard’s ranking criteria, to SVT Group who advised on the criteria, to Julia Tran and Tracy Greene of SVT Group, and to yours truly (Sara insists I include myself), whose time, enthusiasm and meticulous attention to detail made all of this possible.

Thank you – and please review, critique, improve, remix and repurpose!