Meaningful metrics?

[mtift]

Is there any desire for this group to work toward specific, measurable goals? For example, I know that certain states in the US focus on workers, with goals such as "increasing the percentage of people of color who work in state government." Other organizations focus on representation, recruitment, training, etc. I am certainly no expert in this area (my background is critical theory and musicology), but my experiences in various activist organizations has shown me that the successful ones find some way to measure successes and continually return to those metrics when deciding on activities, initiatives, etc.

I am not exactly sure what metrics might be meaningful to the Drupal community, so I am looking for input. I have a few ideas, such as:

• Increase the diversity of active users with a Drupal.org profile (from X% to Y%)
• Increase the diversity of speakers/participants/sessions at DrupalCamps, DrupalCons, or Drupal user groups (from X% to Y%)
• Increase the diversity of organizations/employees active in the Drupal community/codebase/etc (from X% to Y%)
• Increase the diversity of participants/code contributions/etc in issue queues (from X% to Y%)

With some exceptions, I don't know that we have good data for most of these items. However, there has been momentum in recent years around the topic of issue credits, which now play a major role in the order of organizations listed on the Marketplace page on Drupal.org, and I think there is an opportunity there that could -- at least in part -- inform the mission of DD&I.

I thought a lot about diversity goals when I was looking into issue credit data last year. Although Dries Buytaert and I wrote about sponsorship in Drupal (which we are starting to re-examine again), we spent considerable time trying to find some sort of meaningful metric with regards to diversity. I crossed-reference issue credit data with Drupal.org profile information in hopes of better understanding the diversity of our community. We only have limited profile information with regards to diversity, so most of the questions I could ask revolved around the notorious gender field, such as:

• What organizations had the most credits from women/other/transgender individuals? (Acquia)
• Since we started tracking commit data, more or fewer non-male contributors? (mostly flat)
• Overall, what is the breakdown of credits by gender? (93% male)
• How does that percentage change over time? (more men are participating)
• How does that percentage change around DrupalCon? (not much)

While fully acknowledging that issue credit data might seem insignificant, the charts/rankings from it might help understand how well we are doing with increasing d&i of communities and organizations, making the community more welcoming to a diverse group of people, etc. We could, for example, decide on a metric and work to get a section on the Marketplace page that lists (ranks) the companies in the community who are paying people who identify as diverse to work on Drupal, and the DD&I supporters could always make a point to highlight those companies in blog posts, podcasts, conference talks, etc. But that's just one idea.

In the coming months, I will be studying these data either way, but my preference is to make the work as relevant as possible. I've already begun discussions with various people at the Drupal Association about what they would like to see measured, but I am also interested in hearing more from, and collaborating with, (individuals in) this group.

So, in addition to the other goals of providing a safe space to discuss issues of d&i within the Drupal community, providing support for people who are marginalized in technology, etc. is there any interest in finding some specific, measurable goals or metrics?

[mtift]

Someone just mentioned NTEN's "Digital Inclusion Toolkit" in the Slack, which mentions "a few tips for building similar programs in other organizations," including:

Measure outcomes. Your project should be designed so that impact can be measured. Know what change in behavior or knowledge you want to achieve with your program and devise a way to measure it before you begin.

[bradleyfields]

Agree we should be thinking about both what success is and how we measure it. This issue shares a fair amount of ground with drupaldiversity/event-organizer-packet#9. Maybe we merge them.

My comment there was a hesitation re: quotas, which I understand the mere idea of metrics generally isn't. Though, I would want to make sure we found ways to evaluate inclusion and not only diversity. E.g., if we measure the diversity of speakers/participants/sessions, we should also know how many of those speakers speak at Drupal events multiple times. I think your second set of bulleted points offers some ideas that could get us there (e.g., contribution over time).

I'd also be curious about tracking role maturity. Like, which people come in as entry level contributors and over time become speakers, or core maintainers, or board members, etc. If we were to find that only certain kinds of people take those journeys (or are given those opportunities), it'd be good to ask ourselves why.

[sparklingrobots]

As part of the effort to develop a diversity-and-inclusion focused mentorship program, I've found myself thinking about measurement a lot. The metrics tracked on d.o are pretty terrible and I don't expect that to be fixed before our project starts (or...ever).

I'd like to be able to take our own metrics and make our own reports about the work we are doing. It seems reasonable that ddi & the mentoring group’s metrics could be formally aligned; also would make it easier to talk to the community about how to be better.

[sparklingrobots]

Found this via @drnikki's blog: http://www.isdnetwork.org/what-is-diversity.html Seems like a relevant starting place.

[agentrickard]

moving to d.o.

[agentrickard]

https://www.drupal.org/node/2929249

[drnikki]

thanks @agentrickard !! closing