Tip:
Highlight text to annotate it
X
Welcome to the second part of our metrics and measurements. I just saw this amazing
article in the Smithsonian magazine about innovation in the December issue and it talked
about how you need to have communities to innovate. These are the clovis points that
came along with language carved figurines, fire. This was the a-ha moment of unleashing
change in human behavior and becoming widely disperse on earth and I feel like we’re
at that point right now, but we need to pay attention to it, we need to capture it. One
of the things that we talk about why metrics and measurements matter, is because it’s
the story telling Cristin [Dorgelo] was alluding to, the stories and sharing this information,
conveying our lessons learned, giving this insight to the people who are competing, to
the people who are running the challenge, to the people who might be interested in learning
about the solution. All of this needs to happen for advancement in the technology and innovation
space. So one of the key things Karen and I talk about when we meet with you all, if
you haven’t heard it at nauseum, is aligning your challenge with your mission, and this
is going to be key for setting up your metrics and your measurements. If you can clearly
define your problem within your agency, that’s according to your mission, when you’re running
your challenge competition, you’re going to find that this sets a framework for people
to be anchored to what you’re talking about. They are passionate about their memes and
their missions, and that they want to do something with meaning. If it means something to your
agency, you’re going to find a community out there that it means something to them.
You need to be able to clearly, briefly articulate, what it’s going to take to win. And, Cristin
[Dorgelo] talked about this also, you have to write down what these parameters are so
that contestants can innovate, design, and deliver beyond your expectations. Tell them
what you need so they can win, and give them the flexibility and opportunity to go beyond
that. 2
You want to plan for your implementation outlining your options and getting buy-in for what’s
going to be done with the winning solutions. Too often, we are seeing folks come in with
ideas, and just saying “we just want to collect the ideas” and then it dies a very
sad death. So, let’s move on to something more exciting that can get something actually
implemented and talk to other agencies about what you're doing because they may be able
to implement a similar solution. You also want to share these implementation plans and
the challenge competition with your contestants because this could help stimulate the market.
This could maybe give them an a-ha moment of saying wow, if they’re going to do this
at the agency, I might be able to build a business off of this, and we’re definitely
seeing that more and more with apps challenges and software challenges, they’re coming
up on the rise. And measuring the long and short term impact, listing very specifically
what you’re going to track and measure, and do it at the beginning while you’re
just talking about your challenge so you have your baseline. And then, once you start getting
it more socialized and implemented and then out into the marketplace it will be key finding
that will be very helpful. When you’re looking at this in a challenge competition aspect,
share these metrics’ key intervals, with your challenge competitors. So give them some
feedback in the chat boards about what you’re seeing and what they’re doing, saying, you
know, maybe we’re seeing most of our entrants are coming from the pacific northwest and
we’d like to see more contestants come in from the northeast and challenge them to get
their friends to enter or give them a little bit of feedback on this so they know what’s
happening in their space and what they can do to make it better.
In the measurements and indicators, this is pretty dense, I do have copies of this for
those of you who don’t need to copy this down and pay attention to every detail but,
when we look at the buckets, 4
I kind of sense that we have our technology solutions, awareness, and innovation. And
in the technology solutions, you want to look at the number of basics, look at your number
of contestants, qualified entries, calculate your internal cost savings, calculate your
contestant hours on a project, determine whether or not that solution can be implemented or
if it is something that is getting you over a hurdle so your agency can work on it. Sometimes
people feel that we’ve got to have 100% success and implementable solution in order
for it to be a winner and the real truth behind the scenes is we talk with a lot of agencies
that say, you know we just could not get over a hurdle, and it got us 50% there, 80% there,
and it gave us something to work with. So be realistic about what you are looking for
and what’s going to drive that innovation for you. Look at the number of experience
solvers. People who have been playing in this space and look at the number of new solvers
and are you bringing new people to the table and definitely measure your interactions with
the communities, go to gethub, go to data.gov, go to your chat boards, if you have a website
or internal developer board, like the Department of Labor does and Transportation does, go
there to those sites and count the number of messages and see how many people are interacting
in that space. Those metrics do matter. Those are people who are working with you that you
otherwise wouldn’t be able to work with. The awareness challenges are difficult. I
think in one sense, is that we like everybody to understand that a challenge competition
is not going to solve a communications problem you have in your agency. If you have a great
communications department, a challenge is going to help you amplify a message and get
information out there. And so when you’re looking at awareness, you’re looking at
the number of followers on challenge.gov, you can find people who will follow your challenge,
they get your email updates, they get all of those very specific bits that you share
with the community, press attention, increase in traffic to your agency website. Think about
what it is that you want somebody to do at a second or third level interaction
6 with your agency and make sure that the department
responsible is on board for tracking that both before you launch the challenge, during,
and after and go on after a year or two and see what that impact is. So I imagine that
Department of Energy is going to see more traffic people going to buy, maybe they want
to measure how many more Energy Star purchases will happen as a result of the L Prize because
people became more aware of these energy efficient light bulbs. Ok well, if I’m going to buy
energy efficient light bulbs, should I get other energy efficient products? We should
be looking at that wider view of what is happening in the space, not just your linear silo of
what you’re doing. And definitely, you know, we’re not seeing too many measurements happening
a couple of months out. A year out is the most that I’ve seen so far and it’s accidental.
The EPA told me that after they did their apps for the environment, they got 36 entries,
they were thrilled with that, it was awesome, and I talked to the guy who ran that a couple
of weeks ago and he said, “we’ve got 100 apps now, isn’t that cool?” Like, people
just took off with the data and they socialized it in their app developer communities and
now there are 100 different apps related to that original set of data and the follow up
data that are live and available on the EPA website right now, plus the Android and iPhone
stores. So, this makes a difference, that a year later, something that they weren’t
even putting their hands on is still having an impact with the American citizens.
So specifically in innovation, I’d take everything off that awareness list and the
technology solutions list and add into this, did your solvers ask for more material or
data? Did they try to do something beyond what you ask them to do? Did they bring anything
else to the table? Watch that, calculate it, even if they are not your winners, look at
all of your solutions and see where they brought in new resources or if they were demanding
those from you. Can you use more than one solution or a combination of solutions? This
is how we have to anecdotally measure innovation. 8
It can’t just be a flat number of we had this many viable winners. Its can we actually
make a combination of those? Is there something of value in these that could put us in another
level for the next project that we’re going to do. Did the solution bring more than what
the competition requires? That is our hope, and everyone that we’ve talked with you
guys about, that we’d like to see it just blow your mind with driving innovation or
creativity beyond what we were thinking of when we launched the prize competition. Is
there demand for this solution outside of your agency? Is this something that could
become a public product, another agency could use it? Measure these things too, so, it might
require that you socialize a little bit. If you’ve got an algorithm solution, if you’ve
got an absolution, could you say, this is something that another agency does very similar,
traffic data, could we also coordinate that with, you know, our road closures and something
being fixed from Department of Transportation and put those things together, see where you
can merge. And all subjects, or your competitors beyond the usual subjects, you really want
to calculate this, where are people coming from that are solving your problems and working
on your competitions? If they are people that you work with on your RFPs all the time, you
know, it’s not really a challenge competition at that point, you’ve just advertised to
your normal providers that they can work with you. But go beyond that. Cristin [Dorgelo]
gave some great examples of the surveys and the types of information that you could use
for followup. And the accelerated time to solution, figure this out. We talk about one
example where you’ve got a group of three people working for six months on a project
and the just hit a speed-bump, they’re like we really can’t get passed this so we’re
going to issue a challenge and see if we can get passed this because we really can’t
put more man hours behind it. And let’s just presume here that there are 100 contestants
at 10 hour a week for six weeks, 10
and that’s an average aggregate so you get out of this about 6,000 hours invested in
that solution. That’s equal to 150 weeks of a full time person working in an agency.
If you want to go into the numbers you could look at a contract amount, try to wrap you
heard around what the value of crowdsourcing is, the value of the crowd working on your
problem in a very discreet amount of time, on a very discreet solution, and you can’t
replicate that in house. So you’re using the crowd to bring back answers and that is
a huge, huge benefit, and also if we’re accelerating the solution into the space,
we’re not waiting 2 years to have a technology built, there is an inherent savings in that.
So what we can do for you on challenge.gov. You can look at your admin, you can look and
find out your number of followers, you can track your number of entries, you can check
public voting and see how many people are voting and what they’re interested in, we
can look at the number of discussion topics, and the number of participants, these are
all things that you can do as an administrator and some of them you can do just looking at
it as a an observer. Karen and I would like to pilot something
with you all. We have google analytics for your pages on challenge.gov. We can see where
people are coming from, we can see the frequency that they’re coming to your pages, we can
check how often, what times of days are peek. Anybody who’s interested in this for your
challenge, let us know and we’ll take a few of you and pilot this so we can start
getting some parameters around reporting with that. And, think about just overall, while
you’re getting ready to set up your parameters of maybe a couple pages measuring your internal
change, the people participation and process. Is this, running a challenge effecting you
agency in any way? Are you seeing anything happen differently? Are you building any long
term sustainable communities that will interact with your agency?
12 Is there demand for your program and the data?
What kind of partnerships have you build out of this? You’ve anecdotally that a few agencies,
when they run a challenge, they get a partner who wants to invest in what they’re doing
and there they’ve got a new connection out into the business world that they hadn’t
taken advantage of before. And, is there innovation integrated into your agency? Is there something
that sparks new ideas and gets more people to participate? Talked about repeat competitors.
Recognize your evangelists in the general public and in businesses. This is where a
lot of the communication drops off and we end our challenges and there isn’t a strategy
for holding on to this plan. Make sure that somebody stays in contact with them, because
you’ve got people out there who do care about your mission.
If there is no solution found, look at the upside of that, in the time and cost savings
and, i think this is really important, we talk about that there’s always something
to be learned from a failure, and I don’t think it’s a failure if you spent 6 months
working on something and spent multi millions of dollars and then you issue a challenge.
One example of that, Karen and I discussed, is that the VA is trying to get scheduling,
online scheduling tools, and they spent upwards of $190 million/$160 million or something
like that, and they couldn’t get it right so they just issued a challenge for a $9 to
$10 million prize and people are questioning this, like is that a good use of millions
of dollars? Yes, because they contracted for it and couldn’t get it and now they are
going to see, for a fraction of that cost, is there a way to get this done. And it’s
something that is very important to the future of that agency, its very mission-centric.
And, the last two items very near and dear to our hearts. Take what you’ve learned.
Is this going to help you launch a multi agency initiative, go onto grand challenges, do you
build from here? And the final one, are you ensuring that a prize competition is now part
of your toolkit in your agency? Is this something that’s happening as a result of the work
that you did? 14
These things are all phenomenal.