Open Sourcing Government

Anna Shipman

Recorded at GOTO 2016

Get notified about Anna Shipman

Sign up to a email when Anna Shipman publishes a new video

hi so I work for the UK government and
until 2012 this was the UK government's
website this was where as UK citizen you
would find information out and this
credit calculator I think I'm going to
use the hands like okay so this is a tax
credit calculator so a tax credit is an
incentive from the UK government to
encourage certain behaviors like
investing parenting and it gives you
money off your tax bill so this
calculator allowed you to put in
information about your income and your
behaviors and work out how much how much
feel to have a tax credit you could get
so as a UK citizen if you saw a mistake
in this calculator and you wanted to you
know let them go to correct it firstly
you'd have to get in touch with
directgov so somewhere on this page
would be a contact to contact directgov
you get in touch with them and tell them
about the mistake and that might be
difficult but let's say that was all
fine and you found the right person to
talk to then they would then have to
contact the supplier of this website
because the website was outsourced ok to
an external supplier and to get that
external supplier to investigate web as
to whether the mistake you'd found was
actually required a code change together
to investigate that would cost tens of
thousands of euros and then if they did
determine that code change was required
it would then cost hundreds of thousands
of euros no matter how small the code
change was it was cost hundreds of
thousands of euros and the estimate
would be maybe six to twelve months and
the release is what Big Bang releases
once a year so if you timed it badly it
would probably take about 18 months for
your code flixster going but honestly if
it was a small change it probably
wouldn't get done anyway because that's
a lot of money okay in 2012 we launched
the gullet UK website
and this is another tax calculator this
is a child benefit tax calculator child
benefit is money you get from the
government for each child you have but
if you aren't over a certain amount you
have to pay tax on that benefit and this
calculator allows you to put in your
income and the number of children other
facts and find out how much tax you'd
have to pay and earlier this year in
January a member of the public found an
error in this tax calculator so I just
want to point out that it's quite an
edge case it's in the scenario where you
are claiming child benefit so you have
children under 18 you're also claiming a
pension and you also run over 55
thousand euros a year so it's quite an
edge case it probably didn't affect
anyone or very few people as so but
there was an error and somebody spotted
it so on the 20 source of January this
year well so he found the code on github
for this calculator on the twenty-fourth
of January he made a code fix and raised
the pull request against stat code that
was a Sunday on the Monday the
twenty-fifth of January the team looked
into that code fix talk to the tax team
worked out that was in fact an error it
was merged on the twenty-seventh of
january and deployed on the twenty-ninth
of january so that's quite a big change
from the way that the previous site used
to work and how the UK government
handled its code so I'm going to talk to
you now about how we got to that
situation how we are coding in the open
now what that looks like for us and
where we're going next with it my name
is Anna Shipman I am a technical
architect and I'm also the open-source
lead so how do we get here so this goes
back to 2010 when martha lane fox who
was then the UK government's digital
champion was after the report on Derek
gov which was the first website that I
showed you but her report actually went
further and she looked at the whole way
that the UK government interacted with
UK citizens digitally and she made
several recommendations and one of them
was a formation of a unit in government
to work on this kind of thing
specifically and that was the government
digital service that's who I work for
the government digital service
unit of government within the Cabinet
Office which is in the center of the UK
government and our job is to change the
way the government works to transform
the interaction between the citizen and
launched the gov the UK website which
I've shown you and the next thing we did
was we worked with eight departments
eight different parts of government to
transform various services to make them
better for customers for customers for
users for citizens the aim being to make
digital services so good that people
prefer to use online route rather than
the phone or face-to-face or by letter
so by services so that because that UK
website is how you get information as
services like an interaction with
government maybe how you give government
information or pay money or something
like that so I'll give you a couple of
examples of services that we transform
so the first one I want to talk about is
personalized license plates you can't do
this in the same way in Germany in
Germany you can pay some money and you
can change some of the numbers of
letters on the license plate in the UK
you can change the entire license plate
using numbers and letters to spell out
words or names or something like that so
this is one where she's used letters and
numbers to spell out her name Jenny and
if you sell your car you presumably want
to keep the license lately personalized
you don't want that to go with your car
so you have to apply to dvla the driver
and vehicle licensing authority to get
authorization to keep that license plate
and that service is called you know
retainer license plate before we worked
on this I had to fill in this form it
goes on so much longer than I've shown
you here and send it to dvla and they
would send back a confirmation in the
post so it could take some weeks and you
also have to pay 105 pounds which is
about one hundred and twenty euros and
after we transformed this you can now do
this online and it updates dvla records
in real time so you get instant
authorization of whether you can keep
license plates so you can sell your car
quickly and also because of this work
the cost of the transaction went down so
it now costs 80 pounds which is about 90
euros and that was passed on to the
people paying for the service and this
service receives 91% user satisfaction
which is pretty good for a government
service and the code for this is on
github on d villas organization they
coded in the open from the beginning so
you can see the way that that whole
project developed so personalized
license which is nice is kind of fun but
it's kind of trivial so the next service
I want to talk about is something that's
more serious which is looking a visit to
see somebody in prison so studies have
shown that forty five percent of
prisoners lose contact with their
friends and their families while they're
in prison which is sad for them but it's
also bad for society because studies
have shown that people who are not
visited when they're in prison are
thirty-nine percent more likely to
commit another crime when they're
released so one of the main points of
prisoners rehabilitation so we don't
want people to come out of prison and
commit and more crimes and carry on we
want people to be repaired rehabilitated
so it's in the interests of society that
we help make it easy for people to
receive visits from their friends and
family so the service that we worked on
here this is what this is this is what
this is how it used to work the prisoner
would initiate this process by filling
in this form so they'd have to give the
address and date of birth they'd have to
remember the address and date of birth
of everyone they'd like to receive a
visit from they would give this form to
a prison guard who would then send it to
that person the visitor would then send
a letter back requesting a date the
prison would then send enough about
confirming the date or saying no you
have to pick another date and again this
could go backwards or forwards this
you like a 1 i'll visit to visit you
know a close relative you know you'd
have to do this regularly every time you
wanted to visit them and the new service
allows you sorry you could also do on
the phone
but they the call centers were like
understaffed so quite often you tear of
people being on hold for two hours or
ringing several times before and not
being able to get through so it's very
difficult to get a visit booked so the
new service allows you to request up to
three dates online and then it takes a
few minutes to fill in online and then
you get a confirmation by email within
three days usually much quicker but
within three days so it takes much less
time out of the out of your day to book
the visit and the whole process takes
much less time so it's much easier and
again this service gets eighty-five
percent user satisfaction which is
pretty good and also that 85 cent
includes the people who weren't able to
get the time so they'd asked for so and
then so that's quite a high rate of
satisfaction and again we chose for this
service is online ministry of Justices
github organization and they coded in
the open from the beginning so you can
see exactly how the service works and
how it developed so we worked with
departments on those 25 services we call
them example example our services and
our role now is to support and enable
the prompts to do these transformations
on their own and the way that services
are built in the UK government digital
services is by following this process so
you start with a discovery and that's
where you try and what find out what the
user needs are for the service you then
have an alpha and that's where you build
a couple of prototypes or just a
prototype and get in front of users to
see whether you've got the right idea
about how to meet their needs you then
if you've got the right idea you take
one of those ideas forward into beta and
that could be operational and people
using it and then if that's all well the
service goes live so it's worth
mentioning that you might do you don't
necessarily go like that you might do a
discovery and find out that you've got
the user needs completely wrong and you
have you know you don't you haven't
understood your users or you might have
to do a couple of alphas because you
need to work out the right way to
address those needs but it any service
that gets to live will go through all
these stages
and in order to pass from one stage to
the next you have to pass the service
assessment which is it needs to pass
this the digital service standard which
is what it's 18 criteria for what makes
a good government digital service and
there's links at the end of this talk so
you can look right up if you want to
look at it but the one that's relevant
to this talk hits point eight make all
new source code open so that means that
any digital service that's delivered by
the UK government have to in order to
progress and go live make all of a new
source code open so essentially this is
this is a commitment that we've made so
why why we made this commitment so there
are a few reasons why coding in the open
is a good and important thing to do the
first is the potential for reuse so a
lot of people a lot of departments a lot
of parts of government are addressing
the same problems if you code in the
open then there's a possibility that
somebody can another department or
another team can see the code you've
used and that solves the problem for
them and they can just reuse it rather
than wasting time and money by rewriting
it themselves so it's not necessarily
the case that all the code will get
reused but if you make it open you allow
that possibility if it's closed you cut
thinking this problem is completely
unique there's no way that anyone else
will need to use it but you don't know
that so you know you don't necessarily
know what's going to be useful to others
by coding in the open you allow that
possibility to you know you allow that
to happen you allow people to see that
your code addresses their problem and it
also increases transparency ultimately
we are spending public money and this
coding in the open allows the public to
see how that money is being spent so
another nice thing about coding in the
open is it's really nice to show what
you're working on so the Gulf that UK
website went through the stages I was
talking about alpha beta life
and by the time we got to live the
service was fully operational and all
that remained to do was remove the
banner that said this services in beta
so this pull request is just removing
that banner it's essentially launching
the WK site by not removing the beta
banner so this is in public so my
colleague Devon raised this pull request
and people started commenting about how
happy they were so this is finally
happening so a lot of these people work
for GDS but many of them don't many of
them are people you know in the public
commenting so it's really nice to be
able to share what you're doing probably
so coding in the open is the is a
default position but there can be some
potential downsides so I'll talk about
what some of those are it can be
difficult to get started so as a team it
can be difficult to get started as a new
team that's used to coding in a closed
source way I'll talk about that a bit
later but also as a new member of the
team who code in the open it can be
pretty intimidating if you come from a
background where you know when you have
your code reviewed the only people who
see it are you and the person reviewing
the code to suddenly be in a position
where literally anyone in the world
could see what comments have been made
on your pull request before it merged
that can be quite stressful so the way
we deal with that is I mean firstly
we're aware of that so nobody nobody has
to commit code on their first day if
they're not comfortable doing it we do a
lot of pairing and we have the very
clear guidelines about how to do pull
request code reviews and pour across
reviewing in a constructive way so this
is our pull request guidance against
that's on github so you can go and have
a look at that and we've also built some
kind of automated tool we've built in
automated tooling and various other ways
to make the process easier and more
constructive for everyone and this is a
good blog post that's worth reading
so the other potential downside to
coding in the open is the possibility of
committing something that you shouldn't
for example passwords or credentials so
it's worth mentioning that this could
happen anyway even if your code is
completely closed or so you there's
always an opportunity for you to leak
things to get leaked you know for
security to fail and few to leak secrets
but with coding in the open particularly
well in fact with any kind of security
the important thing is to be able to
recover quickly so this is in a
environment of continuous deployment as
soon as a mistake is observed we can
immediately fix it and correct it and
deploy the code immediately within
definite within minutes so that's that's
I'm going to talk a bit more about
security later but as well as thing else
recover quickly you should also build in
processes that make it harder to make
mistakes so in this case talking about
committing passwords you want to
separate your code from your
configuration you shouldn't really be
having passwords or credentials in your
code base and if you separate the code
and configuration then then the
passwords are not going into the code
and they're not being made public and so
this is a reason why coding in the oven
for the beginning is a good idea because
it forces you to do things properly it
forces you to do things like separate
your code and configuration if your
first commit doesn't you know your first
commit in the open doesn't contain
passwords and then you know you're in
the habit of coding in the open there
are some cases though where the code
should not be committed where we should
keep the code closest and I'll talk
about a few of those this is a blog post
where we talk about that in more detail
so one type of code that you might want
to keep close source is configuration
maybe so this is the gullet UK puppet
repository this is also open on github
puppet is how we converted we
as in production so this is the
configuration for God's at UK and that
is coded in the open it wasn't coated in
the open from the beginning so we had to
go through a process of cleansing it and
making sure that there weren't
credentials and that kind of thing and
making sure that we've cycled all
passwords that might have come out in
that and this is a really good blog post
essentially the configuration now apart
from you know a few things private keys
secrets is all out there and you can go
and have a look at it so don't assume
that configuration also has to be kept
closed source another thing that you
might want to keep close sources and I
think to do with security maybe so this
is this is a blog post so this is a blog
post by the gov the UK verify team cuz
that UK verify is a service for
federating citizens like I'm sorry
confirming citizens identity to
government so that's something where
security is very important because it's
people's personal data and this project
uses sam'l which is security assertion
markup language and they write they
write a lot about how they use sam'l and
some of their code is also open so hang
on let me give you another example this
is GCHQ's github organization GCHQ is
government communications headquarters
that's the uk's intelligence agency and
they have some open source they're not
coding in the open but they are opening
some of their code so the sort of the
way we think about security has had the
context has changed it's no longer the
case that you can just say this needs to
be closed you know security says no and
again this is a blog post that's worth
reading but the main point is that
security is now it's not just about the
technology it's a holistic it's about
the users it's about the culture it's
about the behavior things like their
Franco fanned configuration
it's the focus needs to be on how
quickly you can recover from mistakes it
used to be about how you can make it
harder to make mistakes but mistakes
will happen and now the focus is about
how can you reduce the the meantime to
recovery and it's worth talking a little
bit about how security if your security
depends on nobody knowing how it works
then it's probably not very good
security so good example is with say a
padlock everybody knows how a padlock
works still secure what you need to keep
secret is the key so again you might
think that you can't code in the open
with things through a security but
actually a lot of that can also be done
in the open and the third case where we
say that you might want to keep the code
closed source is with policy that's not
yet been announced so if you're you work
in commercial my great organizations and
analogy would be if a feature that the
marketing is not yet been done you don't
want information about that to leak
before you're ready but in that case we
code as if you're going to release it
later once the policy is announced you
know make it public so that means no
credentials or secrets good commit
messages good documentation but you do
that anyway right so I want to take a
moment to say we're not perfect so I've
given you some examples of some great
stuff that we've done the two services
that I chose to tell you about as
examples of coding in the open part of
the reason I chose those services is
because they did it right they coded in
the open from the beginning I can point
you at their code on github and you can
go and see all that but there are some
other services that are also really good
that struggled with coding in the open
because change can be very difficult
coming from a background where
everything has to be closer so it can be
very difficult to have that conversation
and many of these services started
before we had those high-level
accommodations so for example register
to vote it's a service which I love
talking about because as part of
transforming the service to make it
digital we had legislation
changed to allow people to register
individually rather than as a household
but they didn't code in the open and
they've started open sourcing their code
but this so this is some of it but if
you go and you look at it on github it's
one commit and the committee is open
sourcing content so that's because they
didn't they didn't build it they didn't
code in the open and they didn't build
it on github and again it's not like
they didn't do anything wrong the
climate for them was difficult at that
time and change is very difficult that
team is working on how to open source
the rest of it so I'm telling you that
because I don't want to stand here and
say we're amazing and we done and
everything right it's not all perfect
and also because you know to give you a
courage if you want to do this in your
organization you can too it doesn't have
to be perfect from the beginning if we
can make changes like this in an
organization as old as the UK civil
service then you can probably do that in
some of your organization's if you want
to so I talked about the advantage of
coding in the open being reuse there's
quite a lot of people who have used our
code but I'm just going to give you a
visual example of some people who've
used our code so this is WK again this
is gov dollar nzd New Zealand's
government website this is gov the il
Israel's government website and this is
Lexington Kentucky Gulf the website for
Lexington's local government and they've
all used our WK front-end code so it's
really nice to see that they would save
them time and money ok so what I've
talked about it till now is coding in
the open but we make a distinction
between coding in the open and what i
would call through open source because
there are various things about our code
while open that doesn't make it that
means that it's not open source so
firstly we don't promise that we're
going to support it we don't promise it
will work for you in your environment
but it will work for you without all the
rest of our applications it's just open
and we've got committed to maintain it
if for example we find that there's a
better way to do calculators or your
calculators or whatever we'll just
retire our calculator code and start
working on the new code which is not
very good open-source behavior so we do
have some open source projects a DDF
most of those are in the infrastructure
space and this is you can there are
links at the end of the talk you can see
the infrastructure open source we have
it's going to briefly talk about one
project which is the cloud tools this is
a suite of rubygems for provisioning
virtual machines in VMware vCloud
director environment at that time our
gov UK was hosted in a VMware vCloud
director environment and these tools
were originally coated in the open by
led a team to make them truly open
source so I've written that up in a
series of blog posts is one of them and
basically to make them such that they
were maintained and supported and other
people could use them easily and there
were other parts of government who are
also using the same kind of hosting
environment and so they started using
the tool so it was great that we were
saving money across government and we
also found that there were some people
in industry using metals so this was
really exciting we saw this tweet
somebody from a publishing company who
was using free cloud tools and these
these likes are basically all my
colleagues because we were really happy
to see that somebody was using it so so
we have done some open source not just
coated in the open but there are
challenges to maintaining open source in
a work environment additional challenges
for the ones I mentioned about coding in
the open I'm just going to talk about
two of them and how we address them the
first is getting time to work on it so
people once it's open source people will
raise issues people raised pull requests
and that doesn't always necessarily fit
into the schedule of work that you have
for your team the way we dealt with that
on the infrastructure team was we had
open sores Thursdays so on Thursday you
could work on if they were outstanding
issues or pull requests on vcloud tools
or other
of our open source which is mostly
puppet modules and you could work on
those or you could work on maybe patches
for other open source software that we
used and this worked for our team
because a lot of our work was around
those open source tools wouldn't
necessarily work for a team for a
different team that's just how we
address them and then the second problem
is contributors wanting to take your
project in a new direction either one
that you hadn't anticipated or maybe one
perennial problem in open source so I
mean every open source project the
successful has this problem like a good
example is GCC the c compiler which was
a fork running on side for five years so
this is a problem and the way we
addressed it was basically because we
knew pretty much everyone who was using
it we just talked to people again that
wouldn't necessarily work if you don't
know who's using your open source
project but that's that's what we did
okay so that's where we are at the
moment we have almost everything coated
in the open and we have some open source
so what's next for gdf open code so i
said i mentioned on the open source lead
I've got three priorities they are first
of all I'm going to try and work out
what of our coded of our projects that
coated in the open would be most useful
for other people most useful to put the
work in to make them actually open
source or reusable so I'm not just dds
like work in other departments as well
what of their code will be useful to
share I'm also going to be working on
helping teens code in the open so I
mentioned you know the register to vote
team are having difficulties i'll be
working with teams to work out like how
i can break down those how we can break
down those barriers that make it hard to
code in the open and the third thing i'm
going to be focusing on but haven't
really talked about much but it's very
important is i'm going to look at ways
that we can increase our contributions
to open source software because probably
like a lot of you we depend very heavily
on open source projects maybe don't
have enough support we do contribute
patches back but I'm going to be looking
at how we can how we can do that in a
more concerted useful way so how can you
get involved if you want to few
suggestions you could try and get if
you're not already get your
organization's coding in the open I
think even commercial organizations will
have code that's not central to the
mission that can be open can be useful
and I hope that I've given you some
suggestions of how to address some of
the problems that will come up there's
also some resources at the end that
might be helpful as well you can let me
know if there's any of our code GDSS
code or other government departments
that you would like to use or have tried
to use that would be great because if
there's something you'd like to use and
I'd be really interested in working out
how we can make that easier or you can
come and work for you this thanks
if we don't have time for questions okay
so the first question that we got is is
the simple surface shrinking does UK
golf do more with less I don't actually
know the answer if it is but yeah we're
always trying to do more with less I
don't know I don't know if it is
actually shrinking sorry are we using
extreme programming practices and do we
teach them to do don't know okay so
we're not using extreme programming we
do use agile methodology which I know is
something completely different but it's
in the same area so we do with in terms
of extreme programming we do things like
we do do pair programming a lot i'm not
sure why the other ones we use so we
definitely definitely don't follow
strict XP but we do work in an agile
multidisciplinary way so teams are built
made up of mixed discipline so
developers user researchers designers
technical writers project manager
Fremantle's we have multidisciplinary
teams and we do follow agile mostly
roughly scrum again not strictly scrum
but it's pretty much roughly and do we
teach it I
so I think we do go and work with
departments to help with their agile
working which is not quite the same so
sort of so the bestest question was that
there are forces against UDS are we
going back to young old model no I don't
think so we still have the same mission
to transform the relationship between
city and the state we're still trying to
improve things for our users I don't
think we're going back to how it was are
they civil servants or contractors how
do we do support okay so we're a mixture
of civil servants and contractors I
don't know what statistics are and it
varies between projects that we do have
a lot of both and then support how do we
do support yes I was just wondering in
terms of being able to support a project
over a long time you don't want to
maintain the team of ten people but you
might suddenly want to have a fix that
requires ten people to do because it's a
security flaw or something so how do you
support that throughout everything so
that really varies between teams so what
we do on the WK team is we have a team's
build project teams build applications
and a team will own several applications
for the team will be ongoing and then we
have a support rotor where let me think
yeah so what I mean I sort of segue
didn't took them out of our support
there so that's not actually aggressive
basically we do have ongoing like the
idea of a project being finished and the
team leaving is not quite the same as
its ongoing it might be not the team's
main focus but the team will still own
that project basically so if you know
maybe nobody's working on it and then it
suddenly requires a fixer that the whole
team will stop the other things they're
working on and work on it sorry I I am
very interested in our 2nd line support
model finally review board you all with
in a source how we thought about a
middle way open sourcing within the
organization so there is this idea of
inner sourcing where you don't
necessarily share it publicly but you
share it within the organization my view
is that it's not worth the effort
because it's the way it's often easier
to find out from the internet something
about some open source than to find out
what the team down the hall is working
on whatever your organization is and I
think it's if you have the backing which
we do to push for things being properly
open source then I think it's worth it's
worth doing that it's not I don't think
it's worth putting the effort into the
halfway house the sort of the advantage
of the halfway house is that you can
share things that you maybe are not
appropriate to make public but the cost
is the difficulty of communicating what
people are working on finding out what
information and I just don't think that
the benefit like that I'd see the cost
is too great that's my dear so how do
you determine what services need to be
open source at all and how do you
determine what service needs to be
rewritten in a new approach right might
be a real open-ended question okay so I
mean so the the aim I'll take the second
part first how do we determine which
need to be rewritten the aim is to make
services digital by default and the aim
is to make services that are the digital
root is so good that people prefer it so
any service where that's not the case is
the service that we would look at
helping make that the case so it's quite
a lot I mean you know the moment is the
majority of of what we've got and then
how do we determine whether it needs to
be open source everything does the
according to the service standard coding
in the open by default I mean there are
some things where you know maybe it's
not maybe they can present a good reason
for why
it can't be but it's definitely the onus
is on it needs to be open unless you can
really convince why it shouldn't be okay
Oh another question yeah thank you first
of all living in Germany and bit jealous
because we're not that far actually here
and I think in part that's due to the
federal nature of the country and have
you had so you're working for the
central government have you had areas
where you wanted to provide services
that are actually within the authority
of a region or of I don't know if wales
these or do you just leave those kind of
topics out so GDS we look at central
government because we also have local
governments and so local government is
currently outside dds agreement so
occasionally there's you know we work
with them but it's not it's not in our
ok so it basically does central
government is going taking the first
step and then maybe yeah although a lot
of the local I mean local governments
are independent but a lot of them are
also doing really great work in this
area but it's dependent on each one ok
thank you very much