TalkFrom

ING's Journey to Agile

Henk Kolk

Recorded at GOTO 2015

so 30 minutes to talk about a and GCL
journey I was reminded this morning by
Barry that not many financial
institutions really aspire to be this
Lean Enterprise I think we really do and
and we're well underway so I hope to
give you some insights on what we've
done how we got there and what our new
big challenges are so my name is Hank
Dutch indeed so you hear me struggle
with my English now and then a little
bit
I'm the chief architect which means I am
responsible for how the bank works it
also means that one every once every
eight weeks I am on call like the rest
of the management team so we made all
our teams responsible for not only
developing but also running their
applications and we did the same for the
management so we get immediate feedback
on what's going wrong so let me take you
back a little bit in time there was a
time when ing was like many other banks
we believed in waterfall and prince2 and
some process folks had even integrated
prince2 with rational unified process
and they created a whole bunch of
templates for it and basically it meant
that we had to do something like over 70
mandatory documents before we could even
start coding that was five years ago we
also had an app and do you remember this
time it's five or six years ago the
iPhone was just out I think and and we
had an app and our customers also had an
opinion about our app and they expressed
it very well in our apps in the App
Store with given as one star
and they even complained in the AppStore
to Apple asking them to make it possible
average so that was the situation and in
2010 2009 we also got a new management
team in and we were very lucky to get in
Rome from Caminada who became this new
CIO and he was had been the head of a
head of Internet as it was called by way
back then before so he was really
frustrated by the lack of speed that we
got from our IT department so that was
good
and when he started he said this is a
little bit the believes that everybody
had IT is a commodity do you remember
that yeah IT is a commodity everybody
can do IT we had internal customers
which was also fun to watch when I
joined ing at two years later I had this
guy in my team who was called an account
manager because we had internal
customers and he had to get the work in
for my unit the big belief that IT is in
cost Center and that we should try and
save money every year on it and this
whole thing about quality so ing was
great in CMMI and had big programs
running try to get better by producing
lots of paper so from this point on it
got interesting because what these guys
did they went to a tech conference first
and they really started thinking about
hey what we are lacking is a deep
understanding of technology
so this CIO he started an Java community
which was hit way back then and not only
did he started he learned to program
himself and what I really admire in this
guy is that he really rent learned
program he was an
economic economists but also the fact he
was is totally blind you know so he
learned programming while be having this
handicap and this was a powerful message
that he sent into the organization with
this it said hey code is king guys we
want you to really master programming
again so then they went on and they
started the first scrum teams and this
basically started when my now good
friend Danny ran out and good friend Ava
line from they were went to Rome from
came around and told them hey we are at
the point of quitting we don't want to
work here anymore unless you decide to
have let us do a Chow and so Daly and
Ava line started this movement in the
channels Department where they started
creating mobile apps leveraging agile
basically and Rome protected them he
said okay from now on you don't have to
adhere to our processes anymore I'll
protect you and my other good friend
Amira Rooney he just bought them the
machines that they really wanted instead
of forcing them to work with machines
that were supplied to them he went out
to the shop with his own credit card and
got the machines in and got things
working as you can imagine this created
a little bit of havoc and stress here
and there but then Rome really saw that
these guys started to work fantastically
so after that we said agile is now
becoming our new way of working and we
got in a lot of scrum coaches to help us
understand as hell better and it became
like the mandatory thing for the whole
organization to follow and the
interesting piece was that after these
first successes with mobile app
development were created everybody
wanted to do it so interestingly enough
I even saw the process consultants that
adopt agile to create a more better
description of agile doing a job
we saw architects trying to do agile we
saw businesspeople trying to do agile
and scrum so this this really started to
spark here this this little wildfire was
going through the organization this is
successful this is how we want to do it
and our senior management really
supported this move to agile and it was
enormously important to us so then we
got into the the discussions on who gets
to run this and we got to let me just
yeah so who gets to run this who gets to
run the bank in production and we came
with this idea we are going to create
DevOps teams now later on I got into a
little fight with Jess Humble who said
that's a really bad idea but for us it
works basically what we did is we put
operations people and it developers
together in one team which meant that
from now on we have 180 teams and these
hundred 180 teams were responsible not
only for developing the software but
also for running it into production
meaning everybody on call also the
developers and a lot of the hassle and
fights should be gone however this
little cultural change of going from a
developer led organization to DevOps
teams also took some time to really
integrate the people we saw examples of
people trying to move closets between
the ops and the Deaf side of the same
team or trying to really do their own
thing so this took a lot of management
attention also to really integrate the
teams and I am proud to say that today
we are on a situation where these teams
really work as one and that's not the
end of it because what we saw then is
that we had a lot of obstacles really
so we started building our private cloud
we had a lot of discussions whether our
customer data was safe in the public
cloud maybe familiar to some of you but
being a Dutch bank and and privacy for
us is very important we were a bit
careful about that
and this summer our commercial
colleagues fully joined us which is
exceptional so what we did is we also
took down the management layers of to
Kuwait all the management layers from
the commercial side and now we have
around 400 agile teams who report almost
directly into 13 tribe leads and who
have gotten a lot of responsibility to
build and run and improve our systems so
this is where we are today and I can
tell you some of this still scares me a
little bit but I'm also really proud
that we're moving at this speed so what
or what were some of our Vic convictions
during this journey you heard a lot
about continuous delivery well
continuous delivery is for us is
mandatory for all the teams and we're
now basically pulling all the teams into
one continuous delivery pipeline for all
we used to basically leave it to the
teams they just had to prove to us that
day we're doing continuous delivery but
we also saw that they had to spend a lot
of time in every team to build do their
own build servers get their own
automated deployments so now we created
just one another conviction was partners
are welcome and outsourcing is not and
what you hear here is and convinced
conviction that we should really master
the technology ourselves so we do not do
outsourcing a lot sometimes but for us
understanding how to build mobile apps
how to build api's how to build dr.
kanta
that's how to manage them having that
understanding in the bank for us is
critical a third one I will be talking
more about this is nothing beats
engineering talent and our focus is not
process adherence anymore but building
great software so if you walk around our
buildings you will see this big
monitoring screens for each team and one
of the feedback loops that this constant
is the quality of the code another one
production so building great software is
for us critical and now and again I get
the chance to talk to fantastic
engineers that work in different places
and where they try to start to get
DevOps started and where they try to get
continuous delivery embedded and agile
and scrum and they tell me how hard it
is and I think we were very lucky
because we had some great advocates of
agile scrum and continuous delivery in
the company so one of the convictions
that you need to have in place is that
IT is not working for the business but
is not a cost Center but IT is a value
creator and these sounds like big words
when I say them but it's very very very
important that that you focus that you
look at it like this outsourcing for the
lowest price I don't want to bash the
concept of outsourcing there's probably
some great example somewhere what will
but we've seen that we can easily
out-compete distributed teams by having
excellent engineers and basically the
key question is having excellent
engineers another thing that really
shifted is this whole idea of by before
built you know that architects like
myself are saying that these kind of
things
reuse before buy before build
whereas we've learned now that actually
building something even a first
increment or a prototype is a great way
to better understand the software that
we're actually trying to find and the
requirements that we need to understand
more the last one is int used to have
like this this big project like
organization where as they would drive
to change and the line managers like
myself in the past would just be
responsible for running the bank so this
differentiation between changing and
running the bank and also this is one of
the principles that Ron put forward hey
line drives the chains so managers IT
managers they are responsible for the
health of the teams and the health of
the software and nobody else
so is this interesting so far yeah so
which brings us to how do you sell this
right what I wanted to do here is
hopefully inspire some of you who are in
companies where things not moving as
fast or maybe you were consulting to
such companies I'm using this term a lot
you heard it in the whole conference
software is eating the world but
basically the you have to bring across a
message of that that we need to act and
we need to act now and this is passed a
part of my work basically telling all my
colleagues that soon we will be eaten by
one of Adrian
unicorns and that we have to speed up
yeah so speed is marketshare you heard
it a lot but I'm also using this one as
a sales argument internally because I
need money for my platforms we need
money for our platforms and people
should really understand that speed
comes from having these platforms ready
enable right that you don't have to wait
for three months to get a virtual
machine that is securely configured etc
that's that's no longer an option this
needs to be a push-button thing so the
last one the quality of an IT
organization is the quality of its
engineers it's a quote by Rome I admire
this guy a lot and why is this now so
crucial the funny thing is that in the
last six months or so I've come to a
different understanding we have the
agile in place platforms are more or
less in place or on maturing but I still
see enormous differences between the
productivity of engineers staggering
differences to give you an example there
was a little component that was needed
and a friend of mine
Ron cassock built it in two weeks
whereas the team that was supposed to
build it claimed six Sprint's for it of
two weeks and the team was eight people
big right so if you think about it
that's a Productivity difference of 48
and I've seen a lot more evidence of how
we tend to overestimate or spend too
much money on things often or sometimes
with a factor of 10 so maybe the next
problem for us to solve is to understand
what this difference between engineering
skills really is
so the way I think of it really is that
we're that I live in two bubbles of
course the financial crisis I work for a
bank
it has hit his heart we learned an awful
lot and we also learned what a concept
of a bubble is and I want to give to you
the idea that we have a similar bubble
in when it comes to engineering skills
this is a piece that is not well
understood and we kind of ripped it
apart you know the the software
engineering profession in the days of
Rob rational unified process when we
said okay we have a requirements
specifier and of course the guys who
created it intended it only as a
discipline some piece of the skillset
that you had to master but we turned it
into roles in our industry we turned
business analysis into a row we turned
design into a role and we told turn to
roll into a person who had a function
and then we turned developing into a
role basically he should put the design
into code and then we put tester into a
role and I really think and I'm more and
more convinced that these are just
skills of a good engineer and it's it's
a very nasty way of making projects very
big because you need to have all of
these handovers between people whereas
perhaps we have a responsibility even to
focus on making sure that our people
learn all these skills along the way so
we can be faster this morning we had to
talk about healthcare.gov I think it was
in the United States and I read this
fantastic article about it it's called
coding for ankle Sam you should look it
up in this morning we heard that the
rows even more money spend on on this
program that I ever anticipated but
this article says hey we had an upfront
investment of 200 million dollar and it
didn't work and we brought in five
Silicon Valley engineers at the end and
they got it to work and there's this big
difference between this 200 million and
these five Silicon Valley style
engineers and this article says and I
hope it's true or that I'm not lying but
at least it's there on the web you can
find it said we forecast at 70 million
per year for maintenance whereas the
Silicon Valley guy said hey give us 4
million and we're satisfied and these
kind of huge productivity differences
may also be a key that has to be added
to this whole lean agile scrum thinking
we need better engineers and we have a
responsibility therefore to help them
mature as better engineers does this
still make sense so this was actually
the title of a talk that I'm going to
give one day it's about the rebirth of
the master builder right and the master
builder stands for the role that I got
being an architect and an architect I
think in the past in the early in a
medieval medieval world meant that you
had grown as an artisan to the SIRT to
the level that you could build a
cathedral or a house or whatever right
but you learned by doing it you learned
basically by practicing all the tricks
of the trade until you masters them and
I think in a way we need to go there
again because I'm still having trouble
and perhaps I'm alone in this that this
whole new idea of an architect is this
gentleman engineer who creates beautiful
designs within pictures
I'm still having trouble understanding
will this ever really work for software
engineering will this ever work if you
don't have the skills in the mastery if
you haven't gone through all the work
yourself to master these skills and with
the speed that technologies go and I
think it's very hard to create designs
that will stay out there right the speed
hundreds of teams basically changing the
architecture a little bit every day it's
very hard for me to come up with the
picture of how the bank is working today
it has gone become impossible so what we
did last Monday even as we said we need
a better model for this and I met a guy
once called Andy hunt I believe it was
somewhere in mom way I don't know
anymore but he wrote this book about
refactoring your wetware he is one of
those guys that also signed Andy Hunt
the agile manifesto and and this book
has a certain Flair and style and you
must like it but the first piece was
really interesting for me because that's
where he explained how people really
learn and this is the Dreyfus model of
skill acquisition and our experiment
today is that we are using this model
for our peoples strategy you see during
this agile journey we did a lot of
things but just bringing in a lot of
coaches and then saying okay now we're
agile that's not it
we came across hurdle after hurdle that
strategy so we said hey let's really try
and to understand as a company how
people learn and this Dreyfus model
basically says you need to understand
that when somebody is new at a task that
he needs recipes
right if you're a cook if you're a
master cook you know what the pinch of
salt means but the novice wants to weigh
it because he doesn't know what it is
you need recipes you need to be very
strict advanced beginners can already
find their way in
in tutorials and you can see that
they're at that level because they zoom
in exactly to the place where they need
to be people who are competent we
defined as people who know how to
basically solve a new problem all by
themselves which is including the
requirements getting them right
the domain model getting that right the
test cases getting that right so that's
unit testing functional testing load
testing resilience testing getting that
right doing the coding themselves in at
least one technology stack and then we
think you're competent so the next level
for us is hey when people are really
have that kind of mastery and they
become already really productive then we
want to give them different challenges
in different languages basically grow
their skill until they're these experts
I want to ask you to basically look up
this this book of Andy's because perhaps
there's a lot of ideas in there on how
people learn and perhaps we can make the
industry a lot better by understanding
how people learn at different phases for
example if somebody's proficient you
should not send them to a training class
they will get bored they will annoy the
trainer yeah you don't want them in
there you should send them to
conferences and if you want to challenge
them make them become a speaker at a
then about those platforms right if you
have hundreds of teams then there's
bound to be backlog dependencies between
them and you have to go look for how can
I create these platform levels where
they can just use stuff that is already
there now so that means that we put a
lot of effort into shared engineering
platforms basically coupling everything
together and for these platforms there
is less freedom as my colleague before
me said and ing de we give a lot of
freedom that's true but we don't want
you to break the pipeline and we want
require a MongoDB database in a docker
container well go ahead but make sure
that the pipeline is also it's added to
the principles of the pipeline so what
you see here is a visualization of our
pipeline another thing that we had to
create because otherwise for a large
company it becomes very difficult to
handle is a data platform so we
distinguish really between the big data
and if I may say so the slow data and
the real-time analytics and we have a
couple of guys really creating our
real-time analytics platform into a open
source project and then there's of
course our emerging micro services
platform where we use a lot of the open
source technologies that are out there
so this is a little bit of where we are
now of course we want to have the people
in our company really own the code but
also the design we want them to think
like designers we want two of them to be
really proud of how their software is
designed so here's a quote by my good
friend Ron Kasich everybody is a
designer and we're trying to get this
culture of everybody as a designer going
my good friend Flavia designed like you
give them yeah so and she really means
it's fantastic to have those people in
the house so and you can see it you can
see it with the platform teams right
we call them squads these days we some
senior managers visited Spotify and I
think that was not really a bad idea
because we're at a point in time where
we need to create our own words for the
things that we're doing and these
platform teams they came up with their
own machine and their mission is to make
other teams awesome really awesome and
what you see is that what they created
is not intended as a black box but
there's a real community so everybody
can contribute to the platform you can
do a pull request there's lots of time
to discuss this with the folks who
create the platforms and the whole idea
so thank you very much everybody