Get notified about Adrian Mouat

Sign up to a email when Adrian Mouat publishes a new video

okay so my talk is going to be on docker
security so before I start who here has
used docker put your hand up you stalker
okay that's a good indication thanks I'm
glad to see that even if you've not used
Dhaka very much you should get something
out of this there is some specific and
sort of technical stuff later on but
it's also some general stuff that should
be of interest to most of you and so
about me I work for a company called
container solutions there were Dutch
consultancy around containers and also
me sauce things like that and the main
thing I'm doing at the minute it's just
finishing up a book called using docker
for a Riley so let's talk I thought I'd
put a little bit of fun with it so we're
going to start it with a five-round
security box and match although was
thinking the way here it should be now
that rugby match because i think the
rugby world cup kicks off tonight but
it's a vivarium security box and match
between containers and VMS i'm the
second effort talk ask me more specific
ok so containers versus VMs so what we
fightin for when we're thinking about
security it's not actually should give
your Leonard WBA world title belt what
we're fighting for it's nothing so you
get security right nothing happens what
doesn't happen as your site doesn't go
down because somebody's mind to do su
what doesn't happen is your front page
doesn't look like this because some
script kiddies got a degree styled it
for you well it doesn't happen is you
don't start selling farmer by I grow on
the side or at least I assume Tim town
council didn't mean to self buy I grew
on the side what really doesn't happen
if you don't lose all your customers
credit card details Adobe just about got
away with it but a smaller companies you
know could be bankrupt from this and
what really really doesn't happen is you
don't lose since the back training from
nation on military and intelligence
personnel this I think this is one of
the worst acts that we know about
because this directly people's lives at
risk these are people that could be
targeted by a foreign governments and
criminal elements so that was quite
horrific in my eyes then so we were the
contenders in our fight well you've got
to say the VN the vm is basically the
reigning heavyweight champion if you
like VMS are used in banks they used in
government everybody uses VMs be in vain
for a while we trust them compared to
this containers are the unknown upstart
however i think they bring some new
features of executing features that does
actually make for an interesting fight
when you put them head-to-head okay so
going wine isolation guarantees so here
i'm just thinking about how isolated
other processes inside a vm or a
container cancel me break out with a vm
our container and access us of VMS or
containers on the host or the host
itself and here the big thing is the vm
surface of hypervisor layer and the
hypervisor to some extent acts as a
buffer or a filter i can Caillou a lot
of attacks and when you compare this two
containers continuous share the colonel
right so the each container is when the
same care of the host so if a container
minds just cause say for example a
kernel panic that would take down the
running containers similarly if a
container manages to grab all the memory
your star about the other containers on
the host so I think you gotta give Rank
1 isolation to the vm
so vain to attack surface this is
probably the most controversial of my
the Rings in this fight but I'm going to
argue that is a greater attack surface
in a vm idea I'm just comparing directly
the vm to Lance colonel there's a lot
more going on in vm it's a very large
piece of code there's a lot of things
like virtualization and emulations i
just don't happen like by default on
linux and so part of that is doing
things like emulating devices and your
virtualization and there was the venom
vulnerability it's one of the new
vulnerabilities that gets us on web page
which seems to be a brand new thing but
the venom one ability was quite
interesting basically there was a bug in
some floppy disk by regulation code that
nobody really used because he uses
floppy disks anymore ok but that was an
audition participation for ya but anyway
they finally bugged that I'm the that
actually caused a break out the vm could
attacker could break out the vm and
access the horse tour of the VMS there's
also another side to this and that you
can produce containers so this is a
separate issue before it's looking at
that your tax service of the colonel
compared to the vm but the other thing
you can do with containers is build very
minimal containers so you can build a
container that just contains like a
single static binary it doesn't have the
whole operating system wrapped around it
so by doing that you can cut out a lot
of stuff that could potentially be used
to attack you yeah so the minimal
contarme on the container static binary
and by doing that you really are cut and
during the attack surface so for that
reason I'm going to give this round to
the container at 13 controls so here I'm
really thinking about what knobs can you
twiddle or buttons can you push to
restrict the privileges and resources
so if you think about limiting access to
the actual hardware resources the memory
CPU in the disk they're both pretty good
right you can do pretty much the same
thing CP with VMs and with containers
but I'm going to argue that you get more
controls for containers so you can do
things that set file systems read-only
I'm pretty sure you can do it V aims as
well I'm not an expert in VMs but I'm
sure you can however with the continuous
that's quite simple you can also do
things like put calls in it so certain
files can be read right whilst the risk
the file system is read-only and we'll
see that Leah and there's also Colonel
linux kernel capabilities so linux
kernel defines things about 40 different
capabilities and these are just sets of
calls that you can make so what you can
do is you can reduce the capabilities of
a given container and say this kitten
thing can't create network sockets or
can't create set you Eddie binaries
things like that there's also set comp
comment as not available yet but it's
coming to docker containers and that
although even more fine-grained access
to exactly which kernel calls container
can make so I'm going to argue that the
container also wins this friend so vain
for auditing so here I'm thinking if
you're going to like a large vm system
or large container system every now and
again you need to like look at the
system and make sure the images you are
running are up to date they're not using
all vulnerable versions of software for
example it's the first thing that you
might notice is that a typical system
you're going to be running considerably
more containers and you would be varying
the MS so our system it runs in dozens
of BMS may actually run equivalent
system they use stations of containers
so one way of thinking maybe that was
more to audit but that doesn't tell the
whole story the big thing is the MS are
longer lived and you compare that to
containers for sure he didn't write our
ephemeral so VMs overtime diverge
the base image and we have a whole
branch of software called configuration
management that sprung up around this an
attempt to try and make me ms consistent
over time and so what this effectively
means if you've nvm some production
you're never quite sure what state
they're like and two VMS are ostensibly
doing the same thing are quite likely to
have different versions of light burst
and so on so you need when you audit you
have to audit each one of them
separately if you compare it to
containers you're doing it properly what
you do is your place I container you
don't patch it so you take the inner
container and put up the new one that's
built from an image that has been the
highest update and what this means is
you can actually hold it your images
offline I supposed to your online
containers and then you can verify that
the containers haven't diverged from the
image and there's tools like and so the
point of that is you can verify one
image and you may have actually audited
hundreds of containers effectively I can
use tools like dr. death to one make
sure that your containers haven't
diverged from the image they were
created from and also it's quite nice
hotel them if you've been hurt because a
if you say like the script kiddie attack
where they change the front page if that
was one in the container the event
occurred if you immediately see like the
CSS files at HTML files that being
changed or added similarly if you run
the pharma spam thing you'd probably
find a file that was actually contained
a bunch of PHP code I need you know you
find it very easily using the doctor
diff command so I'm going to give that
round also to the container but the
final round going 5 is track record and
here i'm going to say VMS are quite a
strong advantage right they've got a
proven record they've been around for
years later we trust some banks trust
and governments trust them and you
compare this container so you simply
don't have the same track record okay
you can argue that containers been
around for a while but certainly not in
the modern sense of docker containers
and so on I'm just going to take time
before people come to trust them
guard security so I'm going to give that
been quite heavily to the vm now I'm
sure some of you good at maths and valid
this up but I was wondering who thinks
the vm should win this fight p time that
you think the VMS you doing pretty
hundred fifty container should win no
fee okay cool I to be honest I wish
styling call that drawer which I guess
most of you could do mass probably
figured that but but that's my next
slide but i will say beware the rematch
because there's a lot of interesting
work being done so one thing is there's
a lot of work going on to speed up vm so
you're actually going to see VMS i think
being used in a more container style
workflow so it's the projects like clear
container per min tell that such using
VMS in a sort of container style stick
and its container style and also there's
a lot of work going on to secure
container so this thing's like sitcom
from docker but there's also a lot of
work going on the linux kernel mentiras
to make sure see groups and namespaces
and so on our secure and during the
shoot so in a few years time it will be
interesting to see where we are but as
the adrian said that the minute where
you're probably going to go is going to
use both containers and VMS and that way
you've got two layers of security then
so you're the best way to do it is to
use vm to segregate groups of containers
so for example you're going to
multi-tenant application you place each
tenant or user a customer they have
their own VMS and what should we run the
containers and that's what Amazon and
Google do give us a bit of an interest
in one though because you use cout
benetti's your containers are running
VMS but those VMS are in turn running
board which is containers so you got
containers in VMs and containers the
other way you can segregate things is if
you have a received a large web app and
you've got a front-end running PHP or no
GS you put those containers and one vm
and then another vm you do like the
containers when and say
credit card processing so an attacker
gets breaks into an orgy s continue you
still gotta break out of that VM and
into another one to get anywhere near
the credit card information ok so that's
it for the first part of the talk the
next part are just going to be general
security tips for garden containers
actually if there's one thing you take
away from this talk I want to be this
slide so security paradigms the first
one is defense in depth don't rely on
just one layer security so kristen has
talked earlier with some talking
mentioned firewalls and it used to be we
rely quite heavily in a firewall to cut
a lot of attacks but nowadays it's are
much less useful because everything
comes through HTTP so there only is one
poor or too poor to include a GPS so
don't just line the firewall don't just
lie in containers use VMs as well don't
just rely on that keep everything
encrypted that's sensitive and what you
always think is if an attacker gets past
this layer security what does he have
access to and you see always trying to
put an extra hurdle in the way of
getting to your sensitive information
the other paradigm is least privilege so
this is just the idea that a container
should only have access to data and
resources that are essential to its
function this is like quite an old idea
was used to be applied to even like
functions and classes and so on Jerry
Saltz sir I think first articulated it
I'm doing that if the if an attacker
breaks in to the goes back the idea of
breaking into the front end should be
asleep be able to get access to credit
card details things like that you can
liken it to the secure the military idea
of need to know where people are only
two old the minimum they need to know
what to complete the mission and that we
as a culture they can't spell secrets
there's also very good talking this that
was given a doc icon called least
privileged microservices by Nathan the
collie and Google monica from docker so
check that you want know more about
least privilege
okay so more specific tips the one that
most commonly people get wrong if you
look on like a docker hub but you look
at the images the biggest mistake people
make is to do it users and so the thing
to be aware of is that users aren't
names paste milling it's Colonel so your
route inside a container that's actually
the same users routing the horse what
that means is you break out of a
container that's running as a root
process that Jews processes running his
route you'll be root on the horse which
is clearly a very bad thing now there is
work going on a minute and docker to
automatically map the root user and a
container it's like a high numbered
using the host but that's not there yet
and also I suspect will be issues
regarding a file permissions and so on
so in the meantime it's very important
I'm to be honest anyway it's very
important to set a user to earn your
images with your story in your dacha
files you include like two lines are out
what that does is it creates a user and
changes to that user for all the phone
lines in the dockerfile and when they
contain your startup if you look at the
official images they play often include
the first line but not a second line and
or the an entry point script that gets
running the container starts they want
to do something like file permissions
which requires root privileges so what
they do is he'd set those permissions
and then the user to likes to do or go
through to change to the appropriate
then set container file system read-only
this is quite a handy one to cut so a
lot of attacks and it's very simple to
do so if you just pass the dash dust
we'd only flag when you start a
container that the container file system
is completely read only and you can't
write any files now okay most
applications aren't going to be able to
work like that but what you can do is a
create a volume for the files that I
need to write to so say your your
application needs the right to a temp
file just create one volume for that and
then if attacker brig saying they won't
be able to do things like you know write
a script to serve farmers family saw
or edit the file to change your front
page very similarly you can set volumes
to be viewed only with will not drop
capability so going back the point bit
linux capabilities what you do is you
pass a dash dash cup dust drop argument
to drop a capability so in this first
line here we're dropping the ability to
set the UID and GID flags on files you
can also drop all capabilities and just
add back the ones you need the issue of
this of course is that how do you know
what you give lt's you need a little bit
of a black eye you end up just like
testing it so it is very useful but it's
a bit of extra work cpu shares I always
wondered whether or not to talk about
this because CPU is actually limited by
default so by default if your CPU is
pinned if you use number cpu and a host
your containers actually seat share the
CPU equally they reach given an equal
share the CPU but you can change that
you need to be aware that there's a
default rating of one or two for so in
this example we start a container to
default waiting one or two for then we
spin up to other containers with the
weight in a five point to reach and what
that'll mean is the first container can
take up to half a CPU and the other to
get a quarter each but this only comes
into effect when the CPUs pinned before
that point anybody can use as much as
you like there's also offer things like
the completely fair scheduler that you
can now use so there is other ways to do
cpu but because it is limited by default
or shared by default I wouldn't worry
too much about that one what are you
worried more about is memory limits
because I see a container can go and
grab as much memory as a Lakes so you
can pass the dash mne flag to set put a
constraint and the mountain memory do
you do be aware though if you do that
that that sits the main memory but
there's also you get the same amount of
swap so this gives you five new 12 may
get some memory plus 512 make
bites or swap and there's a separate
flag to limit the both of them this is
an empty of someone so you can also
define set new ideas GI de binary's so
if you look at the typical base images
like debian ubuntu okay yeah so you can
so an on-base images include quite a lot
of binaries that I've set you a dnc tid
bits and what this does is it allows
them to have elevated privileges so the
class example is paying you don't really
think of ping as being security
sensitive but actually as elevated
privileges so they can create raw
network sockets what that means is if an
attacker finds a bug in ping I know they
can exploit that they can elevate the
wrong privileges within the system and
the funny thing is you can find these
binaries quite easily by just simply
when find I guess most of you probably
familiar find and you get this just like
the ones in the default debian image and
for example who hands up if you know
what CHF anus there's nobody know what
CHF n does see so why is that in the SRT
stands for change finger name who
remembers finger ok that's better so
yeah a full name is fingernails the past
the video password with the lon Reiman
from the post with fur so maybe well
anybody nobody knows what it is so why
is it there so I'm potentially I mean to
be honest it's probably not much of an
issue because these are all very well
tested utilities so I'd be surprised if
you find a vulnerability in them but you
could and then an attack would be able
to elevate the privileges within the
container so it's actually trivial to
define them because you can just put
this into your dacha file and he put
line all that's going to happen is I'll
go through all the files find the ones
that they are and change the permissions
on them and to remove the set uid bit
and just to prove it if you run it again
you get that another interesting one is
communication so when you launch your
dacha demon we just updated it to 1.8 so
prior it was docker dash D but now it's
dr. Damon he passed a dash dash I see
sequel a false flag you turn off into
container communication so your
containers can't talk to each other and
you can take this kind of talk to each
other they can't attack each other so an
attacker breaks then you won't be able
to go and talk other containers so
arguably peace but it is a bit useless
because now your application container
can't talk to the database so what you
can do is you can sit the iptables flag
which i think is now set by default I'm
not sure it used to be and he said that
allows linked containers to communicate
so it all SEPs iptables rules that
allows to expose ports and tooling
containers to communicate but nothing
else can communicate by default normally
if you don't ICC on the false or
anything else all containers for a cheap
immature through an access each other
sports as their free completely which as
it surprises some people but you of
course you need to know the IP address
of the other containers the final point
and this is a goes back to talk this
morning if you saw Cessnock sharing
secrets so you want to get put things
his passwords and API talking into your
containers and this is a quite a tricky
problem we did have the same problem of
VMs but it was less exacerbated because
the aims were longer lived and he'd do
things manually if you had to but
nowadays the containers to come in and
go in the whole time it really is a
pressure on to automate this process so
the first way you can get a secret into
image right just right in your docker
file and put the API token you
dockerfile hopefully most of you agree
that's a bad idea and you should not do
that under anything to honor any
situation the next way is environment
variables and a lot you probably do do
this and I'm not saying as bad but in
containers the door thing is great
either a reason a lot of people use it
is because it's the way suggested by 12
factor apps and I'm not knocking 12
the vmworld and when I own container
land and the issue is just the
environment variables when you you know
so the deity just be a dashi API token
equal to your secret and the problem is
it can just be seen in too many places
so link containers get access to all the
other continuously linked these
environment variables so if you link two
containers together that container can
see all the secrets near the container
which the prices from people I also
inspect even inspect you can see all the
environment variables in the container
and they can't be deleted so you might
think well it's okay I'll just go in and
operate it that won't work you still be
able to see and inspect but even worse
they tend to get including reports right
so you asked for support request unlike
the your vendor says okay can I get
details your environment so you just
give it to them and you know you just
included all your secrets there which
clearly isn't a good thing so i'm going
to say environment variables okay they
work but they're a bit too visible and i
so the next way and I think this is
probably the way most people are doing
at the minute it's just to make files so
you keep your secrets in a file and you
make that into the image until the
container sorry one of the nice things
about that is that you can just make a
configuration file so you like with a
problem environment variables you also
do some work to get that environment
variable into the application quite
commonly at least with a file you can
just mount it into configuration file
and i'll pick it up without any special
tricks yes oh this works but it's a bit
icky and my main problem with it is that
files tend to get checked into source
control and clearly that's a bad thing
when you got secrets so the future I
think it's going to be these are secured
key value stores so a few of you might
be an incest talk this morning when you
talk to be evolved yeah I think that
really is going to be when the major way
to do things are just going forward
basically the idea is your key value
store as all your secrets in it it's a
design to store secret so it keeps them
encrypted has special features like
leases and so on yeah so you can say how
long a secret is good for it might a
brick glass proceed year so you can lock
everything down if somebody breaks in
things like that as two main ones vault
that Seth talked about this morning and
also key ways ki voice is quite
interesting because it was built by the
guys at square who did a payment process
and so on I domain engineers from there
now actually work at docker so a key
which is probably it likely to be very
well supported a few here probably
thinking houses actually solve the
problem because you have to authenticate
to the key value store so how you going
two ways one is to go back to
environment variables so you pass them
key value store or sending it to the key
value store but you make it a one off
key so it can be used once and then it's
useless so at least an attacker gets it
the camp it's no good to them the other
a bit more complicated but we now have
volume plugins and so you can naturally
there's a volume plugin for both key
ways and vault and what that does is a
it creates a file inside the container
that contains the secrets that that
container has access to based on what
they have the rights to access and key
ways or evolved so now you've moved all
the sort of secret set up outside of the
container itself there's no tokens being
passed in there and just as phyleus them
populated from it was evolved so I think
that's probably a future okay I think
the main thing I wanted to dry form is
like containers odd security so he using
vm as a minute I mean I container sent
to the mix you only added another layer
of security you're very unlikely to make
things less secure it's almost
impossible maybe if you did things like
you know I do what you think is the root
user yeah use containers with VMs if
you're concerned about security I know
always think defense in depth and least
chipotle at the user instruction click
the usual weight rooms
yep I've been playing with that a fair
bit recently one of the problems I've
found is when you do that and then you
use the copy or added Jeff copy file it
ends up being owned by root in the
psychic designer perspective of the user
out site and so what you end up doing is
having to pillage we use a nasty pattern
when you temporarily flip the room copy
the file and change the ownership of it
and then flip back to the original well
what be root you be seated well when we
root user privileges just be the
privileges of I'll just retain the
pillagers where a fail it was on the
host but uh sorry the same oh yeah okay
that's right yeah so and that's exactly
why the official images don't have that
user statement right so and you look at
the official images they do what they do
is like for postgres or redness is a
when its container starts up they do
like a pseudo CH own I want you don't do
th and see if they're all your roots or
just see it shown I'm then they do a
pseudo to the actual user it's not used
to do they use they use that tool could
go so I'm ghost is quite nice because a
few pseudo you end with two processes
but go so like it just Forks that
process or writes that process yeah I've
got a question it's something Chris
one's talked about a bit which is
putting less in your containers you
didn't we talk about that mean this you
could put the whole of Linux in it or
you could just have scratch and go go
binary or whatever see binary yeah
absolutely so that mean that comes back
to attack surface points so you can
really cut down I liked it attack
service within the containment by just
putting the bare minimum minute and also
that goes on to then you think about
unique URLs and things like that you see
seeing their people doing that commonly
or is it the people still sort of well
it's hard to tell what people do really
the minute I'm not sure I've seen people
I don't think many people do it now it's
fairly advanced I thinks though there's
also an issue with debugging right so if
you put the minimum in it and something
goes wrong you want to debug that
no tools like if you just got like a go
binary you can't even get a shell yeah
so the question was I guess is this
month the question is a yeah things I
see its fruit another isolation
mechanisms and how do they compare and
yeah that's a fair point I mean as well
as docker we already have rocket and
it's free PS jails and solaris zones and
so on and I've never SAT and compared to
mall but docker is going to be it's
pretty feature fool I'm a bit more
tasted in a lot of them so I would say
that's probably up there in terms of
maybe not security but certainly
features in isolation chroot I'm there's
always been issued to see it through
that's more of a plaything I think
welfare anyone knows okay well thanks
everyone for coming and have a safe trip
home and hopefully we'll be back again
next year I think that the big
discussions were probably in
october-november rather than September
and probably a single track conference
again but we're not quite sure what to
make it about so if you have any ideas
really looking for input and any
feedback or input on this conference as