I like my tests, like I like my friends, reliable and flake-free

Amir Rustamzadeh

Recorded at JSConf US 2018


Get notified about Amir Rustamzadeh

Sign up to a email when Amir Rustamzadeh publishes a new video

[Music]
[Applause]
[Music]
[Applause]
[Music]
[Applause]
[Music]
[Applause]
[Music]
all right
Lovelace out here well thanks everyone
for for coming and thanks to all the
event organizers before putting together
this great event today I'm going to be
talking to you about end-to-end testing
my talk is titled I like my tests like I
like my friends flake-free and reliable
and some of your laughing but you know
the reason why I chose a picture of
friends camping outside together is
because if you've ever planned the
camping trip with your friends you know
that some of them are not reliable they
flake they don't show up they don't
bring the right things and the
experience isn't that enjoyable but I
want to show you that end-to-end testing
can be enjoyable and it can be
flake-free and it can be reliable if
done properly so Who am I I am Amira
Sam's a bit I am an engineer on the
Cypress i/o team feel free to reach out
to me Twitter or email if you have any
questions about the talk so Before we
jump into things let's visit the testing
pyramid this is the pyramid that's
typically shown when people talk about
testing at the bottom of the pyramid
here we have unit tests and then we move
up to integration tests and end-to-end
tests finally the reason why unit tests
are at the very bottom of the pyramid is
because developers love to write a bunch
of these unit tests right they're very
easy to write we have a lot of fantastic
tools to write them with but as we move
up the pyramid and get to end-to-end
testing we find that writing end-to-end
tests can be difficult costly slow flaky
and this leads to a lot of people
actually just not writing end-to-end
tests which is not very good because
the best kind of test to ensure a great
experience for users is Antoine tests
it's great for them it's great for the
business it's great for everyone
but it's difficult so before I get into
the nitty-gritty parts let's kind of
step back and look at the Renaissance of
the web we've been seeing for the past
decade or so and this is just within the
context of frameworks that have come out
that have made web development
significantly easier you know around
2005 or so we had things like Ruby on
Rails and Django come out and they made
it significantly easier to write
applications this is their in time we we
had things like Twitter and Shopify and
things like that come out it was just
the hell of a lot easier to write these
types of apps it gave us a nice
construct to think and reason about
shortly after we had things like jQuery
come out which made it really easy to
write JavaScript across multiple
browsers that kind of no more normalized
all those little edge cases that you
would experience writing JavaScript and
during this time frame you know 2006
then we have like smartphones come out
around 2007 and more and more people
started using actual software as part of
their daily life they started to rely on
and more and more and they were
experiencing and very well done
applications they were very snappy they
were very user friendly and this is what
people majority of people not us techie
guys but you know they started to to
demand these types of experiences about
every about all the software they were
experiencing and this essentially you
know at around the same time browsers
got their act together right they
started actually playing nice with
standards and computers got a little
faster too and then we ended up with
very fancy frameworks like react and
view j/s so now we're starting to write
very rich web applications which are
experience and just fantastic overall
one important thing to note here is that
this progress is essentially led to the
movement of complexity from the back end
to the front end
the past we would just render very
simple HTML and ship it down the wire
but now we're sending really large
JavaScript applications down the wire so
the way we write applications have
significantly changed but the way we've
been testing them has not changed we you
know we now tools have always improved
they have always progressed but the way
we've done end-to-end testing has
changed very little before all this new
fancy frameworks came out all the way
back in 2004 selenium came out which
many of you probably have used to
end-to-end test your websites you know
shortly after you know selenium
webdriver came out and there have been
other wrappers and tools that have made
using selenium much more easier to use
to kind of give a shame work of time
here
Seifer sale came out in 2014 so the just
at this point here is that the way we've
been writing applications has changed by
the way we're testing them has changed
very little let's look at the status quo
of n2n testing which is selenium now the
architecture of selenium is like this
you write your tests in any language you
want and your tests essentially talk to
the proxy server of selenium selenium
server is just the proxy server and
selenium server makes request to the web
drivers that are actually controlling
the browser to make your test happen and
what we have here is the classical
sinking dilemma we have two separate
systems and we're constantly trying to
keep them in sync together this is very
difficult to do the selenium can only
ask about what's going on in the browser
it doesn't actually know what's going on
at the right time this is what
essentially leads to flakiness and
non-deterministic tests selenium has to
go through this proxy server and has to
wait for all these Network hops to
happen before it can actually know
what's going on it's not it isn't
exactly it doesn't know exactly when an
event took place in the browser and so
you might say well the hell with
at all I'll just use a stone or some
other type of dom emulation well the
thing is none of the users are using
Jaya stomp to run your application
none of them zero so you have to do real
browser testing and that enables you to
test your users real actions it's the
best way to do end-to-end testing and
it's the primary reason you would do end
to end testing because you want to
simulate your at your users actual
actions the thing is which I really want
to emphasize is that don't let your your
users be the first ones to test your app
the answer and test your application
right it's gonna be a bad experience for
them and it's gonna cost you time and
money
and it's just not good all around for
for anyone some people will like to rely
on let's say you know like a crash
reporting service or error reporting
service or like well I'll ship my
application out and if anything goes
wrong well my error reporting service is
gonna tell me what went wrong and I can
just go ahead and fix it well the thing
is not all errors are perfect errors
that can be caught by your you know a
crash reporter they don't all have
stacked races that could be simply
experienced user experience problems and
frankly most users don't actually report
problems when one day when they see them
so this is not the way to go you need to
be testing the exact experience of your
users before you ship yourself where
it's the only way you can confidently
expect that everything is gonna go
smoothly so let's get back to kind of
the historical context of things in the
past
we mostly wrote stateless websites right
very simple stuff ship HTML down the
wire now we're running writing very
stateful apps and it's a completely
different ballgame they they're just
completely different beasts and we need
a much more optimal way of testing these
types of applications and this is why
Cypress exists so Cypress is a really
nice and fast and easy way for you to
test anything that runs in the browser
it's completely free it's open source
and it's MIT license it's very easy to
install all you got to do is just NPM
gonna get everything you need you're
you you've probably seen before like
mocha test framework chai assertions and
sign-on for subbing and mocking so with
a simple install line you're gonna get
everything all in one tool
alright so let's go ahead and see it in
action for a little bit okay so I'm
going to demonstrate to do MBC app if
you're not familiar with to do MBC it's
essentially a to do application that's
implemented in various frameworks to
kind of show how you know how we'll look
in those frameworks if you're working to
write it to do that so the way I
launched Cyprus is by just typing in
after I've installed it I can just type
in MPX let me make it a little bigger
Cyprus is a binary that you know just
goes into my node modules so that's why
I'm using MPX right here to run the
local version of Cyprus and I'm just
going to type in Cyprus open let's get
the sizing down here and give some
people a chance to sit down new people
coming in so the Cyprus command is going
to bring up the desktop GUI
that is Cyprus this is the electron
application and what it shows you when
it first launches is all the spec files
that exist in your project directory
this is a simple to do application
there's only one spec file to test
everything so the way I can start my
test I just have to select the spec file
that I want simply click on it and
Cyprus is going to immediately bring up
a browser electron in this case and it's
going to start running my tests after it
has pulled in that to do application as
you can see Cyprus runs very fast and
actually once the once the GUI is up
it's going to start up the browser very
fast as well so right now this is a
to-do app it's going through all the
various permutations of a to-do
application creating new to do's marking
them complete making sure they're right
that they're the right count'em to do is
exist and so on so let's hit that lets
get that finished
but I mean but and we're done
all test pass yay so this is your
cypress experiences how will look like
application it runs within the same run
loop as your application so it knows
exactly what's going on in your
application it can react quickly it can
make assertions right away
unlike let's say selenium where it has
to just make make a request over over
the wire and hopefully it will get back
so let's look at what we what we're
seeing here on the right hand side this
is your application we reloaded an
iframe on on that's on the right hand
side and up top this is the URL that
correlates to what you're seeing on on
the page and on the left hand side this
is what's called a command log this
shows all your test Suites and all the
tests that were executed let's look at
the first one which is we're very
certain that when the app loads the the
cursor should be focused in the input
box which makes sense because you know
right when you open it to do app you
want to immediately start typing in
to-do items and as we hover over
different Cypress commands that were
executed right here we'll see the app on
the right-hand side change Cypress takes
a dump snapshot of you know whenever it
runs a command so it can show you
exactly what happened with each command
so the first thing we do is we visit the
root directory of this app which is
that's why we're not seeing anything yet
after we visited it we're going to use
the Cypress focus command to assert that
we are focused in this input box and as
I hover over that you know it will
actually select that input box if you
look at another test right here actually
we're going to create a new to do with
this test we're going to go ahead and
make it to do item for buying some
cheese so as you can see if I'm hovering
my mouse over that command its
alternating the before and after
snapshot so I can see what happened
before this command ran and I can
actually pin that Dom snapshot and I can
just see what how the
look like before and after myself and I
can unpin that so what's cool about this
is you can hover and you can essentially
travel back in time and see how how
things look like it's a very nice way to
inspect how your application is
functioning another cool thing is that
Cypress you know is just running a
normal chromium browser so you get
access to dev tools right here and
whenever you hover and click on a
command it clears the console log out
and it gives you relevant information
about that command that ran so this is
great when we're you know especially
inspecting you know requests and things
like that we can see the request and
response payload it's it's very nice so
it's very easy to to debug because you
get access to dev tools right
out-of-the-box now let's look at
something interesting I'm going to go
ahead and let's go bring it let's go
look at that tests we were writing so
here's that test for making an ascension
that assertion that when the app is
loaded we're focused on the input box
and this is how the test for that looks
like all ciphers commands start with
this global CY command and then you
essentially have a chain of commands
that you can execute and that's our
ciphers tests are written so we're doing
sideout focused which is asserting that
hey are we focused on the input box and
it's how that looks like so let's only
run this test
oh there we go that's alright okay so
now we're only running that one test now
this test passes let's go ahead and
break this app right now so I'm going to
go in this app and I'm seeing that the
autofocus for this input box is set to
true well let's go ahead and break in
and set set it to false which is going
to make this make this test fail so
everyone's going to wait and be like hey
I kept I kept retrying to see if this
test will pass but it didn't and notice
this is how it looks like when problems
occur and we go out of our way to give
you very human readable messages these
are all handwritten messages so I'm very
proud of that so nobody wants to see a
kid garbled you know nonsense when
something goes wrong so let's go ahead
and rerun this test and Cypress is
trying to make this assertion but I'm
going to click the info box myself and
force this test to pass so the stat when
you're Cypress loads the app this is
your actual app you can play with it you
can type things in right here it's it's
not some weird version of your app it's
the full thing and me in focusing on the
input boxes is essentially changing the
state of the Dom and making this test
pass alright so exit out of here and
it's actually fix this before we move on
and let's get back to our discussion all
right so very overly simplified view of
Cyprus architecture it's that the Cyprus
test runner runs right alongside your
app on the same run loop as your
application which is drastically
different than anything out there so the
benefits of this as I said before you
get direct native access to the DOM and
therefore your app and this allows for
interesting types of tests you can run
and we'll see some of that shortly your
tests run very fast they can they'd run
actually just as fast as your
application so it's that's just
fantastic you're just going to get a
hell of a lot faster tests and as you
saw
Cypress there was no weight command
there was no like timeouts or anything
like that
Cyprus automatically retries and keeps
me trying to assert the state of the Dom
that you're expecting so you don't have
to mess with timeouts or weights or
anything like that and I get full access
to dev tools which you know it helps a
lot for debugging my app and I can have
my editor and Cyprus side-by-side and as
I'm changing the mice my my test specs
Cyprus will automatically reload and run
the test so that's just fantastic and
what this all leads to is flake-free
consistent and reliable tests so how the
Cyprus make this happen well here's an
example we got three commands that are
running these are very simple they're
just trying to you know select the three
Dom elements Cyprus will look at this
and it will essentially queue these
commands up and run them in the order
that you've declared them so we'll you
know you'd find the first one second one
third one exactly how you've written it
in this way things happen
deterministically so as I said you get
direct access to the Dom and this allows
for cool types of tests my brilliant
colleague Gleb amitabh has written a
series of component testing libraries
for all your favorite frameworks out
there and with this essentially
essentially allows you to do is take one
of your components here in let's say in
react you can just take just that one
component you know in isolation and
mount it to a real browser Dom and then
you can run Cypress tests against it so
let's go ahead and see an example of
this I will be showing this to you in
the view version so let's go ahead and
but wait for that to open up so this is
essentially an example within the
Cypress V unit test library it's a it's
a simple counter
component but which is not interesting
but if the part that is interesting is
that this counter uses view X to manage
its state if you're not familiar view X
it's like Redux it's just a central
state store just see how it runs there
we go whew we got a nice little counter
so that's how it looks like but more
importantly we want to look at the code
for this so the Cypress view unit test
library is going to give you this Mount
View function if you were using Cypress
react unit test you would get a mount
function for mounting react components
and before each test I can go ahead and
mount my component however I wish so I'm
passing in the template for my component
the central store instance for it to use
any extensions or anything else I want
to use along with it and it would just
take that component and mount it to the
dom and here's the cool part i can
directly manipulate the state of this
app with just plain old JavaScript the
Cypress for unit test library is going
to give me a view instance on the
Cypress global variable right here and I
can directly get access to the central
store and I can get the state of the
count in the counter or I can change the
state I can I can change this directly
in my tests now a real world example of
this is let's say you want to test your
login page right and you need you need
to authenticate before you can do other
things in your test
well maybe you only need to do that once
if via the UI of your app right you you
go ahead and you type in your username
password then you hit the login button
in your tests but you really need to do
that every single time you want to
authenticate to run on one test no you
just want to do it once you had now you
have confidence that the login UI works
and then every other time you need
authentication in your test you would
simply authenticate that user via
javascript directly you don't have to
keep going through the UI just to
authenticate you can just do it in plain
old JavaScript and this is going to be a
much faster way of running your tests
let's close that out okay so so far I've
just been showing you the ciphers open
command you've constantly seen me run
Cypress open and that's great for your
everyday you know developmental workflow
Cypress on one side editor on one side
test-driven development whoa ah well
there's a whole other side to Cypress
and that's you can run it headless lis
using the Cypress run command this is
great for writing Cypress tests in your
CI and it's a much more efficient way of
running all your tests you know any
real-world app has like hundreds of
thousands of tests and I there's no need
to run that like visually you can but
it's just a hell of a lot faster running
headless Lee and the cool thing about
running Cypress headless Lee is that you
can actually get a recording a video
recording of your entire test run by
just you know passing in this record
flag and we'll see that let's go then so
that to do NBC app I had now we fixed it
right here and let's go ahead and remove
this only right here and let's bring it
back up okay so now I'm going to run
Cypress run past the record flag and
there's also this key flag this is this
this key flag is provided by Cypress and
a uniquely identifies your project and
this is useful when you're using the
Cypress dashboard which is a
complimentary service to Cypress and
we'll see that shortly but I'm going to
go ahead and run this and this is going
to bring up a headless browser and it's
going to start running Cypress headless
Lee great now it's brought them up so I
don't think I have proper internet
connection here that's probably why it
didn't said that message your votes
ringing let's see
hope
that's the one anyone know the password
what is it
who is sports horse okay that's right
thank you
Wow cool
so while that's doing that my my test
run ran and and it ran headless Lee and
after was done it gave me this this the
path to the video it took of my test so
I can actually bring that up and I can
play that and so this is exactly what
you were seeing in the browser it's the
exact video of that entire test run
okay so now this leads to the whole
other part of Cyprus which is this
complimentary dashboard service which is
actually also optional to use but it's
great one your writing test in CI
because when it's running a CI you have
no idea how what's going on in the
browser you don't know how it looks like
you need a way to see what's going on
and the dashboard it records all your
tests it tells you which specs ran which
ones passed or fail will show you error
messages it will show you video
recordings that like the ones you just
saw will show you screen shots automatic
screen shots are taking when something
goes goes wrong so it's it's a it's
great for that and we can actually take
a quick look at that so here's an
example of actually one of our own tests
this is test for the Cyprus dashboard we
use Cyprus internally obviously if to
test everything we use Cyprus to test
Cyprus so yeah so that's works great
great for that this is an example of a
test run where you know things fail for
whatever reason and we have those exact
error messages of things went wrong
there there right here is a key I assume
you can see the exact standard output
kind of like how you would see them main
CI you get to see screenshots that of
the problem place and you can see videos
kind of like the one you just saw right
now so it's great for then you can
actually you know it's you look at
everyone that like passed so a passing
test will look like this
and we'll get to some of this cool stuff
in a little bit so until recently you
know the dashboard was just for seeing
what's going on in your CI environment
it was great for that you know it's also
great for collaboration because you can
like link your co-workers like here this
is the exact thing that we're on here's
a video of it here's a screenshot of it
but now you know we recently released
the Cyprus 3.1 which adds the
parallelization feature and we've kind
of been you know not very vocal about
this feature and we haven't talked too
much about it publicly so this is the
first time we've ever talked about it
publicly so you know congratulations
so yeah 3.1 is out it's been and we're
offering parallelization in that and
it's been along the way this feature and
what parallelization allows you to do is
it can automatically load balance your
test runs across your CI resources and
all you got to do is bring up how many
CI resources you want you know three
machines time machines whatever and all
you got to do is just pass this new
parallel flag in when you're running
your tests let's go ahead and see how
this works so let's say you you
configure your CI environment to you
know use three machines and this is
what's going to happen the first machine
is going to contact Cypress and it's
gonna be like hey buddy I'm ready for
some work and here all the tests they
need to run for this project we're like
cool great we're going to now
intelligently figure out what is the
first thing this new machine is supposed
to do which test specs should this
machine run well we determine and we'll
see how that's done that this machine
should go ahead and start with the login
spec to test the login page and then the
second machine is going to like alright
I'm ready to and then we're like okay
great you can do the signup page and
then the third one here you can test
this widget and this all happens
obviously in parallel so as you can see
you know now we can optimally use our CI
resources now how do we know which
respect to give first and which to do
second which to give third right how do
we go about doing that well this is how
using the cypress dashboard you know
you're recording your test to it and you
know Cypress is in a very intimate
relationship with with your test and
that was everything you know about them
how long it takes to run each of your
specs and so we have a test duration
history of any given spec and then we do
some clever math do some analysis on
that and then we can forecast how long
the next test run for a given spec will
take once we have that information we've
configured it so that we'll run the
longest test first because
it's gonna take the longest we should
probably get started on that first right
and and we do this on the fly for every
test spec that you want to test so we'll
go ahead and automatically compute and
forecast how long it's going to take to
run the log in spec and sign up spec and
widget spec and then this is how we go
about load-balancing them across your CI
environment and so what this allows you
to do is you know get the most bang for
buck from your CI right because yes you
can say like that you're using circle CI
and you can configure it to have 10
machines running it's great but it's
kind of like having a really powerful
computer and not having the proper
software to use all the cores and all
the memory right but Cypress allows you
to optimize your test runs across your
CI resources and what this essentially
leads to is you saving a lot of time and
money because your tests run faster and
the developers cost a lot so we can
actually go ahead and look at how a
parallelized test run looks like this is
for actually our cypress dashboard thing
I had another one there we go this one's
all right
so this is showing you three different
test runs over here and we've labeled
them and I'll show you how that labeling
works it's actually a new feature as
well and you see like this one says 2x
chrome zoom that a little bit 2x chrome
another next one says like 1x electron
4x electron this is just a naming
convention we're using to say okay this
test run was testing in chrome and it
used to you know to machine CI machines
to run this test and this one uses four
machines so if you look at the duration
history of each one of these for example
if we look at the the electron tests the
one that just used one machine it took
two minutes 38 seconds and the one we
use for machines only took 46 seconds
and you know 2 minutes 38 seconds is not
a lot but when you're looking this is
just the example project but when you're
running this against a real big project
you're really gonna see the time time
savings here and it only aggregates over
over time and the next thing we have
that's with 3.1 is now we enable you to
group all your test runs together it's
it's like a nice general-purpose way of
grouping different test runs so for
example you want to test your main user
facing front-end app and you want to
label that as like front-end and then
you want to test like your admin page or
your ad in an application then you can
you know label that as admin so then
when you go to the dashboard kind of
like you saw here we use like the names
2x chrome 1x electron whatever but these
are the group names we chose and and
we're able to see them all in one test
run we group them all together you get
to see them all in one page and this is
how like you can associate them all
together and so if we kind of look at
these parallelized tests use and this is
kind of like a timeline view it's very
similar to like how the dev tools
waterfall network of you looks like so
you can see like we ran the action spec
first and then you know and so on and
the longest one ran first and as
go down the duration of the test specs
get smaller and you can like hover over
them and get some more information about
that shows you like which machine ran
this so this tells me that you know this
spec was ran with machine number two
cool alright and another thing to know
about Cypress is that it has fantastic
document documentation something that
we're actually very proud of so whenever
you have any questions it's like the
first place you shoot you should go to
has also a great search so you get to
see everything there and we have
literally have a command to do anything
and so refer to that please and then we
also have a best practices page and
there's actually a video of the creator
of Cypress he gave a talk on best
practices and there's the link for that
and I'll be sharing these slides so
that's another one I recommend that you
know he goes over how to you know
organize and write your tests and you
know to conclude you know Cypress you
the status quo of end-to-end testing
it's something that's been very
difficult to do properly in the past and
we're hell-bent on just completely
changing that and creating the best
testing experience that this industry
has ever seen and I'm you know I'm here
with Randal Kent he's one of the
cofounders of Cypress so we'll be around
if you guys want to talk to us and chat
with us and thank you happy testing and