How Klarna Optimized & Personalized the Checkout Experience

Samare Jarf

Recorded at GOTO 2017


Get notified about Samare Jarf

Sign up to a email when Samare Jarf publishes a new video

[Music]
do you hear me yes good hi everyone my
name is Somalia's and I work here at
Florida and today I'm going to talk a
bit about how we what we're done to
personalize our checkout and so first of
all Who am I I work your ex on as a data
scientist what my team and I do is we
try to understand if it's nicer customer
behaviors through modeling and mainly
through statistical models to try to
predict different types of customer
behaviors to improve our processes to
improve the offering to our customers to
of you to shop with us and may need to
personalize your customer experience and
I worked here for two and a half years
and before that I graduated at lgh
England and where I studied financial
statistics so today I will present to
you how we explore and I have optimized
the checkout experience through
personalization so first of all a bit of
a background so what is Nick clone at
checkout so if you go to a merchant
online and you're shopping and you put
something in your basket and you press
by you come to our check out where we
want to make it as simple for you to
complete your purchase with the team
with it that you want we want to make it
as simple for you so you can complete
the purchase which recalls we want you
to convert and one corner checkout was
launched back in two thousand twelve
sarna had a groundbreaking great idea
that the purchase doesn't need to be
both buying and the pain we can divide
the buying experience from the king so
when you come to the checkout you can
buy whatever you want to buy we will pay
the merchants and then we'll wait for
you to
yes and we do that by offering you 14
days free credit which is very very good
for conversion so if you're running to
your train and you want to want to buy
the train to get as fast as you can
before entering the the train you just
want to buy it now and then you can
decide on how you pay later when you're
on the train or the week after whatever
you want to do it this we try it out and
we able to sit and it was really really
good it improved the conversion a lot
for many many many customers but we want
to know still we have two different
sauce when will we show all the payments
is for you and one that is a one-click
experience which one converts for which
type of transaction for which customers
and to learn our product is crucial we
want to improve our conversion because
Sarna is selling to merchants on
conversion that is what what is wanted
by is for because we get customers to
easily convert your purchase that you
want so when when doing this we want to
see okay which type of group of people
can we sum to one check out or the other
now at that time we didn't really have
any hypothesis on which transactions or
which customers we thought would convert
better with the payments lecture one or
the one click one so we started just
random something to both of the floats
just then look at the data and say okay
what can we see from this who can we
send to which flow and to also see what
different opportunities can we see when
when doing this we also want to know
then you have the effect of it we also
want to learn what behaviors are we
changing when doing this we want to know
what the customer what behaviors of the
customer is changed when we do the one
click purchase or if we just plenty with
all the payment methods so there were a
lot of different trends we saw when
doing this but two of the main
main things that we saw when when they
be testing this or when random sampling
was with the famous lecture where we
show you all the payment methods out of
the people that completed with that flow
rather than the one-click flow we saw a
much higher average order value so that
means people that are shopping something
expensive don't want to complete the
purchase when we give them this easy
one-click flow they'd rather they are
much as a higher probability that they
will convert if we show you all the
payments and that could make sense if
you're buying something really small and
cheap you want to have the one-click
experience because it is quite it's easy
you don't need to think that much better
but if you're buying something that is
expensive you probably thought about
this a lot you are going to buy this
sofa then you don't want to separate the
buying from the pain you have thought
about it a lot and you know you want to
pay this right now so then we could find
a group of the the high ticket ones that
we send to the the one with a famous
liquor one other thing that we saw so
even though in the both of the checkouts
we offer you the same payment methods
and so in in theory we should have the
same payment mix and in both of our
checkouts but that wasn't the case when
we looked at the data so for when we
show you all of the payment selectors or
all payments is in the previous lecture
we saw that we have a much higher share
of court payments then when we give you
the one-click flow so then we divided
used part payment with us before and the
ones that haven't and we saw that well
if you how you support payment before
you're much more likely to convert if we
show you all the same incentives and the
reasoning behind it is because when when
you come into the checkout and you
probably don't have money to pay us
within 14 days if we send you to the one
cake flow when we give you an invoice
that you then you can pay with part
payment you don't really read all the
text that we've written that even if we
give you an invoice now you can hand
change it to a port payment plan so we
see that customers don't we
what we write in the check out so it's
better to send this type difficult for
people to the one where you just
presents all the payments lifters and
that increased conversion for for that
group but we also want to see okay so
why when the check out is looking like
this when we present all the payment
members to you this converts worse than
when we show you the one quick slow so
one of the reasons is we showed exactly
the same clothes every one everyone gets
invoice pre tickets on the top and even
if that's not what you're looking for it
could be quite hard for you to go
through all of our pain meds and Sweden
I think we have around nine finger
methods so it's it's very hard to go no
one will read all the nine ten minutes
and have the energy to go through them
so the one-click flow you you we want to
improve but the one-click experience in
this because we don't want to force
anyone to get an invoice if they don't
want any voice for example the high
tickets one so the park plane was they
want to choose the payments and said
they don't want us to choose invoice for
them and then they can change it
something else but also we're selling
the check out on high conversion so we
can't lose conversion we can't move
everyone to this so so we thought okay
so if we could make this a one-click
flow we can get the conversion up but to
do that we need to know what to click
Moelis and so what do you want to wonder
you want to buy with when you come to a
check out do you want to pay by car do
you want to pay by in voice account how
do we know what Tim amis do you want to
use when entering the checkout so one
very easy thing to check is ok so what
payments method did you use it last time
you shot to this then we're not going to
optimize anything for new customers but
at least we might be able to optimize
for the returning customers but
unfortunately looking at the data we see
that only twenty-five percent roughly
out of the returning customers stay in
with the payment method they use the
last time so this is not going to be
sufficient enough so what do we do so
obviously there's not a customer
preference for you so we can't just or a
preferred payment method for a customer
unfortunately which would have made it
very easy so what is it then it is
transaction wise while looking at the
data with
it looks like you two famous master
depending on what you're buying how
expensive it is if you're buying
something very expensive we see that the
account and part payment installment
plans are much more popular also looking
at okay what is it that you're buying
looking at merchant segments the ones
that are within closing issues we see
invoice and we have in any of the other
segments which would make sense because
you might buy something in three
different sizes and what try it out you
want to send back two of them and then
say one they don't want to pay for all
three because you know you're not going
to save all three in the end so that
okay so maybe we can pre take invoice
for for this segment or also looking at
age we see that younger people tend to
use card more all the people like the
bank transfer one and maybe we can look
at age and we also see that younger
people use our new commitments as much
off more often than than the older
population but then this is going to be
a very very basic logic it's not
currently hit rate it's not going to be
that good conversion is not going to be
good enough so this isn't good enough
for us so what can we do we we have a
luxury of having a shitload of the
hitter here at Lourinha every day we get
four hundred thousand purchases through
our system which means that we could do
much better than to look at these three
small variables to be able to predict
what you want so what we did was we
built a machine learning model to be
able to predict what payment method you
want when you're entering the ticket so
you probably all notice but what is
machine learning well it's the field of
study that gives computers the ability
to learn without being explicitly
programmed so
we divided the modeling into three parts
one is determining the problem so what
is it that we want to do well we want to
know what payment method you want to
choose each time you enter our check out
each time you're buying something we
want to predict in real time what
character do you want to use so it is a
classification problem or a regression
problem well a classification is when
you're predicting a discrete or a
categorical variable and regression
model is when you're predicting a real
number or continues number so for
example if we would predict session
length or something like that but as we
are predicting which payment methods you
want is a categorical variable so we
will build a classification model the
next step is the most important and the
most fun part of the whole modeling
which is building the features this is
very very important for the model even
if we tweak all the smaller parts of the
model and the feature part is the most
important as Pietro domingos said some
machine learning project succeed and
some fair what makes a difference and
easily the most important factor is the
features you so we're looking at ok so
what data do we have we have so much
data that we use to predict it if you're
a foster or not we have so much data
that we use to predict or you credit
worthy or not and so much data around
the coaches itself what can we use and
also what count is there any other data
that we can buy in that would give
something for this model so for example
could weather say something about the
payment method that you want to if you
want to use maybe if it's sunny outside
if it's funny you tend to be the
likelihood of you being outside is
higher if you're outside and you have
your wallet close to you and then you
might want to pay with card more often
and it was cloudy you might be inside
your wallet is in the hallway you don't
want to go fetch it you're under the
blanket and you just choose invoice
because it's very easy maybe let's try
maybe shoe size that's something about
your your
preferred payment method it's probably
won't and we always try to think when
using different features in the model
that the model picks up is important you
want to try to back it up with a story
why is it taking this variable of why
could this be important when predicting
different premises so if shoe size would
be important how maybe if it's dividing
between the smaller size and the larger
sizes is trying to find gender in this
and Pyramus said for some reason the
fridge between gender but then gender is
a better variable to use so I always try
to think if a variable is picked up by a
model that is that is important why what
is it doing it is something wrong with
the data or should it actually be
picking this variable so when we look at
our features we divided them into three
groups so the features that are
connected to the purchase so for example
we saw that amount of the the order was
very important where your shopping what
type of item it is whether maybe and
other variables that we can think about
that are around the purchase then we
have the variables that are connected to
the customer so for example age and we
saw is difference between some age
groups which payment that is more
preferred maybe shoe size let's try it
and see what happens and then we have a
lot of data on your shopping behavior
since since before so for example the
payment that you use the last time we
probably be quite important in
predicting what you want to buy with
this time also trying to build the
variables so that the model can find
last time you used invoice and the time
before that you used credit card what
differed was the amount where you shop
trying to find some help the model find
what is different between these and
trying to find the patterns between this
is a typical card purchase for you for
people in your age for people that are
shopping with in your city and try to
differ between the different payment of
use
and the third and last part is the model
so we chose to go with a random forest
model which is one of the most use
machine learning models we chose it
because it is it is very easy to use you
don't need to do a lot of the pre-work
if you have a lot of variables that
could be correlated the various the
model is very good at not overfitting
your models and it is a good model to to
use as your your first steps just to see
can we predict this or not some regrets
quotes are very very shortly of how a
random fourth model works so you take
your data set of millions of
transactions and the model will take a
sample out of this data set and a
butcher sample and train a decision tree
and in each node it randomly picks a few
of your features and see okay which one
would give the most to that decision
tree if I use and does that in
everything of them it templates packet
example and draw the new sample there's
a new tree and does this as many times
as you have you want the model to to
build the trees this collection of all
the decision trees is the random forest
model and so when my purchase comes in
we want to predict the questions data is
run through each tree each tree gets to
vote on which promoted it thinks you
want to pay with and the output of the
model is the combined prediction so it
takes all the votes and you get my
prediction and that is the output of the
run and forth model so now we have a
model and what do we want to do with it
well you want to give the customers the
one-click experience so we want to be
able to use this to to optimize the
check out but for doing so we need to
for every transaction every time you
enter the checkouts we need to run our
model and this is quite hard we have
around 1.2 million calls to our system
every day to to be able to offer you the
payment methods and we we want to do
this quickly because you're not going to
wait five seconds for us to run our
models and we need to optimize on on
conversion and if you don't think if you
get the spinning wheel you're going to
drop out probably are we also during the
time we need to compete with running the
fraud models the credit risk models and
together with us and previously we
hadn't run any machine learning models
in in life so what we needed to do was
to create a scoring service for this to
be able to run the random forest models
and output this in real time and it's
also when building a model and you try
to build the best possible model you can
with as many trees as you can and the
trees are as deep as we we could well as
we build them but this will take too
much time to run in life so first be
able to deploy this alive we need to
scale it down to be how how small can it
be and how good can a position position
beads for us to be able to use this in
life so when you answer the check out
today let's say I'm my own vision is is
is entering the check out we run the
purchase through our scoring service and
through the random forest model it
outputs your preferred form of it this
is sent to the client that's then
predicts this payment method and moves
it to the top so that you get a
one-click experience and we do this for
all payment methods or the models can
predict all the parameters so even if
it's you want to pay in the end of March
like Alexander wants to do and you will
get it free tickets and move to the top
and so we have X gonna have with this
data driven machine learning try to
optimize a check out so that it's both
optimizes for your customer satisfaction
so we're going to take questions from
the room anyone have a question over
there we're going to try something
dangerous today yeah I'm supposed to
throw this but I kind of feel bad so I'm
okay thanks and do you do any customer
surveys samples to back up what you
found to check out with talking to real
people yeah so we do it in many
different ways one is we send out
surveys after you've shopped with us to
collect like how is your customer
satisfaction but those are always very
skewed towards the people that really
hate us answer those and no one else
would she really cares about answering
those but we can try to do a lot of user
testing so we invite people here and try
to see okay how are they interacting
with a check out we also call out
because we have in-house customer
service which is very interesting to sit
in and here like and ask some questions
but what what did you understand this
and that but the user testing I think
gives the most just being there and
seeing how to interact with the check
out because even though we try to build
a lot of the variables through how
you're interacting the check out it's
always good to see what are they
actually thinking did you know what you
were picking or did you just move faster
something to think if they're clicking
through it fast it's amazing but
sometimes it's actually meaning they
have no idea where it is but we try to
always work towards Aaron so it's we
it's the best way to know that something
isn't working is to see what the
customers are calling in to us and
complaining about any more questions
I actually had two questions but I'll
keep the one for now so in the decision
tree when you when the machine does the
selection of the past you know do you do
you make it select it you you talked
about the feature set and and you know
what what features are important really
so how do you put in all the all the
variables that you have in all your data
in the in the feature set or or in the
decision tree from the feature set or do
you sort of make a selection manually
first and we from start just put
everything in and then you look at the
variable important to say like what is
it ticking up and what is it not and
because it's it's going to take time for
us to compute all the variables if
something isn't useful like shoe size
and we don't want to say that data or
have it in the real time system so we
still look at okay does it give anything
to remove the bottom not important
variables and if the model isn't getting
worse the position isn't getting worse
than will remove the parts but the good
thing about running for is models is
that you don't need to pick that much it
if it's useless it's not going to use it
but of course if you're doing it in real
time if there's a variable i want to buy
or something that will need to actually
put a number on how important is it do
we want to buy in weather data for this
model or not so a combination may I take
just a sadder is the sort of follow-up
question we have we have time yes thank
you I have to collect myself so when you
know I will get back to it because I
okay there you go right thank you very
easy questions I was wondering why did
you choose the default method as the
invoice I always had problem and I
always better electron extra so is it
intentional you're probably very hard to
predict them if we don't know what what
famous that
can't be too certain which way matter
that you want we predict you invoice
still I mean do you have previous study
for it and then you choose the default
method would be the invoice one in by
tick all right what is it do you have
any previous study for that and then you
decided that that would be the default
method if you have used the invoice Oh
No s-sorry wise now am the invoice is
the prime method that we use to separate
invoice you can still go into a portal
and purchase or convert it to a part
payment plan or you want to pay it with
something else so we always see that as
the the divider between the buying and
click that when we don't know because
then you can you will give you 14 days
buddy then you can figure out what you
want to do later on just one more and
why do you think that this random forest
you select here and what other
alternative where you could do it to
make it more better prediction from your
experience we have looked into extra
boost together with more ranking couple
models because right now we predict your
preferred term ended but the rest of it
is is not a prediction so we're looking
into more ranking models to be able to
rank everything under it in in your
preference thank you each other you're
interesting there was a question over
here nope take this one so what kind of
change to the conversion rate did you
see when you introduce this i don't
think i can say that i'm not sure well
you work at clonus i can say it
afterwards yeah yeah you mentioned a bit
of thing in the beginning there I'm just
curious how you distribute the variance
is its brandon my story doing more
clever way of you classify this is a
woman let's do an a/b test for my
woman's we know next time we think it's
a woman we take the better variant and
so when we did it from the beginning we
had no hypothesis of who we want to do
this for and so then we just ran elected
everything but later on when trying to
different pre ticking we we have the
logic of these people we think and we a
bit just like half of it we predict in
worth of it before
half of them wing pretty key card just
to see like okay if we would have
protect and voice how would a conversion
has been and what would have chosen and
now the repeated card on them did we do
a good job or not just to see like okay
the models predict we think you want
this but what would have happened if we
didn't pretty it would have chosen card
anyways how what did we make better so
this two parts one is making it more
easy for you to buy but one is to
actually predict something that we think
you want so yeah depending on which what
we want to test it it's different then
if we have time for a couple more
questions but try to keep it short the
gentleman and blue behind em and what
was the biggest obstacle between the
moment you have hypotheses and the
validate something what's the biggest
would take the most amount of time um
this I would say it's the feature
building it takes a lot of the time just
finding the data taking the data from
raw data into building the features and
also getting the features we do this in
life that's one of the problems that we
have in before with this we didn't run
machine learning models and in our life
systems so one was to build the whole
scoring service it's the biggest one and
also yeah getting the model to be good
enough but fast enough I would say is
one of the Hartford a question how much
time have you invested to get the
scoring model I'm not sure it went very
very fast I think it was maybe two
months you want maybe yeah around two
service from idea to to be on probation