How To Be A Web A/V Artist

Ruth John

Recorded at JSConf EU 2017

Get notified about Ruth John

Sign up to a email when Ruth John publishes a new video

regulators regulate any feeling of his
property with damn good too but you
can't be any geek off the street gotta
be handy with the steel if you know what
I mean urge you keep regular oh hello
thank you very much that by the way is
an exclusive for everybody that's
sitting here because they are going to
have to cut it out of the video I am so
sorry I don't have the right to play
that music but I really really wanted to
it's fine it's at the beginning it's at
the beginning they can cut that out
hi huh I'm Ruth this is the plaza talk
where I tell you where I'm from what I
do Who I am
I think I kind of just did that so hi
I'm Ruth I'm here today to talk to you
about how to be a web AV artist yeah I
know I know to be honest with you I
struggled with the practicality of this
talk I'm going to be talking to you
about Web Audio API and web MIDI API and
I'm not sure how many of you are going
to go back into work on Monday and put
the web MIDI API into your website so I
kind of thought to myself when I saw
this about six months ago
there likes to take it literally this is
a Venn diagram this is a Venn diagram
practical talks at conferences and on
the left-hand side you've got what
people actually talk about and on the
right-hand side you've got what end
attendees want you to talk about so I
made tool it just to make sure that this
is a really useful talk by the way I
made two lists one list as things that I
have to avoid talking about and the
other list is things that I have to
include in this talk and then it would
be really really useful for all of you
and my talk anxiety will go away so I
have to avoid analytic operational
excellence chat op orchestration that's
fine I think I can totally avoid these
things I have to include duct tape work
around like shedding and coffee easy
things to include for a developer great
now I've got that out of the way let's
begin so what is an AV artist a V stands
for audio-visual so it's an audio visual
artist it's taking some audio and making
visualizations from that audio another
name for this is a VJ there's not a lot
of making the actual audio we are just
sort of taking what other people are
making and using the data from it too
out there remember this
Laura says yeah right oh nice I'm so
glad so those of you that don't remember
because there are going to be a few
people out there that don't remember
this is the best mp3 player of all time
it's lightweight easy quick to install
free but it actually played music that
you had on your hard drive amazing but
that wasn't the best thing about winner
no no no no this was the best thing
about winner inside this visualization
engine so you could play your music and
then whip up these visuals and they
would go in time to the music and you
could just sit and watch this for hours
right this is probably the most
influential thing of any VJ of my
generation and this is kind of what got
me into that I was like I want to make
this this is great so the first thing
that we need to make things like this is
audio and that's good we have a Web
great there's four main ways you can do
it the Web Audio API I'm sure we've come
across some of these you can create
sound yourself so you instigate an
oscillator and this creates a sound wave
and that's going to create you some
sound you can grab it from the Dom if
you have an audio or a video element in
your Dom you can grab the source from
that and pipe it into the audio API you
can stream it it's usually taking it
from the camera or the microphone on
your laptop and you can stream interview
audio api or you can go and get a sound
file and buffer the data from that sound
taking the data and putting it in a sort
of container that the audio API has and
then you can use that data I'm using the
fetch API in this example here to go and
get my sound file so a couple of
examples this one and taking a sound
from a sound file
and just typing it in and it's analyzing
that and this one is actually picking up
on my voice so picking up on the
microphone on my laptop and it's
streaming it into the Audio API this is
my preferred method at the moment
because I don't know where I'm going to
be doing my visualizations I don't know
what sounds going to be or where the
inputs going to be coming from and so we
actually need to be able to analyze
sound oh okay
I'm not allowed to say analytic that was
a that was thank you very much Cooke
thank you good it's going to trip me up
every single time one of these things is
said but I kind of have to talk about
analytics I'm analyzing sound so to
analyze sound we can do this as a Web
Audio API as well and we do this with
the analyzer node so we can just create
one of these and we can pipe in a sound
that we have either created or got with
your audio API and then it will give us
analyze data back and what it actually
gives us is something a little bit like
this which is an array of frequency
velocities for those of you who don't
know what that means frequency is sound
levels like low sounds too high sounds
and what we're actually scanning for
here is not hurt which is very very very
low to 20,000 Hertz so 20 kilohertz
that's very very very high and we can
create as many items in that array as we
want the more items the higher
resolution that we can actually detect
and the velocity value is just the
volume as that frequency at any given
point it's just a number and then we
could just use those numbers which is
exactly what's going on here this is
just a bunch of AI elements and I'm
taking that data value that velocity
value and I'm making it the height of
those eye elements it's really rather
check forward just looping over them and
changing some properties great we can
all make visuals and my talk is over and
we can go for an early lunch
complicated stuff but it is relatively
easy and we're just using HTML CSS and
JavaScript just to do something that's
actually quite cool and so I thought I
wonder how far we can take this
with just these technologies and I
started a bunch of experiments and
everybody knows when you start
experimenting you need some inspiration
so this is my first inspiration this is
Bridget Riley she's a super cool artist
that's on the slides I actually
discovered her at Scotland Jes last year
and went to the Art Gallery in Edinburgh
doesn't massive master paintings shape I
was like this is brilliant I'm going to
visualize her paintings so I did that so
this one it's just a bunch of divs and
it's all just positioned with sex box
and I'm just taking up more or less room
depending on the data depending on that
frequency data I'm going back it's quite
simple and then this is another one
which is just showing and hiding some
triangles based on that data because I
go quiet you will disappear then they
will come back and this is just just
simply stuffing it on and of course if
we're using things in a Dom we can try
and instigate some lovely CSS because
I'm a JavaScript conference so I thought
I should bring in some CSS it would be a
really really good idea
transforms are cool with visualizations
I really feel like you could do some
symmetry if I could take what's ever in
the Dom and replicate it and reflect and
rotate it round just to the function
then I could do that with whatever I
have in the Dom now be great that'd be a
really cool thing to bring intercept so
I did that so I've literally just got
one quarter of a screen and then there's
a function multiplying that and then
just transforming it around and that's
cool and I thought well we can put
anything in the DOM and that's that's
quite good as well because as a VJ you
kind of want to use videos in imagery
you don't just want to be making cool
animations so that whenever we could if
we could just drop a gift in one of
these quarters and reflect and rotate
that around and you can do that as well
this is a My Little Pony gift my
favourite kind of gifts and so on and so
forth are the lovely CSS things that we
have blends and filters
so the visualization that we had at the
top of the show we can just add some
blend modes for that and we've got a
much nicer visualize thank well I think
the first one was pretty good but this
one has just got some blend modes on so
each element is just reflecting through
to the next element top tip for you
filters so filters if you're unaware and
CSS are things like saturation inverting
things like that work on video so if you
are playing video clips in your set you
can invert with literally just a CSS
property which works really really well
it's really good one CSS thing which is
actually really really interesting and
something was great to experiment with
custom properties now we've got a lot of
you've got a lot of data in our
JavaScript but we might be styling with
CSS so there's almost like prayer okay
we can change styles and set in
JavaScript that's fine but this is this
is an interesting one I'm sure everybody
has seen this but I am just going to go
over it if you've ever used a CSS
preprocessor so fast or less or
something like that you probably come
across variables this is almost the same
idea in spec so at the top there we've
got some CSS you declare whatever you
want your variable in the root element
with a dash dash and you call it
whatever you want this level that I
declared in mine is going to reflect the
frequency data which I'm getting from
the analyzer node and then all I'm doing
is changing the width and the height of
the element later on in the CSS and then
in the JavaScript I can pass that
frequency data back to my CSS so I can
do this as document element style set
property and so in real time it can
update my CSS this is JavaScript all up
in your CSS which is odd but can come in
very very handy and again can come in
this this is an example of all the
things that you've just seen you've got
some Dom elements you've got some blend
modes I've got some custom properties
going on
what you might notice is a little bit
janky doesn't really like operating not
allowed to talk about operational
excellent okay well halfway through the
tour for fifty percent who knows which
way this is going to go ok I'm not
allowed to talk about chat up through
orchestration against ok so yeah this is
a little bit yes shanky and I was
thinking about performance this is my
friend Ben it's really really good to
talk to your friends and not just for
rubber ducky when you have bugs in your
code when you're actually considering
the overall thing that you're building
in the best way and all that kind of
stuff is really good to chat them um
chatot I can avoid orchestration can
totally avoid orchestration and we find
this will be a useful talk I promise you
so anyway I was talking to Ben and he
was hoping we would run interactive P
Thursday which is very similar to all
the stuff that I'm talking to you about
and we were going over performance
because it wasn't performing
particularly well and he suggested to me
maybe as to start using d3.js now I
don't usually use libraries and so I
really have to but it did kind of make
sense d3 is data visualization library
and I think a lot of people when you say
d3 think about bar charts pie charts
that kind of thing but it visualizes
data does it very well and that's
exactly what I'm doing I've got
frequency data and I want to visualize
it it's pretty good for data it's also
really good with math and I'm working
with numbers and lots and doing a lot of
math very good with his dong which is
helpful because that was kind of what
was hindering most formats and I can
start use sugs very easily which is
really really nice and actually comes to
the render so that's we can begins
experiments number two this is my
inspiration for experiment number two is
this called super graphic by Tim Liang
it basically charts data from comics and
graphic novels it's really really lovely
visual book and so stuff like this died
happening so this is a basic sunburst
chart but it is picking up on my voice
it's just
making a nice visualization oh it's
coming up on all the screens oh yeah is
it super cool I quite like screens and
projectors you might have you might
guess I have things like this which is
also lovely and there's quite a few so
we can sort of reimagine the spectrum
that we saw earlier and we can have
circles instead and that can be a nice
thing to be mixing into a set when we
actually get to the end and we do that
this is one of my favorites actually
just a nice little circle a little
rainbow colors I want it just for a
second go back to analysis and this is
just before we have lunch engage brain
I'll try and make this make sense if we
take this array of frequency velocities
that we were talking about at the
beginning of a talk there is one thing
about it it goes from naught Hertz to
20,000 Hertz 20,000 Hertz is very high
we're going to hear it but music
probably is not going to get that high
this is a keyboard or piano this is
where it sits on that frequency scale it
only goes up to about 6,000 Hertz
now I'm not accurately trying to
visualize notes all I'm trying to do is
give the experience of a nice
visualization based on music that is
being taped right so I can use this as a
guide and I can think actually that kind
of taught part of the array I probably
don't need that probably not being used
in music that we listen to the other
thing to note about this array is it
goes up in even step so each item is
going to be like an even step of the
last one that's not how notes work this
is a keyboard I've highlighted some keys
a on a keyboard keys are denoted in
letters H G and they start again so an a
in an a octaves apart and we say octave
because there's eight notes in between
there's 12 semitones and the two A's
sounds very very harmonious which we
play them together even though
one's lower ones higher so this is
middle key here this orange key this is
a this runs of 440 Hertz the one below
it the purple a runs at 220 Hertz that's
a two hundred and twenty Hertz gap the
one above it however the green one runs
at 880 here there's a 440 Hertz gap so
the gaps are actually doubling they're
not equal they're doubling so to get a
better analysis from the data that we
have from the audio api we kind of want
to do something like this now slice the
array kind of don't need that data is no
point me having it and then I need to
spread it out yeah so magic function too
close to lunch won't go through it is on
the internet or it's an article about it
it is basically doing that slicing the
array we're doing some averaging we're
pushing all this new data back into the
array so I'm taking a farm school get
I've got seven and a half minutes left
and I've got a couple of example for you
so this is okay this is a visualization
before I do anything on the data and you
can see it's quite heavy towards the
right so this is where the analyzation
star is then right afterwards after
we've run the function we get something
a little bit like this which is much
more spread out and I think works with
better visualization and also it's going
to be good if I want to do anything like
low mid or high frequency detection but
I can't do it on data that actually
represents the music that's playing not
the data that we're getting back from
that's the crux of it you can all go
away and make visualizations now that's
great I did cope ember last November
with all these experiments in there not
the final versions I really use but it
was really good to experiment you can go
on copán and have a look
one last thing checking my timer how do
I control these visuals
that's important I'm going to be playing
set and you probably going to change
sets of DJs and musicians and they're
going to be controlling stuff there's
lots you know decks and things like that
see you don't really want to be sitting
here pressing your keyboard if you can
control visuals with your keyboard
that's absolutely perfectly fine you
kind of want something a little bit
cooler I tried the gamepad API but the
gamepad it's kind of a little bit funky
and there's not enough control with a
gamepad so I settled on this just MIDI
API MIDI is a data protocol I know when
I say MIDI that people think of a TDP
music it's not it's a safety protocol
for instruments to talk to each other so
it came around about 1983 and some
industry experts and music manufacturers
got together because there was a rise of
electronic and electronic instruments in
the 70s and they thought would it be
nice if all these instruments could talk
to each other their electronics they
could probably send data to each other
and so they then they came up respect
MIDI this is a MIDI instrument the big
difference between a MIDI instruments
and what interested actually is this has
onboard audio this is the keyboard
you've probably seen them you probably
played them I can plug this into my
computer and it would send my computer
MIDI data but it also plays audio
standalone it can also talk to all the
other instruments and kind of
orchestrate with those oh we are so
close to the end of the talk as well
useless to look never mind never mind
haha midian stood big difference between
this and what you're probably more
likely to see and want to use which is
this is this is a MIDI controller this
doesn't have any onboard audio but it's
still sending MIDI data so if you plug
this into your computer and run some
code like this which is a web MIDI API
you can pick up on the data it's sending
which again is just numbered and that's
really cool and there's lots and lots of
different MIDI controllers out there and
if you search for them you will come up
with both of them and I recommend buying
them and trying out that code it's
really fun this is my MIDI controller
this is a minim this is my little one
carry around on my
this is actually running by bluetooth to
my machine usually you'd find MIDI
instruments and controllers running by
USB they used to run by add-in which is
an old fat round cable which I'm sure
some of you remember and I can use this
to control my visuals so hopefully demos
our demo gods that could be inclined to
me I can use this to mix it's working in
here but not in there I'm done see if
that one will know it's not happy today
I can use this and mix my visuals and
change them so I can attach all the
different things I made onto the
ones come and set different times and
then I can mix from one into the other
which is great if any of this room kind
of seems silly doesn't it
thank and making crazy visuals in a
browser controlling them with little
boxes of light cards this is me I have a
little pocket projector so I put my
laptop and my MIDI controller and my
pocket projector in a bag and I kind of
packed the bag so that you can unzip it
and have to protect her out the bottom
tray so you can actually play music and
project onto things as you carry it
around and this is me projecting onto
the streets of my local city and I did
this I made this thing it is running the
software which I wrote all in a browser
and some people started talking to me on
the Internet the internet circuit and
things like this happened so there's a
whole bunch of us doing similar things
taking web technologies and building
things with it they build music software
JavaScript they control lights with
nested leaks at the beginning of this of
today whether the guy is Martin's
controlling all the lights via DMX with
JavaScript lives j/s networks you want
to look us up we have a stand by the
community area where we have lots of
lots of MIDI controllers and we have all
of this stuff going on all the music
that you hear between the break that's
and stuff like this happens on the
internet I mean we've got a github
organization gotta be real you've got to
get have organization right that's like
bad a reality right there and we all
play gig a not just together and not
conferences like we actually play gigs
it's crazy so you do my day job in group
a geek
Magnus puts their festival over the
summer okay there is one thing I really
really want to dress before I go that's
it see I I don't think that my talk
really picks up on what was really going
on I'm sure you guys did hey the
keen-eyed among you will realize at
no okay there is duct tape holding the
port will Vijay get together good there
is duct tape holding everything together
then is on a bike yes still more sound
this whole function is a workaround yeah
and drinking coffee thank you very very
much all the music that you heard if I
can era I recommend you look her up on
soundcloud i'm rings john to find me on
the internet come and talk to me over