Cloud Loading Testing in Visual Studio Team Services

hi my name is Charles sterling I'm a
senior program manager on the visual
studio team and I can't believe it's
been a year since I did a cloud-based
load testing video for Kinect since that
year we've done a tremendous amount of
work for you guys
let's take a look at the list so one of
the top requests that we've got is to be
able to collect performance counters
while you're running a cloud-based load
test and we've done that we buy
integration with application insights we
can actually give you those performance
counters while you're running your load
test a lot of requests are making it
simpler to use so rather than actually
opening up visual studio and starting a
new test project we've actually give you
web-based authoring in addition to the
web-based authoring and visual studio
team services we actually now give you a
web-based authoring experience that's
even richer with better Diagnostics in
the azure portal I'm going to show you
that DevOps integration for you to
DevOps shops being able to integrate a
load test as part of your continuous
integration build or your release
pipeline is now super simple I'm going
to show you that one too scaleable we've
done a ton of work on making the service
more scalable so we now actually support
up to a hundred cores and you can run
that in ten parallel sessions so you
actually get up to what is I the
thousand cores across one run will let
you run it up to 72 hours now and you
can run those load tests from ten
different geographies so you can
actually run it from the West US East US
Asia Europe etc and finally the number
one request that we get on a regular
basis I want to build a load test my
application running behind a firewall
with the cloud base load testing so what
we do now is we actually will give you a
trusted IP address and we'll say this is
the IP address that our load test
service is coming from and we're gonna
update the service to actually integrate
with Azure v-net and we're gonna
integrate that back with the Visual
Studio IDE speaking of the Visual Studio
IDE we're actually gonna refresh the
entire wizard right after the begin of
the year so you expect to see that right
after Christmas so lots and lots of new
value I want to show to you so let's go
ahead and start with a demo of the new
web authoring experience in Azure and
that DevOps
release pipeline integration
I wanted to show you two demos of load
testing of new features that we've added
the first one is the new easy-to-use
web-based authoring experience in the
azure portal you can see that I'm in the
azure portal and I've actually pinned a
bunch of tiles for my application of the
home page one of them is from my parts
Unlimited app now what's new is if you
go to the tools menu for this web app
you'll actually see that right here
there is a brand new option called
performance test I want to show you the
new authoring experience so let's click
on new and you'll see right off the bat
it's actually pulled in the root of my
URL that's pretty cool if I want to go
ahead and use a different website
entirely or if I want to take a look at
a view for instance is an MVC
application and take a look at the home
view I can go ahead and overwrite that I
do recommend that you actually changed
the name from the defaults because these
tests are visible both in Visual Studio
and azure so if I want to go ahead and
do correlation or trend or comparisons
from a on-premise run versus a
cloud-based run I can do that from
visual studio but you're going to need
to have a name for that so let's call
this one sprint 3 as I mentioned I can
actually specify 10 geographies around
the planet where I run this from I'm
gonna choose east us 2 and a user load
or 250 well we we decided as my
development team building this app that
we wanted to have be able to support at
least 250 users for sub 5 second
response times so let's go ahead and
leave that at 5 seconds 5 minutes is a
great duration for this run and we're
gonna go ahead and kick that off and you
can see that this one is now queued this
one is in progress so let's take a look
at this actually as it's running and
you'll see that I've got three failures
over 70,000 requests that's actually not
too bad
in taking a look at my requests per
second you can see after it's getting
warmed up we're running at you know 3
seconds with a request per second of 324
in this case that is great that is great
performance I'm really happy with the
team they've really stepped it up it
looks like we've got some issues when we
start so we'll work
we need to actually do some buffering
and some caching and this application
looks like some front loading but
overall you can see that this is gonna
give me the data to figure out whether
or not I'm actually meeting my SLA s and
you can see that it's still in progress
that run is still going and if we take a
look at the one that we just started a
couple minutes ago
once again you're gonna see that is now
just gonna go out and start drilling
this run so you could go ahead and do
run run long running tests and take a
look at them in real time and here's one
that's completed so you'll see in this
case I had zero failures but I'm still
seeing that shark fin right off the bat
so again we've got some issues so that's
the first time I wanted to talk to you
about was that new web authoring
experience with Diagnostics data that's
gonna easily show you if you've got
issues again kind of the smoking gun if
you will now the second time I want to
show you is how easy is it to actually
integrate load testing into your DevOps
pipelines what you see in front of you
is a build that me and Cory Fowler
created a couple weeks ago in our dev
ops webinar if you've got time I do
recommend you guys watching it and
you'll see that for our CI build we are
building our solution we're public
publishing our artifacts for our release
pipeline and then we're deploying it
into a development slot so our
developers can take a look at it you
know what we found is often the
deployment will work just fine but the
web app won't be available because of
some problem something that was missing
I want to show you how you can actually
easily detect that so I'm gonna go ahead
and add a build step for this build go
to test and we're going to choose the
one that it's called a quick load test
or quick performance test and what it's
asking is for that remote agent where
should it publish the results back to I
want to go ahead and publish them into
this account which kind of makes sense
and what URL do we want to test in this
case we would probably take a look at
whatever was being deployed here so we
are deploying the web app WW awesome too
so we want to go ahead and do HTTP colon
forward slash four slash W awesome two
dot
sites dotnet so it's gonna go ahead and
run that quick performance test against
there again what we're mostly interested
in did it come up are we getting any are
we getting 404 is over 505 100 errors
saying that it's not about running and
let's go out and call this one avail 1
ID at this point if it's not available
we probably do not want to continue on
air so we're gonna leave that checked
but we do want to enable that so easily
are an easy way to for you determine
whether or not your web app deployments
are actually successful is by adding a
quick performance test in addition you
will probably want to go ahead and do a
formal load test or performance test to
see if you're meeting your SLA is like I
showed you before let's go ahead and go
into the release hub for visual studio
team services release management and
edit the web camps release pipeline
that's the one that me and Corey created
and you'll see that it's actually got
the three canonical environments you
know your dev your staging and your prod
in staging we want to know for hitting
our SLA s so we're gonna go ahead and
add the tasks go back to test go ahead
and select a full-blown load test and
the difference is instead of specifying
just a simple URL we're gonna go through
multiple URLs defined in a web test and
I can do additional things like use
plugins do data binding etc so same
question right off the bat where do we
want to publish those results and I want
to publish them back to this visual
studio online account and it's asking me
where are my local test settings file
now in this case I have actually
published those as part of that
published tests that you saw earlier so
I'm gonna go to my drops I'm gonna go to
my my version control or my my build
output I'm gonna find the local test
settings and this is actually gonna do
things like say should we run a batch on
startup
what are we databinding to so that's
what the local test settings is going to
reference then it's saying where did you
put your web tests and once again it was
published as part of that drop
so I don't have to remember to type it
all in I just reference it and it's
asking me what is the load test file
that we want to run you might have
multiples in that same directory I
recommend you do not call it load test
one but in this case I actually called
it home page and in this case you know I
probably want to go ahead and say
continue on air if I'm not meeting my
SLA s I want to be notified about it but
I probably do want to go ahead and if I
have an approval stage to go into prod
let the approval stage actually get
notified so let's recap I can use quick
load test to go out and take a look at
whether or not my CI deployments were
successful and I could do a full load
test into my staging or performance
environment to see if I'm hitting my as
SLA s
I hope you enjoyed the demo as you can
see we've done a lot of work on the
cloud-based load testing I'm just
walking through the list one more time
integration with application insights
easy to use web-based authoring with
Diagnostics DevOps integration with a
simple adding of a task integrating load
test into your releases and into your
builds scalable hundred cores ten
locations in 72 hours ten parallel runs
that's a lot of scale and finally load
testing behind a firewall using IP
addresses and soon with vina integration
one thing I didn't talk about is if you
haven't checked the prices and we
actually have made it much more
affordable for you to get started with
cloud-based load testing I'm Charles
sterling and thank you very much for
joining me