Combining Dev and Test in the Org

so I've given this talk not you know
this exact talk but a version of this in
a quality discussion with customers and
you know we talked about how things have
changed in the cloud cadence and the
combine engineering and ship flap and
you know at the end of the presentation
they say wow that seems pretty drastic I
don't know if you are ready to do that
and the reality is that you know our
testing approach at Microsoft has been
evolving for quite some time I would say
for more than a decade so before I go
into what happened during the cloud
cadence I'll just give you a quick
history of what happened before that
because the customers that you might
talk to they may be at different phases
in this transformation and you know I
won't spend too much time in the h3 but
hopefully that will give you a sense of
how to kind of guide the customers
through this in case they are somewhere
you know prior to where the cloud
cadence happened in bsts
so you know I'm going to take you back
to 90s you know so for as long as
Microsoft shipped product we we always
had three distinct disciplines in in a
product team you know PM Devon Tosh PM's
gather customer requirements wrote specs
dev wrote code and design code and test
road tests roughly we would have a 1 is
to 1 is to 1.5 ratio it kind of varied
by teams but that's the general ratio we
used now within test we are two distinct
disciplines now many people may not know
this but this was a unique set up at
Microsoft where within test we would
have a software design engineering test
these are the s debts who developed the
automation the test infrastructure etc
and then software test engineer or the
STS who ran the automation or who ran
manual tests and this is a key point the
software design engineering touched we
are hired you know by the very similar
qualification as these of design
engineers or the developers you went to
the same colleges and if you hired from
industry would pretty much higher
developers and then convert them into SD
it is you know remember this point where
I come and talk about the combine
engineering because
this is this is important in in
particularly the way the test discipline
was set up at Microsoft so how did it
work well it worked reasonably well back
in the days you know we achieved
commercial success with big products
like Windows and Windows and Office one
of the benefits of this model was that
when we are ready to do a product
sign-off you will have the quality
discipline of the test discipline bring
in a very formal sign of criteria and
and formal measurements in quality and
so that give us a pretty good confidence
in you know declaring a product ready to
release it also developed deep expertise
in testing because the test discipline
was solely focused on testing they were
thinking about this day in day out so
that was the great thing but did it
really work though and the answer is no
it did not work there were problems the
problems were simply masked by the fact
that a we had commercial success of our
big products and B there was a long
product cycle but there are numerous
problems you know so the developers just
through the core over to the wall to the
testers s debts the s debts road
automation and then through that overall
to the ste the the software test
engineers so and these ste is the way
they responded is by just keep adding
more and more STS particularly the
vendors there was really no growth
opportunity for them that ste is because
they really didn't have any mobility
they couldn't go anywhere it was very
expensive to maintain this this set up
and testing became water-like and cause
product delays but again we couldn't see
some we didn't feel as much because our
product cycle was long we've shipped
Windows every two years or three years
so this sort of worked by V around 2000
late 90s it became very clear in the
company that this wasn't working and we
had to change something so a company my
decision was made to get rid of the STS
from the test discipline VRS debts and
STS I've no more STS they are gone
and we did that it was actually very
painful because those STS remember the
STS didn't have the same qualification
as the S debts and so a lot of them we
try to find them new roles in the
companies you know if some of them did
but many
many of them didn't so this this sort of
improved the model a little bit in the
sense that now you have s DT is one you
know responsible for not only writing
automation but operating the auto
automation so they they own the whole
thing so they were naturally incentive
as to a right good automation be you
know right more automation instead of
just throwing tests over to the another
team to take care of just running the
test they were now responsible for for
for it all but the core problems still
remain the developers would you know
through the code or wall to this s
that's a step are constantly trying to
catch up so we got clever and we said
you know what we're going to introduce
the thing called quality milestone or an
mq now these are milestone you
reintroduce or will have after a product
is released and before the next product
is about to start will block off a
certain period of time and say we you
know whatever the quality debt or test
that we've accumulated in the previous
release will just catch up and fix it
there now clever idea but it didn't work
it didn't work in practice
for just a couple of reasons one is that
now people knew that there was a
milestone coming up called quality so
they were just before the quality over
to that milestone and the other issue is
that your milestone dedicated to quality
so people would conjure up all kinds of
quality initiatives and things that they
think is creative inside the quality
realm and try to you know schedule that
work cause priority inversions and
sometimes that what didn't get done and
we just accumulated more more debt so a
clever idea that didn't really work so
the test was still a bottleneck but we
again we survived because we are in this
waterfall waterfall world
then came the cloud cadence so that I
have all of the cloud cadence around 20
2008 timeframe 2010 and it brought new
pressure on the system now there is
expectation that the you know we are
running a much faster faster cycle and
the expectation just continued to
increase you know faster faster faster
we long you know the gone are those
longer stabilization phases we don't
have
the opportunity to create a betta and
give it to customers do dog food you
know so those kind of validation phases
are gone which are crutches in the past
but they are now gone you know you're
living in the world of micro services
these micro services are deployed
independently so there is pretty much
pretty significant complexity in terms
of getting those services right the
quality right on an independent cadence
you know talked about how we had to
support no downtime deployments these
services need to stay up all the time
and so so what did we do well we knew
how to ship software for last 25 years
so we said well just use the same
approach just try to do it faster you
know if we went from two-year cycle to
six-month cycle to three-week cycle and
just try to figure out how to do
whatever we knew just do it faster so
our initial approach was same model run
faster he pushed for kind of getting
automation more streamlined and we got
clever again and we said oh what are the
ways we can deal with this is that we
don't need to run all the tasks guess
what you know we can be very smart about
which tests to run we'll pick some tests
here and pick some tests there and
that's how we'll survive but it was just
a matter of survival
it became very clear to us that the
model wasn't working and so the you know
we started seeing all kinds of issues
you know testing was a major bottleneck
by this time particularly bsts I think
bill or somebody mentioned that we had
some sprints where we do a three-week
sprint cycle we finished that then we go
through another three week of
stabilization by the time is a sprint
got deployed would take another three
weeks and this dead start running around
trying to stabilize the system and
deploy it the meantime though then the
work on the next sprint is already
completed and and so they're just you
know they're trying to catch up and in
the cycle would continue you know same
issues lack of accountability on the day
of the short the short version is that
we recognize that this model wasn't
working and we in fact we were not the
first one to recognize it there were
services before us like Bing was one of
the major services at Microsoft that
that saw this
and we we started seeing observing this
is you know based on the practices some
of the companies born in the cloud they
were following in the industry so so we
we we knew that maybe we needed a new
model in the in the cloud cadence so
that's what we we get to this point my
rest of the talk is about what happened
in the cloud cadence I just want to give
you a flavor of what happened before
that because you might run into
customers who probably still have the
world where you have Ste Zoo are running
around writing a running manual test and
there is not a lot of emphasis on
automation so you have to kind of bring
them along in the journey before you
kind of talk about some of the other
stuff you know I'll walk you through so
what happened what happened in the cloud
cadence in you know this is sort of
pretty much sums up the three big things
that we changed we changed the quality
ownership we fixed the quality
accountability so that's number one
the second thing is that we understood
that in order to ship frequently out of
a release branch you need to have a
master that is also in a pretty good
shape it's in always a shippable state
you know you see you saw Bill talk about
how you work in the master and then the
release will be released it's the
quality is not just about getting the
release branch right it's it's actually
quality you know starts in the master
branch and keeping it in a suitable
State now that you know the statement
about a lot of things sort of the code
flow and you know sort of how the branch
mechanics top that we'll talk about but
from testing perspective we focused on
two things one is this concept of shift
left shift left testing and I'll talk
about that in a second and then the
second thing was energy kicking it a
little getting rid of all the test
flakiness in the system the other thing
that we understood is that there is no
place like production this this is a
this is sort of I would call this the
shift right part of the strategy so on
shift flat you know run tests close to
the code run more unit tests to me ship
Fridays you know run tests close to
production because there is no place
like production and it's a set of
practices about
sort of both safeguarding the production
as well as ensuring quality in in
production so we you know even sense we
got rid of the the testing that was
happening in the middle you know sort of
the your integration style testing
functional testing that used to happen
in the lab that was the big departure
here all right so I'm going to walk
through each of these concepts in a
little bit more detail quality ownership
so though we did we did combine
engineering you heard this term before
you know we've talked about this
combined engineering in a nutshell is
you know those two disciplines Devon
test two roles taking those two roles
and merging them and putting it on a
single discipline single role call an
engineer so we got rid of the two SDNS
that roles just one role engineer the
key thing is that when we did this that
there is so first of all there is a that
individual has a combined responsibility
for both dev and and test so and it so
it's not just an organizational change
where you bring the dev and test him
together
it's an actual discipline merge you know
if you think about the set of
qualification or requirements of SDE for
set of qualification required per se
duties you merge them into a single set
that that's what this was and so
everyone had to learn new skills a lot
of times when I talk about this the
first question I get is this so what
happened to those s Nets they learn how
to write code well the reality is that
you know you remember the qualification
I mentioned earlier they knew how to
write code they got a little bit rusty
in terms of their design skills but this
was also a learning for the for the
developers because developers now have
to learn how to write tests write
automation run - you know do manual
testing do exploratory testing you know
things like that
so this required learning on both sides
and I think that's a key point when a
lot of times people talk about combine
engineering they say oh ok that means I
need to train my testers to be more like
there's no no it actually goes both ways
the other key concept here was that we
you know the idea behind this is that
you want to reduce handoffs there is no
you know in a short cadence you don't
have the opportunity to start somewhere
you know write your code give it to
another team to test and then give it to
maybe another team to do performance
testing and give it to another team to
do deployments the basic idea was that
we wanted to reduce handoffs in the in
the team and give an end to an
accountability to a feature team tour to
an engineer inside a feature team so
this was a big cultural shift across the
company this change happened in one team
but then it over few years every team
across Microsoft changed now different
divisions took a slight different
slightly different approach to doing
this in some cases like us in bsts when
we did combine engineering it was pure
like you know we just merged the two
disciplines there is no other team that
is responsible for quality every feature
team owns its own feature area featuring
equality so in some other orgs they they
had you know they still left another
small team and to kind of look after the
live site telemetry or live site
instrumentation you know things like
that but ultimately if you kind of
fast-forward now just about all teams
that Microsoft follows this model how
did we how did we make this transition I
think this is this is important thing to
talk about because it like I said it's a
pretty drastic change there our first
transition that I talked about where we
got rid of the SD rolls was very painful
and we had learned from that a lot so
when we rolled out this change
particularly bsts
fortunately for us there were a couple
of other teams that had done this at
Microsoft so we went and talked to them
we learned from them we had a lot of
discussions in the org kind of getting
the team ready to do this one of the
things that we were very concerned about
was that you know there is all these
things that
test him does some of it what I
described at the time is dark matter
like nobody understands what they do but
they do it and somehow that's magic
happens and the in the right quality you
know happens at the end
so we meticulously went and inventoried
everything that the test team does it
was a spreadsheet giant spreadsheet I
forgot how many rows but there was rows
of like we you know it's not just like
we done automation or we write
automation it was all the little things
that the test team dead to kind of keep
track of quality in the org and we made
sure that all those responsibilities
were reassigned to somebody in the org
basically to these new roles that was
that was very key the second thing we we
were very clear about is that this is
not just changing roles and
responsibilities we are going to have to
change the way we test period if we
continue to test the way we were testing
before it's not gonna work in this new
world so this is where you know I'll
talk about the shift left testing
testing in production those concepts
were not only internalized but practiced
in the org and we give ourselves about
twelve months to go through this
transition now remember when we when we
did this six months later we were
shipping TFS 2015 so the the litmus test
was getting the quality right for TFS
2015 so we we said we will give
ourselves about twelve months it means
in during that time the the SDS and as
debts will will start off kind of
basically in their old roles but slowly
evolved into doing the combined
responsibility so you have a you may
have a feature team within that the the
people who were former as debts they
continue to do more of the S Network and
the former SDS continue to do more of
that SD works but sprint by Sprint the
ratio kept changing and eventually you
know after six months you cannot
recognize it there from the tests in the
org so that's kind of how how we managed
it now at the end of the transition
there were people some people who didn't
quite make the transition
and you know that was that was the sad
reality but we we supported the
transition through training through sort
of just the development of the new
skills letting people practice practice
sprint after spinning after sprint and
so kind of just giving yourself a more
practical timeframe to go do this is key
well I think feel free to ask me
questions otherwise this will yeah go
ahead
so in this new model where everyone on
the team is an engineer and how do you
take the responsibilities that were
previously spread across the quality
organization and when he ate
responsibilities on a team where
theoretically everyone has the same
skill set of responsibilities I'm
wondering how that how the division of
labor actually occurs is all right now
when I cook on the team I'm on for
instance our test automation occurs with
different engineers and they're
automating to test but there and they're
writing code but they're not writing
features right and it's on the same team
so theoretically we're doing this too
but it seems like it's different from
what you're describing - yes it is
different and that's I think it's
important thing to clarify in the
beginning it looked like what you just
said so think let's take a particular
feature team in the world old set up we
had five developers maybe five testers
and that constituted a feature team we
bring them together under a single
engineering manager so now that any
remain manager has ten engineers working
for him responsible for the same areas
sprint one after combine engineering
happened it probably looked very similar
to what you just described that the the
the the tester was still spending most
of time developing tests the developers
were spending most of the time
developing code and design but the North
Star was clear the North Star was that
an engineer who owns a feature owns it
end-to-end they can take lot of help
they can get lot of peer reviews of the
of the of their test plants of their
design of the
their telemetry they can get in fact
they were encouraged to get a lot of
help in terms of peer reviews but the
expectation was that the next sprint
guess what they will be the one writing
test automation for the feature that
they own maybe they start with a small
feature that where they do that and and
the same is true for the tester the
tester started picking up small features
of the backlog and they said we'll own
these features end-to-end all the way
from design phase to deploying to
production and and then monitoring into
production so it started off like that
and then over time we expected devs to
pick up more and more the tesa
sponsibility and vice versa you flip
roles at times also and and that's how
in that's that's why what I mean by
allowing the team the 12 months duration
to sort of transition into this new
world you speak to a little bit about
what happened to like your team's
velocity and particularly the
development velocity then producing some
business value did that suffered during
this transition period and you guys feel
like you're back to where it was yeah so
on the velocity I don't know if Aaron
showed you a chart that showed sort of
our feature one of the ways we measured
a Pilate was the velocity was just
number of features delivered in a every
year on average per sprint and if you
look at that chart it's it's constant
it's been constantly going up since 2012
I believe we've been tracking and now
2017 so the short answer to your
question is no with the feature velocity
did not did not drop because remember
you still have the same number of
engineers in the future team you just
took the two separate teams you put it
together yes you're spending little bit
more time in terms of learning and
development and training sort of new
skills but there was also an efficiency
gate through this process and that is
the key efficience again is that you are
not handing things off to another team
when you hand things off to another team
guess what happens this is context
switch this is like one thread waiting
for the other thread to complete and
then then it has to pick up the con
and kind of run run again that constant
back input that is to happen between dev
and test that's gone and so you you gain
quite a lot so the longer and you
absolutely gain velocity you absolutely
gain more capacity in the short run you
could argue that hey there is there is a
some period of training and learning so
you so but it's it's it's a good
investment in just building out the
rounding out the skills and you'll see
as I talked about in a second the change
was pretty profound across New York it's
not just by feature team we got rid of
in fact I can talk about that now we we
got rid of basically this notion of
specialization there is you there are no
handoffs
you don't take a feature you write it
you design it you give it to another
person to test it then you give it to
another person to deploy it maybe
there's another person like we'll talk
about is a branch mechanic whose job is
to push code around maybe there's
another person whose job is to make sure
the product is ready and it's got the
right performance metrics like all this
different there's another person who's
testing the deployment configuration
testing you know things like that we we
took the core principle that there are
no central teams there are no
specialized teams that do certain tasks
that was a core principle but at the
same time we understood the importance
of specialization specialization is
important creating a central team where
you hand things off to in a fast cadence
is is a problem so we didn't want to
lose specialization so we did form a
bunch of V teams and I know what
deliberately call them V teams because
these are not dedicated teams the right
now in pranks or there is only one team
you could call that a dedicated team
that does you know sort of the EPS team
or the team that runs our central
engineering system it's a small small
feature team but they again they do core
engineering work for the they they're
contributing to the engineering system
the nobody is handing things off to
their team so set of V teams we form one
of them
was best architecture we team now this
was a new new thing we didn't have this
before we had an architectural we team
that looked after the product
architecture we didn't have anybody
looking after the test architecture and
remember we we learned that we knew that
we had to change the way we test which
means we had to rebuild our test
infrastructure we had to rebuild the way
we ask our tests from the ground up so
we we picked our senior-most engineer in
fact bill you know it's a partner i see
in the orc senior-most i see he said
you're going to lead this team and he
had a set of other engineers from across
york part of the v team and this team's
job was to like I said it not only build
the next architecture for - but champion
set of practices that we were talking
about yes this question can you please
explain the concept of each team via
team means virtual team so these are
members from different parts of the
organization okay it's not a dedicated
team reporting to a single manager
that's what I mean
we had tests sorry tenets champs V team
so what are ten attempts so these are
people who are looking after your
subject matter experts who are looking
after some specialized activity that we
do in - whether it's making sure the
product is accessible whether it's
making sure product has good performance
and reliability it's its global ready
you know things like that this used to
be again you know largely be done by the
by the test team in the past in the new
world we refactor this responsibilities
again every feature team is responsible
for making sure that their feature is
accessible is performance is global
ready but we we would have a we team of
experts from throughout the
organizations whose job is to build deep
expertise in this type of activities
this type of work
so the subject matter expertise is still
valued in the org specialization is
still valued in the org the the main
difference is that it's not you know
sort of consolidated into a
dedicated team I mentioned performance
we team you know this is important
because when you're looking at service
performance product performance
oftentimes you find bottlenecks in let's
say you are a and one of the top-level
feature and you're doing performance
testing for work item tracking and you
find water like somewhere deeper in the
system which is owned by somebody else
so we formed a performance V teams job
was to identify common bottlenecks
across the entire product and come up
with the right design solutions and
drives that you know this kind of work
you cannot just farm it out to
individual feature team because the
performance is an end-to-end end-to-end
problem is not isolated to a particular
layer of the product of the product
couple of other B teams be marred don't
even ask me what to be more stand for
because I'd right here at the moment I
may not be able to figure that out but
it's B mods are the people who look
after our daily build health and the CI
health and these guys are constantly
watching the builds and the runs and if
there are any failures in the in them
they do a quick triage and assigned to
the appropriate owners so we formed a
beam or team and and you'll see that
over time the the size of the beam or
team also shrunk as well as the what we
expected be much to do also shrunk as
the system and the engineering system
got better
finally we retained a small vendor
vendor we team that own sounds really
hard to automate type of tests he knows
like config tests you know we TFS being
deployed on on Prem different
configuration environment but again over
the last three years this team has
constantly shrunk because we every year
we ask the question why do we need so
many vendors who need to do this manual
testing let's go automate that or let's
figure out a different way of running
those tests so that's that's the end of
sort of what happened in terms of
changing the quality ownership and
bility so good question about what
tenant champ sweetie my donors in the
word tenant I really do yeah so tenet
means
so performance is a tenant accessibility
is a tenant like an aspect yeah aspect
of the product there you go yeah an
attribute of the product yeah
quality here to the other question was I
heard from Scott good three in a person
take some time ago the move to
everything being done through the
command line
did that help in automating some of
those aspects that the vendor team was
having to do manually no vendor team is
actually you know there it today our
render teams doing things like we have
TFS on Prem and can be deployed on so
many different configuration that we can
automate that in theory it will require
a significant amount of investment and
you know for things like that where the
cost of automating you know
significantly more than you know kind of
cost of just running it through
extensive the questions so it's it's a
matter of trade off that's right
initially it was a matter of survival
because remember we came from a world
where you're half the team that is
basically running tests and writing
tests to world where suddenly that
responsibility is is kind of you know
distributed out to the org and so
initially just you know as we
inventoried the whole list of things
that the test team was doing and we knew
that there was a good chunk of testing
doing this manual testing and even then
we had the vendor team even test him
used to retain a vendor team that would
run this kind of hard to automate tests
we didn't want that to drop on the floor
the creaky corrected are going through
this was we are shipping TFS 2015 in six
months it needs to be as good quality if
not better then it was in the previous
model not only that we are you know
shipping whenever you would continue to
ship to the cloud every three weeks so
in in going through this the key
criteria was that nothing should fall on
the floor nothing should go slip through
the crack so even if we were doing
something that was not
optimally design or efficient we just
continue to to run that in the new world
until we figured out a way to do it
better doesn't make sense the initial
thing was take whatever we have just
refactor give it to different set of
people meaning give it to the feature
team but don't drop it on the floor even
if it looks questionable like why are we
why are we running this test it's not
adding any value just keep running it
for now until you figure out that there
is there is a different way to do that
yeah question so if I understand well
these are like will CH all teams does
this seem that these people have other
assignments and if this is so was the
ratio between their capacity in these
assignments and some other things that
they do yeah so these are the same
people that we have in the feature teams
these in and so let's take a specific
example let's take accessibility for
accessibility we have subject matter
expert by location so in Redmond we have
two people who are some accessibility
except export and we have another couple
people in North Carolina our locations
and maybe another couple people in India
they they are deep expert in
accessibility techniques philosophy etc
but they are engineers inside a
particular feature team they just happen
to have this secondary responsibility
accessibility happens to be one of those
tenets that requires quite a bit of it's
not work but there's a quite a bit of
responsibility on that subject matter
expert so the person who's the
accessibility champ probably spends half
their time doing accessibility and half
the time doing feature work but the idea
is that that responsibility also over
time rotates so it's not the same person
every single sprint we pitched it to the
same person for a given release so TFS
2018 there's 1% maybe TF is 2019 we'll
try to give that respond you to somebody
else so nobody is doing this for for
life if you will
in the number of experts vary by by the
tenant that we are talking about I
performance via team I think it has
about dozen people yeah the question so
in this model I see that now individual
is probably taking care of many things
now one who did testing had to take care
of so many tenants and things now as a
developer he has other responsibilities
so I imagine you manage that by maybe
making smaller features and things like
that
did you have any challenges with
entering coverage because now my
responsibility as a little is even
smaller and how did you manage that end
to end were there any gaps that were
unveiled by these things so if I
understand your question correctly so
yes you know it it appears on the
surface it looks like your engineer is
now doing twice the amount of work you
know feature team because you know the
previously you have I'm our dev our
tester who's paired up with me and he's
doing half the part of the feature work
or the testing work now I am responsible
for the feature but remember now I have
if I am the manager of the feature team
I have twice as me you know sorry I'd
pass as many engineer size as I had
before you know so the net capacity
hasn't changed net capacities are still
the same what I give you more time one
of the two right you know yeah
[Music]

Comics fight each other with jokes at Laugh Battle Royale

>> Let's play. Ready?
Where do baby cows eat.
The Calf-eteria.
>> What's the sole purpose
of middle names?
So kids can tell when
they really in trouble.
>> Yeah.
>> I'm just excited to
be part of that future.
>> Oh man.

Comics rely on Microsoft AI to win Laugh Battle Royale

>> Seth Herzog is here
and I'm ready to get crazy in
the laugh battle here at
the National Comedy Center.
>> But the AI technology.
I'm just excited to be
part of the future.
>> Let's play. Ready?
>> Happiness detector.
I've been waiting
my whole life for that.
>> Where do baby cows eat?
The calf-eteria.
>> Now, we have for round two.
Janelle James versus
the very bearded Nick Thune.
>> The AI won't pick
up any laughs from me
because they don't call me
cold blooded for nothing.
>> I'm Nick Thune and I
want to thank Microsoft.
Because I'm going to win.
>> Very close battle.
They blocked each other,
they laughed at each
other a little bit.
I think Nick's beard kind
of helped him.
>> I'm Jordan Carlos,
Corinne, I like Corrine.
>> I'm going to have
a great time doing this.
I went to the dermatologists
recently and she was like,
"Your skin looks so young."
I was like, "Yeah,
I've never smiled."
>> In our championship finale,
the two very funny men.
Try not to laugh at other people
because they're
very self involved.
>> I lost three days already.
>> Nick Thune just won
this round of laugh battle.
>> You need a fix.
>> We are number one, all of us.

Comics go head to head to head at the Laugh Battle

>> Seth Herzog is here
and I'm ready to get crazy in
the laugh battle here at
the National Comedy Center.
>> But the AI technology.
I'm just excited to be
part of the future.
>> Let's play. Ready?
>> Happiness detector.
I've been waiting
my whole life for that.
>> Where do baby cows eat?
The calf-eteria.
>> Now, we have for round two.
Janelle James versus
the very bearded Nick Thune.
>> The AI won't pick
up any laughs from me
because they don't call me
cold blooded for nothing.
>> I'm Nick Thune and I
want to thank Microsoft.
Because I'm going to win.
>> Very close battle.
They blocked each other,
they laughed at each
other a little bit.
I think Nick's beard kind
of helped him.
>> I'm Jordan Carlos,
Corinne, I like Corrine.
>> I'm going to have
a great time doing this.
I went to the dermatologists
recently and she was like,
"Your skin looks so young."
I was like, "Yeah,
I've never smiled."
>> In our championship finale,
the two very funny men.
Try not to laugh at other people
because they're
very self involved.
>> I lost three days already.
>> Nick Thune just won
this round of laugh battle.
>> You need a fix.
>> We are number one, all of us.

Committing code changes (2 of 5) Getting started with GitHub

on this episode of Visual Studio toolbox
we're continuing our mini series on
getting started with GitHub we're going
to see what happens when you actually
write code
[Music]
hi welcome to visual studio toolbox I'm
your host Robert Greene and this is
episode 2 in our five part miniseries on
getting started with GitHub in the
previous episode we looked at how to
create a project in visual studio and
then we created a GitHub repository with
that code and in this episode we're
going to start writing some code and see
how we work with GitHub
so I created a simple console
application we have a simple line of
code here let's make a change to it so
I'm going to change the greeting to
hello Visual Studio toolbox
now get is watching locally and I can
see that we have a change so if I come
down here to the lower right and I look
at this little pencil in the status bar
it says one telling me I have one change
I click that and I go to the get changes
window where I can see all of the
changes I've made and I can see that
program.cs has been modified well from
here I can undo the changes if I want I
can also stage these or stash them which
means I'm not ready to commit now but
I'll commit them later on but in this
case I think I'm ready to go so I'm
going to enter a message which is
required when I do a commit I have to
enter a message which is a good thing so
I'll just say changed greeting
and then I have some choices I can
commit all commit them to my local
repository which does not send them to
GitHub I might do that if I want to uh
if I'm done working for the day and I
want to make sure that I've kind of
backed up that code committed it to the
repository locally in my machine and
then when I'm ready to send it up to
GitHub I'll do a push so I'm going to do
this right now this is one of my
favorite features you have to save your
files before you commit I almost never
remember to so like Visual Studio warns
me so I'll save this and commit it and
what that's going to do is copy these
changes up to the master branch in
GitHub so successfully pushed so not
presumably we can pop over to GitHub
and look in the code and look in program
and there's the change I made very cool
now what we can also do in GitHub is
click on this changed greeting and see
what the changes were oh I changed it
from world to visual studio toolbox very
nice
now if I come back into Visual Studio I
know I have this change here what I can
do is again go to our Branch history and
see that I changed the greeting so again
Robert is me locally Robert Green are
any changes I make on GitHub
representing another developer or me on
a different machine and then I change
this locally and if I double click on
this I can see the changes so this is
the code that was this is the code that
is I can switch around from side by side
mode to inline mode whichever one works
best for you
okay
so now
I'm not the only one working on this
application turns out
so
maybe the other developer also decides
to change the greeting and I'm going to
do this in GitHub just to keep it simple
typically you'd probably do this in
Visual Studio or Visual Studio code but
rather than have multiple visual Studios
floating around I'm just going to make
all my changes in GitHub
so I'm going to change this to visual
studio developers I mean commit the
changes
commit message defaults to update
program name and extended description
I'll type changed greeting and I'll
commit this to the master branch
okay so another developer is made a
change
now back in Visual Studio
I'm told hey somebody changed your code
now I've got a couple choices I can do a
pull which will bring down those changes
in overwrote what I wrote but maybe I
want to see what the changes are ahead
of time so I can do a Fetch and what a
fetch will do is bring down from GitHub
the latest version of the master branch
and store it in my repository history
but not overwrite my version
so now I can see what the differences
are so if I click here I see I have one
incoming
and I can see that
program.cs was updated by somebody on
the server I double click on that and I
can see
the code that I have Visual Studio
toolbox and the code it was changed to
visual studio developers and I'll decide
okay that's fine I'll take that so now I
can do a git pull and copy down the
remote version to my local version
and now if I go look at my program.cs
I've got the latest version so this is
how I can keep my local version of the
repository in sync with the remote
version up in GitHub which will be sort
of our official version and then if I
come into the history and do a refresh
here I see that I changed the greeting
locally somebody else changed it on the
server I did a poll to bring everything
down and now my version of the code
locally is in sync with the version of
the code remotely that's basically how
we're going to have multiple people
changing code and everybody then pushes
it up into the remote repository which
is the official version and then by
polling we can keep our local copy in
sync
so what we've seen in this episode is
what happens when we start making
changes to code so we made a change we
pushed it up to GitHub somebody else
made a change they pushed it up to
GitHub so we saw you can have multiple
developers or you working on multiple
machines or multiple versions of Visual
Studio it all works how we can keep our
local repository in sync with the remote
repository which is essentially the
official version of the code
if you're watching on YouTube and you
like this please like us share with your
friends and come back for episode three
in our mini series we're going to look
at working with branches and see what
pull requests are used for
we'll see you next time on Visual Studio
toolbox
foreign
[Music]

Compliant Cloud Computing

Okay, great.
Well first off we just
wanna welcome everyone for
coming to our session today.
Thank you very much.
My name is Dennis Garcia, I'm
an Assistant General Counsel for
Microsoft.
I'm based in Chicago.
I lead Microsoft's legal
support function to our
US Central Region Enterprise and
partner group team.
We're gonna have a conversation
today about compliant cloud
computing.
I will let my fellow panelists
introduce themself individually.
Edward.
>> I am Edward Efkeman,
I am with Fed-Ex.
I am a director of compliance
within the legal department.
So we have a compliance
department but
it is within
the legal department.
My responsibility is somewhat
outward facing customers and
data privacy.
Data protection is within
my area of responsibility.
>> Hi, I'm Darryl Hobbs.
I'm also at Microsoft.
I support what's known as
Microsoft's mid market business.
So I like to think of it like
we have our top customers, and
then our mid market,
which is everything else,
about 6 million customers
that we sell to.
So it's a breath business,
I sell across almost
every industry.
So I see a lot of different
things on a daily basis,
and happy to be here.
>> My name is Jude Soundar.
I work for
the Army General Counsel.
I'm an ethics attorney and
my profile includes financial
disclosure and social media I'm
also the program manager for
Financial Disclosure and
Management.
It's the main application
that DOD uses for
financial disclosure.
Basically all senior
officials in DOD and actually
government must disclose their
assets and liabilities so
we can do a conflict analysis
to make sure that there's no
conflicts with the DOD and
the work that they are doing.
I'm also an Army reservist,
a major in the Army Reserve.
And I'm a JAG attorney also and
I do work around
conflict minerals and
other issues like that.
>> Well, great, thank you very
much guys, I appreciate it.
So in terms of a format today,
we thought what we'd do is,
recognizing that folks may
have different degrees of
understanding as to what
cloud computing is all about.
We wanted to take a step
back and provide a basic
overview of cloud computing,
what I call a Cloud 101.
So I'll provide a 10 to 12
minutes overview of the cloud,
if you will.
And then we'll switch gears and
we'll open it up to various
questions to the panelists.
And we want this to be
highly participatory.
So to the extent that
folks have any questions,
don't be shy in answer,
asking your questions.
We're here to address any
questions that you have.
So feel free to chime in.
So what I'm gonna give
is sort of a layperson's
overview of the cloud.
Of course I'm a lawyer,
I'm not an engineer,
so I'm not gonna get into
the tactical details per say.
So this is gonna be more
of a cloud computing for
compliance professionals.
And I think a great place to
start when you're thinking
about the cloud is to realize
that whether we like it or
not, whether we realize it or
not we're all using the cloud
each and every day as part of
our personal lives, right?
After all, many of us have
been using web hosted email
since the late 1990s, right?
To send message use to
friends and family.
I know when I started using web
hosted email Was it was through
an AOL.com account,
I moved to a Yahoo account.
Of course now I'm
working at Microsoft,
I have a live.com account.
But all of us are sending
emails each and everyday, and
that is largely powered
by cloud computing.
So many of us have smart
phones nowadays, right?
I know I have a smart phone,
I use my smartphone
probably too much.
A lot of that data which is
generated though our phone is
stored in the cloud and
of course,
many of us are using
social media.
Each and every day.
I'm a big fan of Facebook,
I use LinkedIn.
I'm a big fan of Twitter.
Social media is largely
powered by the cloud.
So the cloud really
isn't anything so new.
And it's become more and
more a ubiquitous part
of our everyday lives.
In terms of a definition
of the cloud.
Like a lot of technologies,
there's no one uniform
definition of cloud computing.
There's a variety of definitions
but the National Institute of
Standards and Technology which
is part of the Department of
Commerce, and of course they're
only a few blocks away.
Back in 2011 they put together
a white paper with a pretty
involved definition of cloud
computing which has become more
of the defacto industry standard
definition of cloud computing.
I'm not gonna cover
that definition.
It's a pretty involved and
robust definition.
If you wanna look
at the white paper,
by all means take a look at
the citation on my slide.
But I prefer this more
straightforward and simpler
definition of cloud computing,
which I found in a legal ethics
opinion from the state of
Pennsylvania a few years ago.
And in that opinion they talked
about how cloud computing
is really a fancy way of saying
stuff's not on your computer.
And you may say, well Dennis
that's really an over
simplification.
But I think in many respects
this captures the essence
of what the cloud is all about,
because the cloud is really
all about this notion of
off-premises remote computing.
So when I joined Microsoft
back in December of 2002,
the primary way which we
provided our solutions and
our software was through
software licences so that our
software could be downloaded and
then installed on our customer's
servers and work stations and
machines which were on their
on premises environment within
the firewall of their companies.
But the whole notion of moving
to the cloud is that you'll get
services and software on
a remote basis by a third party
cloud services provider.
So it's being provided
remotely and off premise.
The cloud is also all
about the data centers.
A lot of folks think about the
cloud as being something which
is amorphous and
touchy and feely, but
it's really bricks and
mortars at the end of the day.
It's really all about
these data centers.
And of course these data centers
are massive, massive facilities.
They are the size of
football fields and
they contain racks and racks of
servers and computing equipment.
And at Microsoft,
of course we have lots of data
servers throughout the world.
We actually have over
100 data centers in
over 40 different
countries in the world.
Last week we announced
that we're gonna be
having data centers,
effective 2018,
in Africa,
in South Africa actually.
And we're the first major cloud
provider to have a data centers
in the African continent.
So when you think about
the cloud, it's really tangible,
it's really powered by these
massive data center facilities,
which should also be highly
secured environments.
When you think about the cloud,
you're always thinking
the cloud is provided in
three different forms, what I
call the big three of the cloud.
The first part of that Big 3
is something known as software
as a service, where in essence,
software is provided to
you through the Internet.
We think a great example
of software as a service is
Microsoft Office 365 Solution,
where you can get anytime,
anywhere access Microsoft Office
suite of products just by having
a connection to the Internet.
A second type of cloud
computing is something known as
infrastructure as a service.
Also really known as
hardware as a service.
So leading provider in this
space is Amazon web services,
Microsoft also has
a competing solution known as
Microsoft Azure.
And the whole basis of
infrastructure as a service is
that you can buy on
a subscription basis,
computing power,
hardware power, server power.
You can store your data
with an infrastructure
as a services provider.
A final mode of cloud computing
It's something known as
platform as a service.
This is generally
geared to developers,
people who are creating
information technology such as
web applications,
mobile applications and
platform as a service provides
you with a virtual sandbox
where you can engage in
development sorts of activities.
Now there's lots of benefits
associated with
moving to the cloud.
One of the benefits we speak to
with our customers is that we
believe that you can save a lot
of money by moving to the cloud.
By moving away from an on
premises environment
because you no longer have to
buy servers or rent servers.
You don't have to power
those servers up,
you don't need to have space for
those servers,
you don't need to have somebody
to maintain those servers.
So there's lots of business,
financial advantages by
moving to the cloud.
It's also highly scalable,
so you can, if
your business is very seasonal
in nature as an example.
If you need more cloud
services you can ramp up or
ramp down depending
upon your need.
We feel it also allows
our customers to
focus on their core business.
So they really get out of
information technology,
let the expert handled that,
and that they can better
serve there customers.
And assuming that your working
with a trusted cloud provider or
a reliable cloud provider.
There's also this perspective
that you could probably be
more secure providing your data
to a trusted cloud provider and
that they could do a better
job at protecting that data
versus what you can do
on premises environment.
In fact,
most cloud providers push out
updates to their solutions and
to their software.
So we saw a few weeks ago
with the Wannacry situation
where various companies did
not upgrade their Windows
operating system.
If you're working with
a cloud services provider,
those updates are pushed
out automatically.
Now some folks have
identified various concerns
in moving to the Cloud.
Some folks believe that the act
of providing their data,
and their customer's data,
and their partner's data,
to a third party can be
inherently unstable and
it can cause maybe
some security concerns.
We hear that, on occasion,
from some of our customers.
There's this perspective that,
perhaps the larger cloud
services providers could be
viewed as a bigger target for
potential cyber criminals and
hackers.
Sometimes, when you're moving to
the cloud you may have to also
migrate data to a cloud
services provider.
There could be time and effort
and cost associated with that.
So folks have also
identified some
potential concerns in
moving to the cloud.
But regardless, we have seen
that over the last few years
that the cloud marketplace
has grown exponentially.
Here is a survey done by
the Forrester Research Group
where they talk about how
the cloud marketplace is gonna
be growing to almost
$250 billion by 2020.
And so it's become big
business cloud computing.
And because of that
there are a number
of cloud services
providers out there.
It's become what I
call sort of a crowded
cloud provider marketplace.
A lot of times,
companies aren't sure which
cloud provider they
should select and choose.
And so
when I look at the marketplace,
I view the marketplace really
being in four categories.
The first category
are the traditional information
technology providers who
are providing cloud solutions.
Companies like my company,
IBM and Oracle.
Then there is this
other category of
providers who like to say
they've been born in the cloud.
I'm not sure what born in
the cloud really means.
But those providers
are the Googles, the Amazons,
the Salesforce.coms.
There's also a third category
of smaller cloud providers which
are out there, some of them
make money, some of them don't.
Some of them maybe
an acquisition for
a target for an acquisition
by another company.
Then I think a fourth category
are these providers who perhaps
were really never in the
information technology space,
but they've re-engineered their
business to get involved in
cloud computing and
offer cloud storage solutions.
Examples of those companies
are the phone companies,
telephony companies.
Companies like AT&T and Verizon.
But a key point and one of the
key takeaways, I think from our
discussion today is if you're
thinking about moving to cloud,
and lots of entities are and
have already moved to the cloud.
It's really important that you
spend your time conducting some
thoughtful due diligence and
ensuring that you can
trust your cloud provider.
Something which our
Microsoft President and
Chief Legal Officer, Brad Smith,
likes to say time and
time again is that in these
uncertain times when we've
seen cyber attack issues, data
loss issues, that companies will
only use technology that
they can truly trust.
So make sure you're spending
your time properly evaluating
cloud services providers.
So when I think of a possible
framework which folks may want
to use in evaluating potential
cloud services providers,
I think a key foundational
element is this whole notion
of compliance.
You wanna work with
a cloud provider who,
hopefully, can help enable you
to meet your compliance needs.
But there's also other key
areas to really hone in on,
such as security.
What do they do to protect
your data in the cloud?
How about privacy and
data control?
What are they doing to ensuring
that you continue to own your
data, that you can get access
to your data even though it's
stored in a cloud
provider's cloud?
And also transparency, you wanna
work with a cloud provider who
hopefully is very clear.
And truly transparent to you
regarding their cloud services,
business practices and
where your data is
located in their cloud.
So, what we'll do now is
change gears a little bit and
we'll have a conversation
about the cloud.
And again, we invite
the audience to ask whatever
questions that they may have.
But let me throw out
a first question.
It's sort of a two
part question.
Who are the key stakeholders
that should be involved
in deciding which cloud
provider to use and
what should be the role of
a compliance professional during
the cloud provider
evaluation stage?
Edward, you may have
some thoughts on that.
>> I do, it's good that's first
question, because Dennis,
when you first asked me to be
on a panel on cloud computing.
My first thought was, well,
that's not my primary focus,
I don't do cloud computing.
But then when you start thinking
about it, there's no one person
in most corporations that,
that's there focus.
It's a team, it's a very
varied team of people
that you need to collect.
And it's not just legal and
InfoSect,
the one's that
first come to mind.
It's not yeah, the infosec or
IT say based on cost savings and
based on efficiency here's
what we want to do and
then the legal is it's
a binary go, no go.
It's not like that or
it shouldn't be like that.
There are a lot of other
factors and considerations
on moving to the cloud and you
need the entire team to do that.
We focus on going
to the business and
figuring out what
the use case is.
What data is being proposed
to put into the cloud.
How it's gonna be used?
Where it's gonna be going?
And then that really informs
the entire decision,
so maybe want to narrow the type
of data you are moving in.
Maybe there is some data
that is too sensitive for
you to put it into the cloud.
Maybe there are others that you
are not comfortable with that
particular proprietor.
There is a lot of different
patterns and variations to go.
But you need to have all
those discussions right from
the beginning, gather all your
team right from the beginning.
I think that is the second
part what is the compliance
professional in my world
of privacy professionals.
Role is to make sure all
of those people have input,
make sure all of
the voices are heard.
And make sure that each concern
is addressed along the way.
>> And just to highlight
a few points which you made.
I think it's when we see this
in our customers when they don't
get the key folks and
stakeholders involved early and
often.
And that's something which
you really need to do.
We've seen time and
time again some of our customers
have signed our cloud computing
contracts, but after they sign
the contract then they wanna
start using our solutions So
we get involved in sort
of a second sales cycle,
a second negotiation
because they haven't had
key decision makers
involved in the front end.
So make sure you group
those folks in very,
very early and often.
I think there's an interesting
role for the compliance
professional to play sort
of a quasi quarterback,
to making sure that the right
folks get involved.
>> Yeah, I would just say, also
from an efficiency standpoint,
I mean, to your point Dennis,
oftentimes people will
sign cloud contracts and
the board of directors have no
idea that things are actually in
the cloud or
that a contract has been signed.
And you wanna get everyone
involved cuz oftentimes
you'll have these really very
valuable security briefings
about the cloud.
And ultimately the right
people are not in the room.
And so when it comes to legal,
when it comes to procurement,
the CMO,
they've had no participation or
no involvement or
buy in on this decision.
So you find yourself
doing it over and
over again trying to
get the right people so
that they will embrace cloud
services, essentially.
>> No doubt and
I would also make another point
that you definitely wanna get
the right stakeholders involved,
but
be sensitive to maybe getting
too many folks involved.
Sometimes at Microsoft we'll get
involved in our cloud contract
negotiations and a customer
will get 25, 30 folks involved.
We can't tell a customer how
many folks they should get
involved.
But I think focus in on the core
stakeholders who can really
provide value,
who could be sort of nimble and
sort of efficient
in the process.
>> Dennis, I just wanna add,
so we have a very structured
DOD acquisition process.
So it's very structured and
every cloud service provider and
cloud server offering must be
certified by the Defense
Information Service Agency.
That being said, we spend
$8.3 billion on IT just for
the Army alone.
So moving to the cloud,
we're looking to reduce cost.
But one thing I wanna point out,
it also harmonizes our security.
So some of this security
risk can be reduced by using
the cloud, since we don't
have so many systems.
The other thing I like to
point out is like we always do
a cost-benefit analysis on the
systems that we wanna move to
the cloud.
Because, in some cases,
moving to the cloud won't
save us money actually.
If it's just a one-off system,
that we not gain efficiency
by moving to the cloud.
>> That's great points,
great points.
Any questions from the audience?
Just pause here for
a second, okay.
So let's move on to
another question.
What are the key criteria that
customers should consider when
evaluating a cloud
services provider?
>> Yeah,
I'll jump in on this one.
So the things that you
had up there before.
It's security, compliance,
transparency, control.
Not everything fits neatly
into those buckets.
But those things tend to be,
again,
the board of director
level involvement.
So when you think
about security,
security is still a big issue.
I think it really depends on
the industry, quite frankly.
I sell to, again,
a broad base of customers.
So security is
a really big issue.
But you have to think about it
from the standpoint that your
cloud providers, the big ones,
it's a core competency security.
So they can do
the patch management,
the virus monitoring,
the firewall, things like that.
Whereas in a lot of smaller
companies it's very
difficult and
not cost-effective to do that.
When you start thinking about
other things like data loss,
I mean, that's a big issue for
people.
But in a public
cloud environment,
it's very very rare for
that to happen,
simply because of the way
the data is stored in the data
centers and we'll probably
get to that in a little bit.
In terms of compliance and
control,
that's another big issue for
customers.
And my advice to people
is to do some research
on your cloud provider.
Make sure that they're using
the data just to provide
the service, and nothing else.
So there should be no targeting
of ads, no data mining,
things like that.
I kind of like to look at it
from a philosophy of that you
should be almost in
no worse position,
because the CSP should be,
in a sense, a custodian.
It's almost like a safe
deposit box, if you will.
You're putting that data
in that safe deposit box,
you control it.
You should have all the legal
benefits as if you had control
of that data yourself.
So if you think about
what that looks like,
an example would be
government access, right?
So does your CSP
have in protocols,
procedures, process that if
government access, if government
comes to request data, that
they're gonna notify you first.
And if they can't notify your
first, what's their procedure?
Are they willing to contest it?
Are they willing to
take it to court?
That's a big control issue.
The compliance aspect of it
is that you can't comply if
your standardized service that
you're using does not comply.
So all cloud providers
should strive to meet
any regulation that applies
to cloud providers.
And that is a situation too
where you need to look at
the history of your
cloud service provider.
Where were they when
ISO 27018 came out?
27001, Safe Harbor,
model clauses.
Did they comply with
those regulations?
And that'll give you sort of a
backdrop as to their history and
their core competencies and
whether they take it
seriously or not.
>> Two things I wanted to add.
The one thing for DOD and
Army, it's very important,
is past performance.
We always look at the past
performances of how
they performed in
other contracts.
And the other thing is the value
add services that they can add.
One thing that's very important
to us is migration support.
So when we're moving that data
from our existing system to
the cloud, that support
is very important to us.
And how they can plug in and
help us with that migration.
>> Just a few follow-up points.
For those of you that don't
know what ISO-27018 is,
that's an international standard
regarding how a cloud services
provider needs to protect
data in the cloud.
It was the first
international cloud service
certification compliance
standard, Microsoft was
the first major cloud provider
to achieve certification with
that standard.
But make sure that your
cloud services provider
complies with that
important standard.
One thing which we've seen
with some of our customers,
in terms of the due diligence
in evaluation process is that,
often times they will send us a
detailed security questionnaire
document.
Sometimes they'll send us
a request for proposal documents
asking us a variety of questions
as to how we secure data in
the cloud and what our practices
and our protocols are.
So that may be a practice
which you want to consider.
Yes, ma'am?
>> So with respect
to that [INAUDIBLE]
[INAUDIBLE].
>> Thank you for the question,
it's a great question.
The question is at
the end of the day,
Microsoft talks about its
practices and policies.
Are they willing to step up to
the plate, if you will, and
to provide what I would call
a proactive hold harmless
indemnity provision?
It's a very good question, we
see that from customers time and
time again.
We're gonna be talking about
the allocation of risk.
So at Microsoft, we have what
we think are commercially
reasonable limitational
liability terms and conditions,
whereby we are willing to
protect up to 12 months of fees
paid for the services.
We don't technically have
an indemnity provision built-in
in our standard contracts.
I know we're on video tape here.
But I will say that depending
upon the nature of the customer,
the size of the deal, we have
been willing to provide what
we call a reimbursement of
customer mitigation costs,
if there is a data loss,
a security incident.
But that reimbursement is always
capped by whatever cap on
limitation liability
which we agree to.
So it's not an unlimited
stance vis-a-vis indemnity.
And so we've spent a lot of
time over the years, of course,
benchmarking the standard
contracts of our customers.
So we think that, for
the most part, our terms are
very commercially reasonable.
But I think that's
part of the give and
take of the negotiation
process with the customer.
But that's a terrific question,
thank you very much for it.
>> Let me add one more
consideration that I really
hadn't talked about before and
actually for
the three of us, it probably
doesn't matter as much.
But I heard yesterday from
a gentleman from the FBI and
he was talking about
the potentiality
of future ownership.
So his example was
Amazon Web Services,
which might be a little
bit example too far.
But there are a lot of
smaller cloud providers,
a lot of startups that are.
And his thought was if you go
ahead and put your data with
them and then they get sold to
somebody who's got a beneficial
owner in China or North Korea or
some other state which may not
be as friendly to data privacy
law, what's gonna happen?
I don't know, it's only been 24
hours since I thought about it.
I'm trying to decide whether
that's too paranoid or not but
maybe it's not.
Maybe thinking ahead a little
bit to how that provider's gonna
be and what the likelihood is in
a year or two of the stability
of your data in that cloud
provider, is probably not bad.
>> I think it's a fair point,
and that's something which
you should ask as part of
the due diligence process,
to understand the makeup of
the cloud services provider,
what their reputation has been.
Do they make money or not?
What's the likelihood
of them being a target?
How good are they
doing financially?
If they declare insolvency or
bankruptcy,
what happens to your data?
Also, just sort of as
a corollary point on
the indemnity piece.
There may be some smaller
cloud services providers who
are willing to indemnify you,
and that's all well and good.
But what does that indemnity
really mean at the end of
the day?
So when you're thinking about
indemnity, I would say to you
that indemnity probably means
more from a well capitalized
cloud services provider
versus a smaller provider.
>> Yeah,
one thing I wanna add is also,
we talked about indemnity, but
also the portability
of the data.
If all else fails,
I think it's very important
to have a contract clause
that talks about
the portability of that data and
how it's gonna be exported and
what format that will be.
So that's very important to us.
>> Yeah and I would say for the
indemnity standpoint as well,
our duty to defend.
There are things you should be,
in a sense, made whole.
If you're utilizing
the service and
you're sued as a result of
just using the service, right?
And to your point,for
the FBI example,
I mean you still
have a contract.
You still have breach
of contract remedies.
You also have encryption.
And as cloud services evolve,
there's encryption where you
bring your own key, essentially,
where you have
control of that data.
So if you're using best
industry practices with respect
to encryption,
there is some protection there
even in a cloud service.
>> Yes sir?
>> So on this issue of
portability of data, [INAUDIBLE]
took over [INAUDIBLE] of
the cloud a few years ago,
reacting to
the company's contract,
there's nothing in there
about affordability.
Whether or not they go bankrupt
or whether we decide to split.
So is there
an industry standard,
is there standard terms for
something that we
should have been savvy,
to insist on deployment.
>> Well a few thoughts and
I'll let others jump in.
But I think you should always
make sure that it's part of
your contract,
the cloud provider,
that you always have
access to your data.
That you always very clear
that you retain ownership to
your data.
I know at Microsoft,
in the event that there is
an end of our contract, we
always provide the ability for
our customers to get access to
their data for a period of time.
Or they may want us
to delete their data.
And if they want us
to delete the data,
we're willing to do that too.
So we give options, but
I think that's an important
consideration to include and
to think about in your contract.
You really need to be cognizant
of the so-called exit strategy,
if you will.
>> Right, and one thing I may
add is the migration services.
I think you need to have in that
contract is a period of time.
If this relationship
doesn't work out,
then you have a period of
time where they're gonna help
you migrate that data
off that system.
And then you specify the formats
of that data's gonna be,
I think that should
be in the contract.
>> Yeah, I think also
with GDPR coming out,
there are gonna be some
provisions in there
on data portability and
migration of data.
So if your cloud service
provider provides services to
people in Europe, then they're
gonna have to comply with GDPR.
Or if they have offices
that are based in Europe,
will have to comply with GDPR.
I think it's almost impossible
from a cloud contract standpoint
to say well, how much time
is enough time to migrate
data to the new service, right?
Most cloud contracts
will say 90 days,
that that's their
responsibility.
And that's
a standardized service,
keeping in mind that you have
literally tens of thousands,
hundreds of thousands
of customers, right?
So you're standardizing
that service, so
what happens if it takes
you longer than 90 days?
And so that's something that you
can talk about with your cloud
service provider, but
there should be a plan in
place to migrate the data.
And as long as you're paying for
it, by the way,
they have to maintain it.
So even if the contract says 90
days, if you still have a valid
contract and you're going
month to month, for example,
they're still providing
the service and
you could be planning to migrate
that data as you move along.
Just food for thought.
>> Just a point on the acronym
GDPR, that stands for
the General Data
Protection Regulation.
It's a new law which will come
into effect about a year or
so from now.
And it's gonna be if you
have a personal identifiable
information of folks in the EU,
or companies in the EU,
if you do business in the EU,
you're gonna wanna make sure
you comply with this new law.
It contains a number of
different requirements
from a legal perspective,
I think there was a session
yesterday on the GDPR.
And if you don't
comply with the GDPR,
you stand to be responsible for
up to, I believe,
4% of your revenues if you're
in violation of the GDPR.
So make sure that as you're
moving to the cloud,
give some thought as to how
do you become GDPR ready.
I'll just give one more plug,
if you don't mind, for a second.
Tomorrow there's gonna
be a webcast with our
Microsoft President and
Chief Legal Officer, Brad Smith,
and our Chief Privacy Officer,
Brendon Lynch, where they're
gonna talk about what Microsoft
does from a GDR perspective.
It starts at 9 AM Eastern
time tomorrow morning.
I know all of us will be still
here at Compliance Week, but
you can still listen to it on
demand as your schedule permits.
Yes ma'am?
>> Yeah,
I am coming at this from
the GDPR perspective.
And we're trying to
engage a lot of new
SaaS services for
security purposes.
And I'm trying to make sure that
we are GDPR compliment now, so
that I don't have
to do it next year.
And get the right privacy
clauses in a contract.
And try to get our works
council approval for
these sorts of new services.
And every single provider
I talk to acts like
I'm the first one that's
ever asked and so
the responsibility is falling
on me to draft those clauses.
And to put together
presentations and materials for
works councils to convince
them that this is a good idea.
And I feel it's not sustainable
and it can't be the right way
cuz I don't know anything
about SaaS services [LAUGH].
I'm not the best person
to be doing this.
And so I'm wondering, am I
talking to the wrong people?
How can this process
go more smoothly?
It feels like I must be
doing something wrong.
>> Well, first of all,
it's a great question.
And I don't think you're
doing anything wrong.
I think you're asking the right
questions at this point in time,
cuz it's really
important to gear up for
GDPR because it's
a pretty involved law.
At Microsoft, at our Microsoft
Trust Center we have a number of
resources which you
can take a look at,
which provide some advice,
some sorta rules of the road if
you will,
as to how to become GDPR ready.
At Microsoft also we're
the first major cloud services
provider which is offering GDPR
contract terms as part of our
cloud contract.
So also make sure that your
cloud provider isn't just
talking about GDPR,
if you will, but
they're willing to back it up in
terms of actual commitments in
their cloud contracts.
I'll also say that we've
been seeing more and
more of our
sophisticated customers,
asking about what are you
doing vis-à-vis GDPR.
So you're asking the absolute
right questions.
And I do think you're
probably gonna see more.
At Microsoft, we like to feel
that we sort of set the tone for
the industry, if you will.
I think you're
gonna see more and
more cloud providers
have a conversation and
talking about what they
are doing to comply with GDPR.
But definitely ask
those questions,
drill down in more detail.
>> [INAUDIBLE] Does
this [INAUDIBLE],
is there typically someone
internally who has this in mind.
We want our service to be
used by this company, so
we're going to help this
company convince its work
councils that-
>> People within the cloud
providers business bascially.
>> [INAUDIBLE] People, or?
>> We don't have folks who,
I would say directly would
advocate on your behalf in front
of those regulatory authorities.
But we have a bunch of lawyers,
regulatory folks who constantly
interact with these folks.
And we've been getting
our house in order,
to make sure that
we're GDPR ready.
So you wanna work with
a cloud provider who has
resources available.
Which then you can show to
these regulators to say,
hey we've selected this provider
because they really know what
they're doing there
with GDPR readiness.
>> So
I'm asking the same questions,
I'm in the same boat as you.
>> It's like why am
I putting together
a presentation on crowdstrike,
I don't know anything about-
>> I do understand.
And I have not found that
person in any cloud providers.
That one person that is like,
yeah, here's the deck we use for
the works council
on our last client.
I mean, we don't have that.
But on the works council, I will
say it's not apples and oranges,
but there are separate issues.
And each company's
works councils, and
actually each country's work
council within that company,
do have different tolerances.
So I do from that side.
I can understand
why a Microsoft, or
it doesn't say here's our
works council presentation,
cuz it does differ.
But you're right, you do
need to ask those questions.
First of your HR, this gets
back to our first point.
Get with your HR and say,
what are your worse councils and
you're going to be looking for.
What sorts of information
do I need to gather
about this new solution?
So from the provider, and from
the business and from IT, so
we can package it all and go
to the works councils and say,
here, this is what
we plan on doing.
But it's not an easy answer.
Especially if you're
an American company and
you're dealing with a US cloud
provider, a lot of times, you're
gonna hear from them, we don't
have anything to do with GDPR.
We don't wanna deal with it.
That's not the right answer.
And they'll find that
out in about 369 days.
>> [LAUGH]
>> But it's evolving.
>> It's the answer for army,
all our systems, right?
>> [LAUGH]
>> So I'll give a plug,
microsoft.com/gdpr has
a lot of materials on it.
There's a whole industry
popping up around this as well.
They estimate it's
about a 4 to $5 billion
industry in terms of helping
companies prepare for GDPR.
So there are a lot of people
out there who are doing this.
Microsoft has a lot of partners
who help our customers
in this format as well.
So there's stuff out there, but
certainly willing to talk with
you more about it if
you have questions.
>> Yes, ma'am?
>> [INAUDIBLE].
And they've been very helpful
in going through agreements.
And put in clauses
[INAUDIBLE] for
the providers to need
to comply with GDRP.
So I would have no idea.
But if you get a [INAUDIBLE]
they should be up to speed
with it.
>> [INAUDIBLE]
>> So
yeah, and keep in mind too
that it's a partnership with
your cloud service
provider on this as well.
Obviously the cloud service
provider has a lot of things
that it needs to do, to make
sure that it complies with GDPR.
And it's a huge,
huge engineering commitment.
I mean very expensive to do.
But on the flip side,
you've also got a lot of things
that you have to do as well.
I mean,
if you have loyalty programs,
all kinds of things where you're
collecting personal information,
you've gotta ensure
that you're complying
with GDPR also in how you
collect that information.
So it's not as simple as just
dumping it in the cloud,
problem solved.
I mean, there's a lot of work
to be done on both parties' end,
to make sure that
you're successful.
>> So I'll just make one
point about Microsoft.
So I'll be happy to talk more
offline, but Microsoft stands
ready to help you navigate
through those issues as needed,
so, okay?
Any other questions?
Okay, well,
let's move to another question.
>> I think we had
one over there.
>> I'm sorry, yes ma'am.
>> You mentioned
earlier about
[INAUDIBLE].
>> Well, we're transparent
as to where our customers,
where their data is
located in our cloud.
So as an example, let's say you
provision, you are what we call
a tenant, or a data center
in North America, or the US.
We say that your data,
at rest, will remain in US or
North America.
I'm assuming we will make
similar commitments vis-a-vis
Africa and maybe the Middle East
& Africa region, if you will.
>> Yeah, I would just say that
that's correct as I understand
it, that we can keep, the way
we engineer our services,
we can keep your data
in a particular region.
There is some redundancy,
so it's something that we
can talk about with our
technical specialist to ensure
that you have data where
you think it should be.
The other thing that I
would say is important for
a CSP is to ensure that wherever
the data rests, the laws
of that particular jurisdiction
will apply to the data.
And that might sound sort
of counter intuitive.
But you don't want some foreign
government coming in and
issuing a subpoena, trying
to get access to your data,
because it's in the US, or
some other place, or whatever.
So again, look at your CSPs
track record in defending that.
I mean,
are they willing to stand up.
I can tell you, and this is
not a plug for Microsoft, but
we've sued the US
government over this.
So right now it's, I guess,
in the Appeals Court.
We won the last round but it may
go higher, depending on one of
the the government decides to
appeal or not, but anyway.
>> Not just the data, but
it's also how [INAUDIBLE].
>> Absolutely, it's all
really important for sure.
And that's the other thing.
It comes back to control,
which there's a whole bunch
of questions we probably will
not have time to get to.
But that's part of control, and
who has access, and is it
logged, that kind of thing.
And the physical security
aspect of the data,
while it resides with your CSP,
is extremely important as well.
So when you have that
conversation, I think that's
an important aspect of
the conversation to have.
And how do they control
the data, what's the physical
control, and who has access,
and how is that logged in third
party access, subcontractors,
all kinds of things.
I think those are discussions
that you will need to have.
>> Critical question,
okay, well,
let's move on to
another question.
So what other trends do you
guys see in the space of cloud
computing?
>> So a big thing for
Army and DoD is SaaS.
So we're Service as a Service.
That is Software as a service.
That's a huge thing for us.
And it looks like we can move
many of our applications to
the cloud, using SaaS,
especially around
the office environment.
>> Yeah, I'd say for
me the biggest trend seen across
various industries, is just
embracing of the cloud, right?
So three or four years ago,
a lot of companies would have
never put sensitive
workloads into the cloud.
Not only are they doing that,
but they're doing it a public
cloud format, taking advantage
of the huge economies of scale.
So I think public cloud,
there was a survey by LinkedIn,
their information
security community,
which spans across seven or
eight different industries.
And this survey is
about two years old.
But 71% were either
in the cloud or
actively planning
to go in the cloud.
So I would say that it's not
only where you are today in
terms of your planning, but
where will you be in a year from
now vis-à-vis your competitors.
And if you're not in the cloud,
is that gonna put you or
your company,
at a competitive disadvantage.
So I mean those are the big
things that I see.
And it used to be number
one conversation, security,
security, security.
Now the number one conversation
is about privacy and control.
And security's still important,
don't get me wrong.
It's still a big thing,
but most people have done a lot
of research now about cloud and
they feel a lot more
comfortable about security.
And I've heard a lot
of CIOs say this.
That the fact of the matter
is that the major
cloud pro vendors can
protect their data,
better than they can do
it in-house themselves.
And that, again,
is because of the monitoring.
We spend over a $1 billion
a year, just in security alone.
So it's now a core competency
of all of your major cloud
providers.
>> I think the other thing is
having your data at one cloud,
is you can do big data analysis.
So when all your data is
residing at one system, allows
many big data analysis that
wouldn't be available before.
So it's a huge
business analytic.
>> The other point I would
just make before going to
the question, is that we've been
seeing customers atop what is
known as a hybrid approach.
There are some customers who
don't wanna go all in the cloud,
and so they keep some of
their data on-premises,
maybe some of the more
sensitive data.
And they move other workloads
and data to a cloud.
Yes sir, you had a question.
>> I do, thank you, and I wanna
turn back to the indemnification
issue, if I could, as we were
talking about data security.
In our situation, we're actually
taking our customer data, and
then we would be putting
it on the clouds.
So there's an expectation
from our customer,
that we will be
safeguarding that data,
to the extent that the 12
month subscription cap exists.
That's not cutting it
with our customers.
They want a more absolute
commitment from us.
How do you suggest
we bridge the gap?
Because I have seen over
the last five or eight years,
where the 12 month subscription
value, becomes the standard.
But it's wholly inadequate in
terms of the potential liability
that we could expect.
How do we bridge that gap?
>> Well, we see that come up
many times as part of our
discussions with our customers.
And I think at the end of
the day, from my viewpoint,
it's ultimately a business
risk financial issue, right?
Depending upon the economics
of the deal, the nature of
the customer, we may be
willing to elevate that cap.
But it's all really based
upon the financial aspects of
the deal.
It has come up time and
time again too.
>> I'll be honest with you.
I mean I very rarely rarely not
see a deal not happen because of
limitation of liability,
or because of indemnity.
Normally, the parties are able
to get to a point where they're
both comfortable.
And it takes a while
to get there.
I mean many times there's
security briefings,
understanding what the service
is, and how we store the data,
how it's encrypted.
So if the issue is
data being stolen,
some companies get more
comfortable when they
understand how we store
the data, how it's encrypted.
How your data's not
gonna reside on one box.
It's gonna be stored across the
data center on many different
servers.
So hypothetically, if a server
were penetrated, not only is it
encrypted, but it's only gonna
be a piece of your information.
It doesn't answer your question.
I think there's flexibility
there among cloud
providers, right?
So yeah, I've often heard,
this cloud provider will take
uncapped liability and
that sort of thing.
I would just tell you to read
the fine print on those.
Because it's not really true in
what some cloud providers say
about uncapped liability, and
what they're willing to provide.
I think at the end of the day,
there's no cloud provider that's
willing to bet their
entire company.
Because essentially,
it's not gonna be one
customer that's hit.
It's gonna be a multiple,
across hundreds,
thousands, tens of thousands
of customers potentially.
So there's gotta be some
reasonable cap on liability.
What that is, I think you'll
have to get comfortable with
your cloud service provider.
But those are one of the things
that, quite frankly,
a cloud service provider
can negotiate with you.
There's certain things that,
because of the way the service
is engineered, I can't
change even if I wanted to.
But indemnity, limitation, and
liability, my commitment to
you or your cloud services
provider's commitment to you.
Those are things that,
quite frankly, I believe can be
negotiated where you can
get to a level of comfort.
>> Well,
it's a risk analysis, right?
And you have to look
at the alternative.
So I don't know what
your situation is, but I
mean there is a lot of companies
that are starting out, and they
don't have very sophisticated
IT resources themselves.
So while you might say, look,
we wanna go into the cloud but
we want protection.
But from your customer
standpoint, and even from your
board of director's standpoint,
what's the alternative?
You are to maintain it inside,
on premises,
what protections are you
affording there?
I mean, what would be
your exposure there?
You might not be increasing your
exposure dramatically to go into
the cloud, either from
a security standpoint, or
from a financial standpoint.
So it's not a simple math
>> [INAUDIBLE]
>> Yeah.
>> [INAUDIBLE]
>> Yes sir.
>> You talk about [INAUDIBLE].
>> What are the cloud providers
doing to ensure that their
component pieces comply with
data protection requirements?
>> Well, I'll throw
a possible answer there, and
feel free to chime in.
At Microsoft, we use a number
of subcontractors as part of our
provision of cloud services.
We use these subcontractors, cuz
quite frankly, it's a way for
us to provide competitive costs,
associated with our cloud.
But we take responsibility on
behalf of those subcontractors.
We put that into our contract,
that we're responsible on
behalf of our subcontractors.
We also notify you and tell you
when our Microsoft Trust Center
site, who our subcontractors
are, so you're aware of it.
So we flow down a number
of those terms and
conditions, which we're signing
up to with our customer, through
our sub-contract agreements
with our subcontractors.
>> I'm sorry,
just the question's are you more
concerned with ensuring that
we're keeping up with rules and
regulations with respect to our
hardware, that it's
best of breed?
Maybe I didn't understand
the question fully,
so if you can either drill down.
I'm sorry about that.
>> No problem, no problem.
I work a lot with
component systems, right?
My job is to go in and
look at them, and
make sure that they meet the
data protection requirements.
But dealing in
the digital world, right,
it's a combination of a lot of
different pieces glued together
to make the cloud.
>> Gotcha, okay.
>> What I wanna know is,
what is the focus of
the cloud service providers?
What is their due
diligence focus right now,
on ensuring that one of
those components, or
all of those components, meet
data protection requirements?
Is the focus just
on the software,
is the focus on the hardware?
Cuz we talk a lot
about security and
how impenetrable a network is.
But do we also
look at whether or
not each component of that
cloud system, the hardware and
the software, meets the data
protection requirements?
>> Yeah, so I would just say
that we're building data centers
around the world.
Some of them we own outright,
we construct.
We build some of them.
Our list but
we stand behind our SLA.
So we're looking at
the entire system.
We're ensuring that your
hardware specifically, I think
we take our useful service
out of life every two and
a half to three years.
They're shredded, broken into
pieces, new servers replaced.
So, we try to ensure that we're
using best of breed services,
hardware.
Again, there's penetration
testing, all types of patching,
things like that that go on
on a daily basis to ensure
that we're meeting our
SLA requirements and
that we stand
behind our service.
We say that we're going
to comply with all laws.
So it comes to any cloud service
provider not just Microsoft,
but you want to make sure
that they're putting these
things into their SLAs,
into their contract and
if not then it's a breach,
right?
So you want to make sure that
they have some skin in the game,
with respect to these things.
>> I'll just add a few
other points too.
It's my understanding that as
part of our contract terms we
have a robust definition of
Microsoft Online Services.
So it's really the combination
of our services,
which go into our
cloud services.
It's a combination of items
basically, if you will, and
we stand behind
all of our terms,
stand behind that definition
of Microsoft Online Services.
I also think, I believe that
it's part of the servers which
we have in our data
center environment.
I think we design the
blueprints, if you will, as to
how these servers are supposed
to be architected if you will.
And then we have our providers
package their servers
appropriately.
>> So one of the riskiest
elements that we see and
we look out for
is that human element.
So that insider attack is
the other thing that we're
very much concerned
about at DOD.
So we have very strict
requirements on security
clearances on the contractors
that work on our system,
so that's the other risk
factor is that human element.
>> Just a question,
I know we touched upon
this a little bit is how
about thoughts about
third party or
law enforcement access
to data issues?
Any perspective on that?
>> Yes,
I deal with it all the time.
My system is financial
disclosure of management and
there's a formal protocol to
request every senior leader
in DOD or actually in the
government needs to file what's
called the office of
government ethics 278 form.
And there's a formal process
where that can be requested and
so we provide that
data as requested.
We've also provided bulk data
to researchers, anonymized data,
so that they could do big data
analysis on the data that's
in our systems actually.
To match the assets to see what
DOD officials or what assets
they hold and in aggregate
>> Any thoughts about that?
>> Other then making sure that
your CSP has a process in place
with respect to
government request.
I mean, make sure that there
is transparency around that.
Make sure that, for example,
the results are published so
you know how many
requests there have been.
How many times your CSP has
actually had to turn over data.
Is your CSP,
is there a built in process?
I mean,
when I talked earlier about
the safe deposit box analogy, I
mean our position is that you're
in a better position than your
CSP to decide what complies with
a third party requested data.
So, our position is that
we will always, and
this is engineered in, we will
always redirect any request for
data to the owner of that data.
And if we're not permitted to
do that we will contest it.
And so I would encourage you
whatever CSP that you look at to
make surethey have
a process in place.
And make sure they have
a demonstrated history
of doing that.
Have they ever sued the
government with respect to this?
Or are they just saying?
You can write anything
in a contract?
But do you or are you
willing to stand behind it?
>> As a customer we
absolutely expect that.
If we're putting data and
assets into the cloud
then we are looking at it like
a safe deposit box too and
we expect the cloud
provider to protect it and
defend it against all requests
as we would ourselves.
>> And another question.
I know we touched upon parts
of this question earlier, but
with regard to limitation and
liability and indemnity but,
how much contract negotiation
changes should a cloud provider
be willing to entertain and
or accommodate,
in terms of a sometimes
we see with our customers
a proverbial battle
of the forums.
Microsoft has got a standard
cloud services contract.
Sometimes our
customers come back,
and they have a specialized
exhibit with regards to their
security protocols and
practices.
So, any thoughts on that?
>> Yeah, I have a comment
it goes to the question
over here about the GDPR and
new.
A lot of times you are not
negotiating against your cloud
service provider, you are
negotiating with them to ensure
that the contract that the two
of you put together is going to
be satisfactory to a regulator,
or to whoever looks at it later.
Or to a work's counsel.
Sometimes there is some
work's counsel think
about Germany in particular
that can sometimes get so
detailed they actually
want to see contracts.
So they want to see
the protections are in there.
So you need that discussion with
your cloud service provider is
not necessarily adversarially,
it's not.
I want two years worth
of fees instead of one.
It's not that sort of thing.
IS what do we need to add
to this contract so that
the regulators from data privacy
perspective believe that we're
both protecting the individual
data that we put into it?
So sometimes it takes
a little getting used to
from a negotiation standpoint.
But if it's done right, it's
not as painful as it could be.
>> Okay, great.
Another question.
Are there any third party
resources out there which you
would encourage folks
to take a look at?
I'll just do a quick
answer to that question.
I think a great resource is
this organization known as
the Cloud Security Alliance.
They've become more of a leading
think tank in the area of cloud
services, so check out their
websites and their resources.
Another terrific organization is
the International Association of
Privacy Professionals.
IAPP and they have got some
great information about data
security privacy, especially
as it relates to the cloud.
But any other
research you wanna-
>> Yeah except for the DOD.
There is the Defense Information
Services all of our cloud
are related there in
the public domain.
I'd say for the private sector,
feel free to leverage what we
have there, it's all available
in the public domain.
>> Okay, okay.
Just maybe one last question
I'll throw out there.
So once your cloud
contract is signed,
how important is it to
consider managing and
monitoring your contract in
this ever changing environment?
>> Yeah, I think at
the end of the day, again,
it's a partnership, right?
So I think you have to do that.
I mean there's a whole industry
that's sort of sprung up around
that today.
To be honest with you,
we're struggling to keep
up with it because we're
really trying to have some
consistency around the process.
So, I can tell you what we do.
We publish a lot things,
we republish the results
of our audit.
We republish the results of
log enforcement requests.
We try to get everything out as
much as possible on our website
so that our customers
can look at it.
Our challenge is
when you have so
many customers, how do you have
a standardized process for
them to evaluate cloud
service after the fact?
And this industry that's
sprung up around it.
You get these security
questionnaires from
existing customers, and
they're all different.
And so from a cost to sales
perspective it's extremely
difficult to reply to every
single security questionnaire.
So we try to have
a standardized response.
For some customers, quite
frankly, that's acceptable.
For some customers, it's not.
So, I'll be honest.
For us,
it's a real challenge right
now in how to do that because
our prior model much like most
of the old line companies
was to deliver software, and
you sort of maintained it.
Now we're doing that.
And so there is that continuing
partnership that you
go through the life of the
contract that you have today.
>> I would just consider, really
to encourage you guys once you
sign a cloud computing contract,
actively manage and
monitor that contract.
Especially with regard to
the service level agreement SLA
commitment.
You want to make sure that
you keep your cloud services
provider honest in that area.
So I think we're right up
against time or so, but
do any folks have
any other questions?
Okay, seeing that there's
no other questions,
we again really appreciate
your attention and for
joining our session today.
Thank you very much.
>> Thank you.
>> [APPLAUSE]