CosmosDB Serverless NoSQL for the .NET Developer

[Music]
hi and welcome back to another episode
of visuals to your toolbox I'm back
again with my guest Jeremy hey Jeremy
welcome back to toolbox thanks glad to
be here
it's awesome to have you and it's almost
like we never left the studio or we
weird the same thing all the time that's
right this is the only actually have a
closet full of hangers that have nothing
but there's your logo shirts on them
that is amazing the cats out of the bag
there you go so we were doing two
episodes today but they'll be published
separately so folks might have not even
seen the previous episode that's totally
fine so today we're gonna really focus
down on cosmos DB right absolutely
cool so what are we gonna talk about
give us give us a summary so I want to
just drill into a lot of the aspects of
cosmos TV I've gotten pretty excited
about it in the other episode we talked
about a link shortening tool that I used
yeah and I enhanced that tool to store
data into cosmos TV so I could find
insights from my application what I
found is just you know it works
incredibly it works really well as a.net
developer there are a lot of ways to
interface with it so I want to talk
about you know what cosmos TV is why
it's important to dotnet developers and
show a little bit about how to get
plugged into cosmos TV and start using
it as a service right and you know folks
didn't see a previous episode that's
fine we'll link it it's not a grindin C
or anything you know the core function
thing and that was a really good thing
to watch your folks are interested but
let's start with what cosmos DB is again
I want to make sure folks coming in may
be really not knowing what does database
technology is all about if you can give
us a quick summary that'd be great sure
so you know when we talk about functions
we talk about serverless which I talked
about flipping that last server cosmos
DB is what functions are to compute
cosmos DB is to storage right it's the
idea of sort of a hands-free ability to
host an enterprise-grade database it's a
sequel right I mean is kind of the
defining way to describe right at a
baseline and then it has a lot more
because of what's being built into it
right so it's no sequel it's it's
document based but what's neat about it
is it supports a bunch of different
interfaces to connect to it
one of those interfaces is what everyone
who has followed the history as familiar
with document DB right so that was the
previous incarnation and what's
interesting about document DB is that
supports sequel syntax so even though in
the database we're storing documents
that can have different schemas and
properties I can still go to my
comfortable select star from group by
order by and use that familiar syntax to
quarry the database if however I'm a
traditional no sequel developer and I'm
used to something like MongoDB we have a
MongoDB driver that works out of the box
in fact I can take an existing MongoDB
application update the connection string
and connect to cosmos DB and use that
yeah that's very cool and then there's
two other api's there's a table storage
API which again you update the
connection string your app that uses
table storage is completely transparent
and then there's one that I always love
to bring up this interface name because
it's called gremlin right so gremlin is
what we call a graph interface and the
idea is that you're storing nodes and
vertices that connect the nodes and the
canonical example is airports so you
have airports as nodes and then there's
flight paths that connect them and
gremlin is a a language that can
describe how to traverse those paths and
then find information things like what's
the shortest path what is aggregate
information along the path etc that's
really cool in folks that you know might
may be googling later on trying to find
some information about about this
product the rename is always like a
moment in time with Microsoft and it
changes after like that so to be clear
like that community B was what this
product was called right it's now called
cosmos DB but we still refer to document
DB interface right over the way the way
we you can interact with it
right but that that does just a historic
factor right of what it was called so
the name might be dead from the product
name perspective but it's still alive
from a interface perspective it is so
alive in fact I did a workshop that was
a node workshop based on or based
on document DB and of course it's
iterated now to cosmos TV so I created
cosmos DB and was able to migrate that
existing application without changing
code I just
they did the connection string and that
existing document DB driver worked yo
asses out of the box yeah we changed the
name because it was honestly just
causing confusion and remember the
conversation for students realize the
power of this technology and hopefully
the new name gives us a new new life and
it's a lot more fun that's yeah sure
yeah yeah I was impressed that you know
sometimes we don't pick the best names
I'm I'm in marketing so I've spent time
naming stuff you you can blame me for
code lines for example oh praise me or
blame me I was on that project but yeah
this this was a good one I I like cosmos
DB and let's jump in to see what it's
all about
sure so the first thing I'm going to
show you just from the portal is the
experience of creating a new cosmos DB
I'm not going to go all the way through
it because I have an existing one but if
you click in to add a new resource and
just pick cosmos DB it's going to ask me
for an account ID so this is just a
unique identifier to access the most as
your resources there's some URL and that
URL needs that and you can then find a
beginning you can always change it kill
it later but right so we'll do my vs
toebox DB and then you pick your API
this is what we're talking about it
supports different ways to interface the
traditional document DB the MongoDB
interface the gremlin or graph interface
and then key value storage table simple
storage so we can pick any one of these
let's just do table storage I picked my
subscription that's my billing I can
create a new resource group I pick my
location and then I can enable geo
redundancy what this is key because this
is going to give me the ability off a
map to basically click different regions
and have it automatically span those
regions for me which is a pretty cool
feature yep what what I often hear
people describe this technology of ours
as a globally distributed database you
know something they can live in lots of
data centers and I believe you mentioned
there's different kind of retention
policies right of how how many
triplication you need we kind of
concurrency you need oh that's right
exactly in fact this is one that I've
created and this is a live one that I'm
using for my link shortening tool and it
stores metadata and you can see that for
my purposes because I'm the only one
he's actually exploring the day
yeah I just have this in the Atlanta
region which is close to me the eastern
US I've got an activity log that'll show
me how it's been accessed so I can
quickly look at you know basically have
there been any errors or issues or
anything along those lines and this is
just basic you know what's available in
Azure by default you didn't configure
any of this till incorrect this is
completely out of the box what's really
nice with this is the the data Explorer
and this allows me to look at my
collections which are containers of
documents they're kind of the corollary
of tables if someone's used to
relational databases you can see I have
these URL stats and this is using a
sequel like syntax like star from from C
and it's giving me IDs so that I can
expand these on demand so if I click on
one of these you can see the actual
document that's stored in this case
there's a unique identifier it
redirected to a page has a timestamp and
this particular document was a facebook
redirect I click on another one though
this is the the power of Cosmos DB and
document databases is this one as a
property called Twitter yeah so I'm not
locked into a specific schema I'm
counting different types of items for
that yeah this is a great power I mean
as a person that's used to creating a
table you know back in the sequel world
and having that table be sort of the
definition of what you look kind of data
you can store that this is a different
way of thinking about it right right no
sequel well and here's what's really
nice I want to jump into some code and
show an example project that I created
but what I love about the way that both
the document DB and the DB
interfaces are written is developers
used to sequel are also pretty used to
object relational mapper z-- yeah so you
know entity framework is a very popular
one and the whole idea is I deal with my
code and classes the database stores
them as relational tables so I need
something to map between the two of
those
what's nice when you take a document
approach is even though the document is
schema lists and you can have open data
and you could do you know a dictionary
key value pair to set the document yeah
you can also create strongly type
classes and you don't need to go through
any intermediary to store them the
driver just accepts that classes and
stores it right into the database cool
let's see so if we we take a look and
I'm going to come back to the the link
shortener in a second but I've got a a
Visual Studio code window opened here
and in this project what I did was took
the USDA food database and imported into
cosmos TV and then stood up an
application so you can search food items
cool and if we look at these links in
the configuration I'm going to just
click on this food group link you can
see the way it's stored actually on the
website is this format of these weird
squiggly lines and arrows and whatnot so
part of what this application does that
won't spend too much time on it says a
parser and that parser is going to read
the lines in and split them up cool and
then it has an importer that actually
imports the collection the transformer
is how it maps them to this is a food
group item or something different but
the importer this this is the actual
code that I'm using to populate the
database and you can see that we have a
helper method that gets our database
connection so we're pulling in our
secrets making a connection then what I
love about this is I have git collection
so I'm getting a collection of a
particular type it might be food group
food item something like that if that
collection exists because this is the
import utility I'm going to drop it and
recreate it so I can just import fresh
this is the code that inserts an item
and that's all there is to it this item
is passed then you can see this is a
generic type of tea so the item might be
a food group class and I'm gonna show
those in a second but literally
collection insert one a sink and that
loads it into the the database for me
I've got a completely separate project
that has my models my classes so here's
food group for instance this could be a
completely poco right plain old yeah see
sharp objects ELR object if you will but
I've chosen to go ahead and add some
metadata that says okay this is a unique
identifier right so using that attribute
to to make code the
Rayo so nae and you know some people
might want to do this fluently so keep
it out of the class completely and you
can do that yeah but for the purpose of
this demo we did metadata the other
brain power and flexibility a great
responsibility exactly with great powers
great responsibility the other thing I
wanted to show and this is a nuance I
think it's important to call out because
this is what people run into when they
move from that relational model to the
document model the first time I decided
to create a food item what happens is
every food item in the database has a
set of nutrients and it can be protein
fat carbohydrate M&C whatever right the
database doesn't necessarily have all
nutrients for all foods so I may have a
food that only has three I may have a
food that has 200 right so the way we
would typically model that in a c-sharp
class is just like this right I have a
nutrient and I have an array of
nutrients which is straightforward the
problem though is when it stores that in
the document database it's just an array
that can have whatever inside of it cuz
variable is a schema schema 'less so if
I wanted to ask a question like what are
the foods with the top protein content
it would have to go into each one of
those records right scan the array find
the entry and pull that out right
becomes a very intensive operation to
come right because of the distributed
nature of the data so what I'm going to
do is search for this cosmos database
that I have pre-loaded so that I can
show you exactly what the data looks
like inside the database and once again
I have this experience that I can go
into the data Explorer I can expand my
collections and here you see I have food
groups nutrient definitions of food
items the way this is stored in the
database and we'll pull up this document
you can see that I have an array of
weights which is fine but this nutrient
doc has nutrients and then water here is
a property it's not an entry in an array
this is nutrient doc dot nutrients dot
water or nutrients dot energy so now I'm
reference
just a path on the document right and I
can do that indexing I still want to be
able to work with it as an array from
c-sharp but all I had to do for that is
just put a helper method that basically
when we pull it from the database we
look at this freeform document format
that's this nutrient doc and build up
our strongly-typed list of nutrients and
then on the flip side when we're saving
we just go through our strongly type
list of nutrients and create a more
loose form JSON type document yeah so
this makes sense I mean I've seen this
kind of pattern before like processing
the XML in some cases you wind up in
very similar because some other systems
XML data right and you're converting it
into some structured thing all right and
your database and they're the way
they've organized it doesn't make sense
for you as a as a query you know
performant thing and you have to
reorganize it right so that will
actually map in and I want show the the
simple case of accessing this data now
yeah so I showed the experience of
importing it I've got a little test
application and this is just making sure
that I have my connection set up
correctly so you can see here this is
the power of dotnet core right we've got
a very flexible configuration model I
can have my keys stored in app settings
I personally don't want to check those
in and have someone steal secrets yeah
so in this case I've set environment
variables you can see here it'll pull
from the JSON unless it's overridden by
the environment and then those secrets
come in and when we deploy this into a
website we can always set them up in the
application settings right so it's very
flexible like that this is getting the
list of groups and this is the code this
is using the MongoDB driver it's
basically getting the name of the
collection that's just a helper I have
as Cori able to a list this is link
syntax that everyone should be familiar
with yeah looks a lot like in any
framework only I don't have to add that
as an intermediary so we here doesn't
developer although this is like being
home you know nothing nothing unusual
so it just iterates the groups and write
some and then here we're gonna grab the
first food item so again it's getting
that collection as Coria Bowl first or
default and then I just pull some
information out so let's take
look at that from my terminal and run it
let me see where I'm at I'm in the
console test and of course this is
dotnet you can just do the donít build
release and it'll come through and in
this case I've got a helper for
connecting to cosmos DB and I share my
models
between several projects so you can see
it built the models the connector and
now built the test so let's run that
that test project really quickly so
we'll go and this is USD a console test
DLL and so it's going through the
connection string getting that list of
food items it's listed it out and then I
tapped that and it grabs that last food
item for me cool so we know it's working
the next logical step in most
applications is to build a Web API
so I've got this Web API and this is
where the scenario that I talked about
with you know give me the top nutrients
comes into play if I want to sort for
example from protein so if we look at
the controller and again what I love
about this is the dotnet developers are
going to be just very familiar with this
type of syntax so here I'm getting a
specific food group so that's my route
passes in the code I'm gonna return not
found if you pass me an empty code
that's just logical right being safe now
I come through and I grab that
collection and here we go find async
food group where code equals code just
link syntax straightforward when it
grabs that I decided to enhance this as
well and grab all the food items that
are in that food group so if you get
that group detail you can get a list of
items and drill into those right and
then it just returns an okay result
that's it that's the API endpoint so if
we come inside of this application now
let's actually expand this out and ctrl
K is the secret code to clear the screen
I did not know that yep so I think this
some right-click equivalent that I use
well go into this and again Don that
bill will go to release you can build
the debug if you want and we'll let that
spend
again models connector being reused
across projects and then we'll do dotnet
and we'll go and run this and one of the
cool things when looking at your code is
is like you said it's very familiar to
dotnet developers of the cosmos
you know client library that the cosmos
team builds gives you a lot of that link
sinks in think against this so that
meant developer you can come in here and
be like yeah in a couple of hours I'm
building against kozmo so you don't have
to really write some new thing and you
have your Explorer in the cloud to see
the data is but they usually question
like my apps malfunction is it's writing
the data is it reading that exactly you
have multiple points they're just there
well there's actually a quick start to
that I love because when you build a new
cosmos DB you can literally click a
button in the portal and it'll create a
populated list of items and give you a
download for a project to hit the ground
running so it's not building it yourself
but it's having an app that you can look
at and interact with so we're going to
just hit this food groups in point to
get the list and again what's happening
here is my API is going out grabbing
cosmos DB it just pulled back those two
food groups let's pick one of the food
groups I'm going to do 200 which is baby
products so for the actually spices and
herbs so then we got this list of foods
with spices and are your baby like
spices and herbs on that they're here to
tell you exactly so now that we have a
database and we have an API we have
everything we need to do to build a
front-end yeah the focus of this is
cosmos DB so I'm not gonna go into too
much detail with the front-end but what
I do want to show you is actually let's
keep this running and we'll open up a
new terminal and in this same project I
have a web actual web page it's a single
page application that just uses Vijay s
I'm not here to say angular react
whatever it was just a great at picking
side but that's a quick way to get up
and running yeah so what I'm gonna do is
see if I have any docker pieces running
let's actually make sure docker is good
to go that little detail yeah it is
good to have go docker spinning for that
there will give it a second what I've
done is I've just created a really tiny
application you can see it's got a bunch
of HTML that uses templates and data
binding to do things like list nutrients
and lists food items just you know one
page of information it there's no
there's no back-end code this is all
JavaScript using yes you render off the
public endpoint that you've built yes
exactly exactly and for this the the
JavaScript include is just using very
simple code to make calls out to the API
endpoints so we've got queries and and
that actually reminds me well Dockers
spinning up it might be done here I
wanted to show you this inside the web
api for the foods controller this is the
one that allows me to do things like
sort or filter by a group so I've got
from the query string I've got a group
ID I've got a from the query string a
search and so this will come through and
basically project the item so MongoDB
has something called projections where
you can say even though there's this
much data in the database this is what
I'm really going to show so subset you
subset of records exactly filters and
sorry yeah and I'm just like link that
I'm used to you know entity framework
and all that exactly it's the same thing
this should be very familiar description
contains the search and the food group
ID equals the food group and you can see
I can chain these together or basically
do one the other or both and then I go
out get the Cori order it and and list
it so again very straightforward in that
sense let's come back and see how our
Dockers doing we'll do doctor all right
there you go so I'm gonna run the image
I created there's a very simple docker
file that uses something called busybox
which is a really tiny web server
because all I'm serving is an HTML page
and there's book page basing right so
I'm gonna map for day today because port
80 is an explicit inexplicably
inexplicably taken on my laptop and we
call this USDA web so we run that I'm
just gonna do this to make
it's the running it looks good let's
come back into our browser and hit like
a foot dev is so powerful it is
undervalued sometimes and here's my
simple view J s app and I can select a
food group let's do spices and herbs
click filter boom it pulls these and
then when I click on them it's going to
give me all the nutrient information so
zinc magnesium and C and then as a
person who eats a hundred percent
plant-based diet
what I might be interested in is
breakfast cereals which one has the most
protein so I'm picking the group
breakfast cereals nutrient protein give
me the top foods look how fast that was
I just came back sorted and these when I
click on this presumably it's gonna have
a higher protein kind of well actually
that's kind of small there but it's also
a small serving size so if we change to
something that has a bigger serving size
like a cup you can see the okay well
let's do one and a half let's actually
take the maximum cup amount there we go
now we've got good good protein yeah and
then we can search with the text so this
is a fully functional app on cosmos TV
and what's really neat is for the front
end I don't even have to care I'm just
working with straight api's yeah for the
back end I can use either document DB
driver or now what I showed you in
this dotnet core app was the MongoDB
driver sure I want to take one quick
second and show you what I use for my
URL shortener and that is the document
DB and what's really nice and we talked
about this in the episode with Azure
functions is the the fact that we have
these things called bindings that make
it really easy to interact with external
sources
yeah there's triggers and bindings so if
I open up this project I'm using the
document DB interface to store all my
information and when I look at this
function host what I did was added this
attribute for document DB I gave it the
name of the database the name of the
collection created if it doesn't exist I
love that about bindings the first time
it calls it can do that
and my connection string is a secret
that I store an app settings so that no
one else sees it I notice I have this
out dynamic dock
so remember schema lists so I can create
the dock however I want using this API
at the very end of my method I create an
expander object which is just you know
key value pair basically for dynamic I'm
adding a gooood to make it unique
putting the page in the count and then
based on the custom event and this is if
it's Twitter Facebook Linkedin my blog I
add that notice that there's no other
code this is an out parameter so by
setting the value of the parameter when
this function exits automatically writes
the document into Cosmos DB formance now
even though Cosmos DB is a document
database I can plug it into power bi and
pull up rich metrics and data and
filters so what I've got here is where
my redirects are coming from I'm just
doing one day so I'm saying where are
the most click throughs what hour of the
day is popular and these are my popular
links and I can come into the filter for
example and let's look at change this
from last one day to last one month and
apply the filter and now you can see
this is Sunday Monday Tuesday so
Wednesday wins for click throughs this 4
o'clock UTC which is around noon Eastern
Time is popular for clicking during time
of day and then I've got all my links
here so I get all the information I want
pulled out that Cosmos DB cool very nice
you've got the whole story basically end
to end if you can store data your
flexibility in how you query it you've
got the ability to add power bi
dashboards the top of that data what
more can people one specially you know
just to give it a try
exactly it's working great for me and
again on on the site there's plenty of
quick starts that let you download the
project pre populate the database and
get up and running in seconds all right
sounds good anything else you wanted to
show about this today that's it and
everything I've shown today is open
source on my github we'll put the links
on the site there's the cosmos TV that
will import the data for you from the
USDA it's got that built in as well as
the URL shortening project that I have
here all right well that sounds good and
we really hope people
give cosmos to be a try and thank you so
much for coming on yet again
yeah thank you I'll have you back it's
always a pleasure and again hope folks
give our technology I try at sober we're
trying to show you all the coolest stuff
and this is definitely some of the most
cutting-edge things that we've been
working on so glad to do it
a lot of fun thanks for having me on
I'll make sure to wear my my eyes are
shirt you always gotta wear it's from
now and if you don't worry ban from the
show thank you for watching everybody
have a good one
[Music]