Halo 4 - Services in Azure with Caitie McCaffrey

from hanselminut.com. from hanselminut.com. It's Hansel Minutes, a weekly discussion It's Hansel Minutes, a weekly discussion It's Hansel Minutes, a weekly discussion with web developer and technologist with web developer and technologist with web developer and technologist [music] Scott Hansselman. This is [music] Scott Hansselman. This is [music] Scott Hansselman. This is Lawrence Ryan announcing show number Lawrence Ryan announcing show number Lawrence Ryan announcing show number 358. Recorded live Thursday, February 358. Recorded live Thursday, February 358. Recorded live Thursday, February 7th, 2013. 7th, 2013. 7th, 2013. Support [music] for Hansel minutes is Support [music] for Hansel minutes is Support [music] for Hansel minutes is provided by Teleric offering the best in provided by Teleric offering the best in provided by Teleric offering the best in developer tools and support online at developer tools and support online at developer tools and support online at teik.com teik.com teik.com and [music] by franklens.net training and [music] by franklens.net training and [music] by franklens.net training developers to work smarter and now developers to work smarter and now developers to work smarter and now offering gesture pack a powerful gesture offering gesture pack a powerful gesture offering gesture pack a powerful gesture [music] recording and recognition system [music] recording and recognition system [music] recording and recognition system for Microsoft connect for Windows for Microsoft connect for Windows for Microsoft connect for Windows developers. Details at gesturepakk.com. developers. Details at gesturepakk.com. developers. Details at gesturepakk.com. [music] [music] [music] In this episode, Scott talks to 343 In this episode, Scott talks to 343 In this episode, Scott talks to 343 Studios engineer Katie McAffrey about Studios engineer Katie McAffrey about Studios engineer Katie McAffrey about how they use the cloud to run services how they use the cloud to run services how they use the cloud to run services for Halo [music] 4. Hi, this is Scott Hanselman. This is Hi, this is Scott Hanselman. This is another episode of Hansel Minutes and another episode of Hansel Minutes and another episode of Hansel Minutes and it's Halo week. I just made that up, but it's Halo week. I just made that up, but it's Halo week. I just made that up, but it would be cool if it was Halo week. it would be cool if it was Halo week. it would be cool if it was Halo week. Uh, but I do have on the phone uh Katie Uh, but I do have on the phone uh Katie Uh, but I do have on the phone uh Katie McAffrey from Microsoft who also works McAffrey from Microsoft who also works McAffrey from Microsoft who also works for the internal studio 343 Industries.

for the internal studio 343 Industries. for the internal studio 343 Industries. How are you, Katie? How are you, Katie? How are you, Katie? >> I'm good. How are you? >> I'm good. How are you? >> I'm good. How are you? >> Very cool. Very cool. I met you at the >> Very cool. Very cool. I met you at the >> Very cool. Very cool. I met you at the uh at the nerd dinner that we had a uh at the nerd dinner that we had a uh at the nerd dinner that we had a couple of weeks ago at the mall, which couple of weeks ago at the mall, which couple of weeks ago at the mall, which is very exciting. is very exciting. is very exciting. >> Yeah, that was cool. >> Yeah, that was cool. >> Yeah, that was cool. >> And uh and you were up there with my >> And uh and you were up there with my >> And uh and you were up there with my buddy Hoop. And you guys work on uh the buddy Hoop. And you guys work on uh the buddy Hoop. And you guys work on uh the back end of Halo. Is that right? back end of Halo. Is that right? back end of Halo. Is that right? >> Yep. Yep. We work on all the services >> Yep. Yep. We work on all the services >> Yep. Yep. We work on all the services that power all the experiences and the that power all the experiences and the that power all the experiences and the Halo games. Halo games. Halo games. >> Okay. So, I'm looking at I've got Halo >> Okay. So, I'm looking at I've got Halo >> Okay. So, I'm looking at I've got Halo Reach, I've got Halo 3, Halo 4, I've got Reach, I've got Halo 3, Halo 4, I've got Reach, I've got Halo 3, Halo 4, I've got uh Halo uh Anniversary Edition. I just uh Halo uh Anniversary Edition. I just uh Halo uh Anniversary Edition. I just got got got >> all of those have backend services. >> all of those have backend services. >> all of those have backend services. >> Yep, they all have back-end services. >> Yep, they all have back-end services. >> Yep, they all have back-end services. And in addition, um we use Halo, we And in addition, um we use Halo, we And in addition, um we use Halo, we power Halo Waypoint and um all the power Halo Waypoint and um all the power Halo Waypoint and um all the mobile experiences, the Xbox client, um mobile experiences, the Xbox client, um mobile experiences, the Xbox client, um things like that. things like that. things like that. >> Okay. So, I'm I mean, I'm I'm kind of >> Okay. So, I'm I mean, I'm I'm kind of >> Okay. So, I'm I mean, I'm I'm kind of ignorant to this. I know that when I do ignorant to this. I know that when I do ignorant to this. I know that when I do multiplayer, multiplayer, multiplayer, um it's keeping track of my profile and um it's keeping track of my profile and um it's keeping track of my profile and my really really low rank um and my really really low rank um and my really really low rank um and [clears throat] my horrible horrible [clears throat] my horrible horrible [clears throat] my horrible horrible scores. Uh, and I assume that it's also scores. Uh, and I assume that it's also scores. Uh, and I assume that it's also sending real-time information about sending real-time information about sending real-time information about where I am and what direction I'm where I am and what direction I'm where I am and what direction I'm looking and stuff like that. But what looking and stuff like that. But what looking and stuff like that. But what are the services that Halo needs to to are the services that Halo needs to to are the services that Halo needs to to operate?

operate? operate? >> Sure. So, we have a bunch of different >> Sure. So, we have a bunch of different >> Sure. So, we have a bunch of different services that we're currently um that services that we're currently um that services that we're currently um that are evolving over time. We have the are evolving over time. We have the are evolving over time. We have the stats service which records um all of stats service which records um all of stats service which records um all of your stats data. So, like your your stats data. So, like your your stats data. So, like your headshots, your kills, um you know, like headshots, your kills, um you know, like headshots, your kills, um you know, like who killed who with what weapon. That who killed who with what weapon. That who killed who with what weapon. That all gets sent up to us and we keep track all gets sent up to us and we keep track all gets sent up to us and we keep track of that and make it available um on our of that and make it available um on our of that and make it available um on our waypoint clients. And also some of that waypoint clients. And also some of that waypoint clients. And also some of that data is available in game. Um we have a data is available in game. Um we have a data is available in game. Um we have a presence service. So um when you are presence service. So um when you are presence service. So um when you are playing Halo online, you are telling us playing Halo online, you are telling us playing Halo online, you are telling us periodically what you're doing and we periodically what you're doing and we periodically what you're doing and we make this information available on some make this information available on some make this information available on some of our waypoint clients, but mostly in of our waypoint clients, but mostly in of our waypoint clients, but mostly in game. You see this in the in the active game. You see this in the in the active game. You see this in the in the active roster on Halo. Um this is actually a roster on Halo. Um this is actually a roster on Halo. Um this is actually a different service than uh most Xbox Live different service than uh most Xbox Live different service than uh most Xbox Live titles use. It's a little more robust. titles use. It's a little more robust. titles use. It's a little more robust. We update more frequently and send more We update more frequently and send more We update more frequently and send more data. Um, we also have a title files data. Um, we also have a title files data. Um, we also have a title files service. So, this is where you get all service. So, this is where you get all service. So, this is where you get all your hopper files, so your different your hopper files, so your different your hopper files, so your different playlists, which tells you what maps and playlists, which tells you what maps and playlists, which tells you what maps and modes you can play. Um, in Halo 4, this modes you can play. Um, in Halo 4, this modes you can play. Um, in Halo 4, this is how we deliver like some of the is how we deliver like some of the is how we deliver like some of the Spartan ops playlists.

Spartan ops playlists. Spartan ops playlists. >> Okay. So, slow down cuz you you're you >> Okay. So, slow down cuz you you're you >> Okay. So, slow down cuz you you're you work on Halo all the time and I play it work on Halo all the time and I play it work on Halo all the time and I play it like twice a year. So title files, like twice a year. So title files, like twice a year. So title files, that's like when I go in and I say, I that's like when I go in and I say, I that's like when I go in and I say, I want to play uh deathmatch or I want to want to play uh deathmatch or I want to want to play uh deathmatch or I want to play whatever with the skulls and those play whatever with the skulls and those play whatever with the skulls and those graphics that come down. graphics that come down. graphics that come down. >> Yeah. So it's it's basically just a list >> Yeah. So it's it's basically just a list >> Yeah. So it's it's basically just a list of different maps and modes that you can of different maps and modes that you can of different maps and modes that you can play and matchmake into. So um we update play and matchmake into. So um we update play and matchmake into. So um we update those um weekly right now in Halo 4. those um weekly right now in Halo 4. those um weekly right now in Halo 4. >> So like we didn't originally launch with >> So like we didn't originally launch with >> So like we didn't originally launch with SWAT, I believe, which is just a game SWAT, I believe, which is just a game SWAT, I believe, which is just a game type in the Halo universe. And so a type in the Halo universe. And so a type in the Halo universe. And so a couple weeks later, we then um made a couple weeks later, we then um made a couple weeks later, we then um made a SWAT playlist cuz people wanted to play SWAT playlist cuz people wanted to play SWAT playlist cuz people wanted to play SWAT as well. Um so it's just it's and SWAT as well. Um so it's just it's and SWAT as well. Um so it's just it's and and so when we release new map packs, and so when we release new map packs, and so when we release new map packs, we'll also release new um playlists we'll also release new um playlists we'll also release new um playlists because that will make the the new maps because that will make the the new maps because that will make the the new maps that just came out available for people that just came out available for people that just came out available for people who want to play on them. who want to play on them. who want to play on them. >> And these things like stats and >> And these things like stats and >> And these things like stats and presence. There's there's like the real presence. There's there's like the real presence. There's there's like the real time multiplayer aspect of services, but time multiplayer aspect of services, but time multiplayer aspect of services, but then stats like you know how many times then stats like you know how many times then stats like you know how many times you've shot someone with a shotgun. Are you've shot someone with a shotgun. Are you've shot someone with a shotgun. Are those things being calculated on the those things being calculated on the those things being calculated on the server or on the client?

server or on the client? server or on the client? >> Um, so some of it's it's a it's a mix. >> Um, so some of it's it's a it's a mix. >> Um, so some of it's it's a it's a mix. We get uh we get some aggregates on from We get uh we get some aggregates on from We get uh we get some aggregates on from the client for what you did in the game, the client for what you did in the game, the client for what you did in the game, but your lifetime of like kills and but your lifetime of like kills and but your lifetime of like kills and deaths and all of that is being deaths and all of that is being deaths and all of that is being aggregated on the server. Um, it's also aggregated on the server. Um, it's also aggregated on the server. Um, it's also kind of interesting to point out that kind of interesting to point out that kind of interesting to point out that our stats service is not real time. Um, our stats service is not real time. Um, our stats service is not real time. Um, so you play a game and then we get your so you play a game and then we get your so you play a game and then we get your stats and we may be able to process it stats and we may be able to process it stats and we may be able to process it right away, but we we also have the right away, but we we also have the right away, but we we also have the ability to wait if we're having a lot of ability to wait if we're having a lot of ability to wait if we're having a lot of load. Um, presence on the other hand is load. Um, presence on the other hand is load. Um, presence on the other hand is completely real time. So everything completely real time. So everything completely real time. So everything you're seeing there is is up to date and you're seeing there is is up to date and you're seeing there is is up to date and happening right now. happening right now. happening right now. >> That's really interesting because I was >> That's really interesting because I was >> That's really interesting because I was I was talking to the Stack Overflow guys I was talking to the Stack Overflow guys I was talking to the Stack Overflow guys before and you know how Stack Overflow before and you know how Stack Overflow before and you know how Stack Overflow has achievements and some of the has achievements and some of the has achievements and some of the achievements are achievements are achievements are >> super complicated. you know, it's just >> super complicated. you know, it's just >> super complicated. you know, it's just like I don't know, you know, got like I don't know, you know, got like I don't know, you know, got downvoted on a Tuesday by a guy from downvoted on a Tuesday by a guy from downvoted on a Tuesday by a guy from Russia and it's like, whoa, how did they Russia and it's like, whoa, how did they Russia and it's like, whoa, how did they do that? You know, and they they said do that? You know, and they they said do that? You know, and they they said that they've got these triggers where, that they've got these triggers where, that they've got these triggers where, you know, something gets set up and then you know, something gets set up and then you know, something gets set up and then they'll fire off a trigger and that's they'll fire off a trigger and that's they'll fire off a trigger and that's kind of real time, but then they've got kind of real time, but then they've got kind of real time, but then they've got batches that'll run. Some might be every batches that'll run. Some might be every batches that'll run. Some might be every hour and some might be once a week hour and some might be once a week hour and some might be once a week depending on the complexity of the depending on the complexity of the depending on the complexity of the calculation of the achievement. Is is calculation of the achievement. Is is calculation of the achievement. Is is your system similar?

your system similar? your system similar? Um it's it's it's not like we're running Um it's it's it's not like we're running Um it's it's it's not like we're running batch jobs on SQL is sort of the analogy batch jobs on SQL is sort of the analogy batch jobs on SQL is sort of the analogy you're making there. Um it's we use the you're making there. Um it's we use the you're making there. Um it's we use the way what we do in stat service and is we way what we do in stat service and is we way what we do in stat service and is we use um service bus Azure service bus use um service bus Azure service bus use um service bus Azure service bus cues um that we get them from the game. cues um that we get them from the game. cues um that we get them from the game. We get your data and then we send back We get your data and then we send back We get your data and then we send back as soon as we've stored it durably. We as soon as we've stored it durably. We as soon as we've stored it durably. We send back a okay we got your data game send back a okay we got your data game send back a okay we got your data game to or response to the game and then the to or response to the game and then the to or response to the game and then the players can keep playing without having players can keep playing without having players can keep playing without having to wait for us to do a more complex to wait for us to do a more complex to wait for us to do a more complex operation. Once we do start processing operation. Once we do start processing operation. Once we do start processing your stats though, we process and update your stats though, we process and update your stats though, we process and update everything at once. So it's it's just a everything at once. So it's it's just a everything at once. So it's it's just a transaction. Oh, transaction. Oh, transaction. Oh, >> okay. So you're using So the interesting >> okay. So you're using So the interesting >> okay. So you're using So the interesting part there is you're using Azure and part there is you're using Azure and part there is you're using Azure and you're using Azure service bus. So you're using Azure service bus. So you're using Azure service bus. So you're kind of using it in a in a you're kind of using it in a in a you're kind of using it in a in a queuing sense. You throw it into the queuing sense. You throw it into the queuing sense. You throw it into the queue, fire and forget so it doesn't queue, fire and forget so it doesn't queue, fire and forget so it doesn't slow down the game. slow down the game. slow down the game. >> Exactly. >> Exactly. >> Exactly. >> And then the server gets around to it >> And then the server gets around to it >> And then the server gets around to it and if the queue gets full, well then and if the queue gets full, well then and if the queue gets full, well then you know you'll you'll get to it when you know you'll you'll get to it when you know you'll you'll get to it when you get to it. you get to it. you get to it. >> Exactly. Exactly. >> Exactly. Exactly. >> Exactly. Exactly. >> Okay, that's cool. So stats, presence, >> Okay, that's cool. So stats, presence, >> Okay, that's cool. So stats, presence, title files. Uh yeah, we have a couple title files. Uh yeah, we have a couple title files. Uh yeah, we have a couple more. We have a user generated content more. We have a user generated content more. We have a user generated content service. This is whenever you take a service. This is whenever you take a service. This is whenever you take a screenshot or you make a map in um the screenshot or you make a map in um the screenshot or you make a map in um the world editor called Forge. Uh you can world editor called Forge. Uh you can world editor called Forge. Uh you can upload these things to our service and upload these things to our service and upload these things to our service and then they're available via waypoint or then they're available via waypoint or then they're available via waypoint or other players can download them um as other players can download them um as other players can download them um as well. So we have a service for that. Um, well. So we have a service for that. Um, well. So we have a service for that. Um, and then we also have a cheating and and then we also have a cheating and and then we also have a cheating and banning service which if you are doing banning service which if you are doing banning service which if you are doing bad things in the game from, you know, bad things in the game from, you know, bad things in the game from, you know, quitting or griefing other players or quitting or griefing other players or quitting or griefing other players or being obnoxious on your microphone, we being obnoxious on your microphone, we being obnoxious on your microphone, we detect all of these things and then we

detect all of these things and then we detect all of these things and then we can actually automatically ban you um in can actually automatically ban you um in can actually automatically ban you um in real time. We'll semi-real time and uh real time. We'll semi-real time and uh real time. We'll semi-real time and uh and keep the the sort of the integrity and keep the the sort of the integrity and keep the the sort of the integrity of the Halo matchmaking experience good of the Halo matchmaking experience good of the Halo matchmaking experience good for most players. for most players. for most players. >> Wow, you can really tell if I'm being >> Wow, you can really tell if I'm being >> Wow, you can really tell if I'm being mean. mean. mean. >> Yeah. So, we we do we look at a bunch of >> Yeah. So, we we do we look at a bunch of >> Yeah. So, we we do we look at a bunch of different things like um you know, are different things like um you know, are different things like um you know, are players muting you? Are a lot of people players muting you? Are a lot of people players muting you? Are a lot of people disconnecting from games when you're the disconnecting from games when you're the disconnecting from games when you're the host? A lot we have a lot of different host? A lot we have a lot of different host? A lot we have a lot of different metrics that we use to to run and metrics that we use to to run and metrics that we use to to run and determine determine determine >> uh what we should ban people. And we >> uh what we should ban people. And we >> uh what we should ban people. And we have a bunch of flexibility in our have a bunch of flexibility in our have a bunch of flexibility in our system from we can ban you for a day if system from we can ban you for a day if system from we can ban you for a day if we think like maybe you don't really we think like maybe you don't really we think like maybe you don't really aren't being really mean or like doing aren't being really mean or like doing aren't being really mean or like doing it inadvertently or if you keep doing it inadvertently or if you keep doing it inadvertently or if you keep doing it, we can ban you for longer. And and it, we can ban you for longer. And and it, we can ban you for longer. And and this was a system that was also in um this was a system that was also in um this was a system that was also in um Halo Reach and I believe three as well. Halo Reach and I believe three as well. Halo Reach and I believe three as well. >> Wow. Okay. So, you're not like listening >> Wow. Okay. So, you're not like listening >> Wow. Okay. So, you're not like listening to my conversation and like, oh, he said to my conversation and like, oh, he said to my conversation and like, oh, he said a naughty word. You're you're looking at a naughty word. You're you're looking at a naughty word. You're you're looking at how people react to me. how people react to me. how people react to me. >> Yep. Yep. We're using a lot of the user >> Yep. Yep. We're using a lot of the user >> Yep. Yep. We're using a lot of the user generated data that's sent up to us. generated data that's sent up to us. generated data that's sent up to us. >> That's cool. This sounds like a lot of >> That's cool. This sounds like a lot of >> That's cool. This sounds like a lot of compute work.

compute work. compute work. >> Yeah. So, cheating and banning is really >> Yeah. So, cheating and banning is really >> Yeah. So, cheating and banning is really interesting. And um it's kind of it's interesting. And um it's kind of it's interesting. And um it's kind of it's one of the the services that I wrote and one of the the services that I wrote and one of the the services that I wrote and I really enjoyed cuz we uh it used to be I really enjoyed cuz we uh it used to be I really enjoyed cuz we uh it used to be done um on an on a nightly batch basis done um on an on a nightly batch basis done um on an on a nightly batch basis sort of uh like the Stack Overflow sort of uh like the Stack Overflow sort of uh like the Stack Overflow achievements. Um we went and used a achievements. Um we went and used a achievements. Um we went and used a complex invent processing system to do complex invent processing system to do complex invent processing system to do that. Uh so we can do it in more real that. Uh so we can do it in more real that. Uh so we can do it in more real time and so things are happening. You'll time and so things are happening. You'll time and so things are happening. You'll get the fe generally when the system's get the fe generally when the system's get the fe generally when the system's up and running, you'll get the feedback up and running, you'll get the feedback up and running, you'll get the feedback and get banned um you know within 10 to and get banned um you know within 10 to and get banned um you know within 10 to 15 minutes of us detecting that you've 15 minutes of us detecting that you've 15 minutes of us detecting that you've done some a number of undesirable things done some a number of undesirable things done some a number of undesirable things whereas in previous games you would get whereas in previous games you would get whereas in previous games you would get banned uh you know the next day. banned uh you know the next day. banned uh you know the next day. >> Wow. Okay. And you wrote that. So I show >> Wow. Okay. And you wrote that. So I show >> Wow. Okay. And you wrote that. So I show up and I and let's say that I I I join up and I and let's say that I I I join up and I and let's say that I I I join games and I start being a jerk and then games and I start being a jerk and then games and I start being a jerk and then people start leaving and I'm the only people start leaving and I'm the only people start leaving and I'm the only guy left. And this happens a couple guy left. And this happens a couple guy left. And this happens a couple times. times. times. >> This is these are basically stats. These >> This is these are basically stats. These >> This is these are basically stats. These are events that are throwing into a a are events that are throwing into a a are events that are throwing into a a big pile, but they're not correlated big pile, but they're not correlated big pile, but they're not correlated events. They're just that gosh, when events. They're just that gosh, when events. They're just that gosh, when Scott shows up, people run away and Scott shows up, people run away and Scott shows up, people run away and they're sitting in a queue somewhere.

they're sitting in a queue somewhere. they're sitting in a queue somewhere. >> Yeah. So um without getting too much >> Yeah. So um without getting too much >> Yeah. So um without getting too much into the depths of the system, it's it's into the depths of the system, it's it's into the depths of the system, it's it's basically like an event driven basically like an event driven basically like an event driven architecture where we look at all the architecture where we look at all the architecture where we look at all the events that are coming up from the events that are coming up from the events that are coming up from the players um over a period of time and players um over a period of time and players um over a period of time and then then we're processing and then then we're processing and then then we're processing and aggregating a bunch of different events aggregating a bunch of different events aggregating a bunch of different events based on rules that we've decided that based on rules that we've decided that based on rules that we've decided that these that our uh community management these that our uh community management these that our uh community management team has decided this is if these things team has decided this is if these things team has decided this is if these things are happening this player is probably are happening this player is probably are happening this player is probably not playing the way is not is not being not playing the way is not is not being not playing the way is not is not being a good Halo player, right? They're a good Halo player, right? They're a good Halo player, right? They're they're probably doing something wrong. they're probably doing something wrong. they're probably doing something wrong. >> Sure. I I'm not I'm less interested in >> Sure. I I'm not I'm less interested in >> Sure. I I'm not I'm less interested in the uh in like the details like I don't the uh in like the details like I don't the uh in like the details like I don't want to give the griefers like data but want to give the griefers like data but want to give the griefers like data but but I'm interested in like you so you but I'm interested in like you so you but I'm interested in like you so you wrote and this runs in Azure and you wrote and this runs in Azure and you wrote and this runs in Azure and you wrote this in C#. wrote this in C#. wrote this in C#. >> Yep. It runs in Azure. Um we used a >> Yep. It runs in Azure. Um we used a >> Yep. It runs in Azure. Um we used a combination of this couples with our combination of this couples with our combination of this couples with our stat service that gets all the events stat service that gets all the events stat service that gets all the events from the game. Uh and we also used uh from the game. Uh and we also used uh from the game. Uh and we also used uh stream insight in Azure. stream insight in Azure. stream insight in Azure. >> Mhm. when you when you write something >> Mhm. when you when you write something >> Mhm. when you when you write something like this that is big big scale giant like this that is big big scale giant like this that is big big scale giant scale how do you keep um when you're scale how do you keep um when you're scale how do you keep um when you're working this how do you keep from asking working this how do you keep from asking working this how do you keep from asking for too much data like you're asking for too much data like you're asking for too much data like you're asking questions of a back-end system that questions of a back-end system that questions of a back-end system that could explode you know it could get like could explode you know it could get like could explode you know it could get like combinotaurics or order you know n plus1 combinotaurics or order you know n plus1 combinotaurics or order you know n plus1 type questions where it's like oh man I type questions where it's like oh man I type questions where it's like oh man I didn't realize that I just brought the didn't realize that I just brought the didn't realize that I just brought the entire system to its knees because I entire system to its knees because I entire system to its knees because I asked a question that I thought was asked a question that I thought was asked a question that I thought was reasonable but it turned out to be a reasonable but it turned out to be a reasonable but it turned out to be a terabyte of data terabyte of data terabyte of data >> so there so there's two actually we have >> so there so there's two actually we have >> so there so there's two actually we have a we have a BI system as well that I a we have a BI system as well that I a we have a BI system as well that I haven't mentioned that's you know haven't mentioned that's you know haven't mentioned that's you know running Hadoop and in also in Azure. Um running Hadoop and in also in Azure. Um running Hadoop and in also in Azure. Um and that's interesting because that sort and that's interesting because that sort and that's interesting because that sort of we have our data in a different

of we have our data in a different of we have our data in a different store. So that's sort of how we prevent store. So that's sort of how we prevent store. So that's sort of how we prevent like taking down the stat service when like taking down the stat service when like taking down the stat service when we're running a really long pulling a we're running a really long pulling a we're running a really long pulling a long query that pulls a lot of data. long query that pulls a lot of data. long query that pulls a lot of data. >> Um >> Um >> Um >> for GDN banning the interesting thing >> for GDN banning the interesting thing >> for GDN banning the interesting thing with complex event processing is it's with complex event processing is it's with complex event processing is it's more it's the model is sort of like a more it's the model is sort of like a more it's the model is sort of like a normal thing is you have all the data normal thing is you have all the data normal thing is you have all the data and then you run queries over it. With and then you run queries over it. With and then you run queries over it. With complex event processing, you define the complex event processing, you define the complex event processing, you define the query and then you run a bunch of data query and then you run a bunch of data query and then you run a bunch of data through the query. through the query. through the query. >> And so, um, it in the temp the data >> And so, um, it in the temp the data >> And so, um, it in the temp the data needs to be temporal in nature, but needs to be temporal in nature, but needs to be temporal in nature, but you're not going to have this problem you're not going to have this problem you're not going to have this problem where you're trying to pull a bunch of where you're trying to pull a bunch of where you're trying to pull a bunch of data and then like, you know, hitting data and then like, you know, hitting data and then like, you know, hitting your servers really hard. We're just your servers really hard. We're just your servers really hard. We're just consuming a a feed of data from the game consuming a a feed of data from the game consuming a a feed of data from the game and then, you know, creating higher and then, you know, creating higher and then, you know, creating higher level events on top of it and then level events on top of it and then level events on top of it and then acting on those. acting on those. acting on those. >> I see. So, it's like a seieve that >> I see. So, it's like a seieve that >> I see. So, it's like a seieve that you're pouring this kind of fire hose of you're pouring this kind of fire hose of you're pouring this kind of fire hose of data through and you're you're kind of data through and you're you're kind of data through and you're you're kind of seeing if the cheaters fall out of that seeing if the cheaters fall out of that seeing if the cheaters fall out of that as opposed to throwing a bunch of data as opposed to throwing a bunch of data as opposed to throwing a bunch of data in a pile and then shoving your hands in a pile and then shoving your hands in a pile and then shoving your hands through it and looking for this this through it and looking for this this through it and looking for this this cheater in the haystack. cheater in the haystack. cheater in the haystack. >> Yes. Yes. So, it's it's it's sort of a >> Yes. Yes. So, it's it's it's sort of a >> Yes. Yes. So, it's it's it's sort of a different way of um processing large different way of um processing large different way of um processing large amounts of data.

amounts of data. amounts of data. >> Mhm. That's cool. Okay. So, stats, >> Mhm. That's cool. Okay. So, stats, >> Mhm. That's cool. Okay. So, stats, presence, title files, userenary presence, title files, userenary presence, title files, userenary content, cheating and banning. That content, cheating and banning. That content, cheating and banning. That sounds super interesting. Uh when when I sounds super interesting. Uh when when I sounds super interesting. Uh when when I log in, I'm talking to Xbox Live. That's log in, I'm talking to Xbox Live. That's log in, I'm talking to Xbox Live. That's not part of Halo. not part of Halo. not part of Halo. >> Exactly. So, Xbox Live provides a set of >> Exactly. So, Xbox Live provides a set of >> Exactly. So, Xbox Live provides a set of um standard services like that allows um standard services like that allows um standard services like that allows the game to find our servers as well and the game to find our servers as well and the game to find our servers as well and talk to our um our services. It also talk to our um our services. It also talk to our um our services. It also provides things like when the provides things like when the provides things like when the matchmaking that is running is actually matchmaking that is running is actually matchmaking that is running is actually running through Xbox Live, achievements running through Xbox Live, achievements running through Xbox Live, achievements run through Xbox Live, things like that. run through Xbox Live, things like that. run through Xbox Live, things like that. >> Mhm. And do you uh how do you deploy >> Mhm. And do you uh how do you deploy >> Mhm. And do you uh how do you deploy this stuff? Like you have like staging this stuff? Like you have like staging this stuff? Like you have like staging where like people who are working on where like people who are working on where like people who are working on this are playing, you know, either a this are playing, you know, either a this are playing, you know, either a version of Halo or they've pointed version of Halo or they've pointed version of Halo or they've pointed they've pointed that version of Halo at they've pointed that version of Halo at they've pointed that version of Halo at the staging server and then you can the staging server and then you can the staging server and then you can simulate a bunch of bad guys being bad simulate a bunch of bad guys being bad simulate a bunch of bad guys being bad without being on the live network and without being on the live network and without being on the live network and then you roll it over into production then you roll it over into production then you roll it over into production later. later. later. >> Um yeah, so we use the Azure portal to >> Um yeah, so we use the Azure portal to >> Um yeah, so we use the Azure portal to do a lot of our deployments and we also do a lot of our deployments and we also do a lot of our deployments and we also use um this cool tool called uh Sarah use um this cool tool called uh Sarah use um this cool tool called uh Sarah Brad. They have this thing called the Brad. They have this thing called the Brad. They have this thing called the cloud storage studio and Azure cloud storage studio and Azure cloud storage studio and Azure management studio that it looks like the management studio that it looks like the management studio that it looks like the portal but it has a lot of nice UI portal but it has a lot of nice UI portal but it has a lot of nice UI features on top of it that makes um features on top of it that makes um features on top of it that makes um deploying builds um editing config doing deploying builds um editing config doing deploying builds um editing config doing a VIP swap type of thing in Azure really a VIP swap type of thing in Azure really a VIP swap type of thing in Azure really simple simple simple >> um right now yeah we deploy to staging >> um right now yeah we deploy to staging >> um right now yeah we deploy to staging and then do a VIP swap uh to change out and then do a VIP swap uh to change out and then do a VIP swap uh to change out our builds in production.

our builds in production. our builds in production. >> What is a VIP swap? Oh, so in Azure you >> What is a VIP swap? Oh, so in Azure you >> What is a VIP swap? Oh, so in Azure you for every sort of um subscription ID you for every sort of um subscription ID you for every sort of um subscription ID you have the production deployment which is have the production deployment which is have the production deployment which is what is hit uh available via your your what is hit uh available via your your what is hit uh available via your your nice friendly DNS name and then you have nice friendly DNS name and then you have nice friendly DNS name and then you have a staging deployment which is only a staging deployment which is only a staging deployment which is only accessible via a gooid um and like isn't accessible via a gooid um and like isn't accessible via a gooid um and like isn't what the Azure load balancer is sending what the Azure load balancer is sending what the Azure load balancer is sending work to. Um, and so what we do is we work to. Um, and so what we do is we work to. Um, and so what we do is we spin up actually two full deployments at spin up actually two full deployments at spin up actually two full deployments at once in staging. And then a VIP swap is once in staging. And then a VIP swap is once in staging. And then a VIP swap is when you tell Azure to um change what's when you tell Azure to um change what's when you tell Azure to um change what's in staging with what's in production. So in staging with what's in production. So in staging with what's in production. So it just does an atomic swap. it just does an atomic swap. it just does an atomic swap. >> Is VIP is VIP like VIP like virtual IP? >> Is VIP is VIP like VIP like virtual IP? >> Is VIP is VIP like VIP like virtual IP? >> Yeah. >> Yeah. >> Yeah. >> Okay. So you you don't really uh like >> Okay. So you you don't really uh like >> Okay. So you you don't really uh like when I'm when I was used to doing when I'm when I was used to doing when I'm when I was used to doing staging to production roles in the in staging to production roles in the in staging to production roles in the in the pre-cloud days, there were a bunch the pre-cloud days, there were a bunch the pre-cloud days, there were a bunch of machines that I could point at and of machines that I could point at and of machines that I could point at and say, "Look, those are staging and then say, "Look, those are staging and then say, "Look, those are staging and then these are over here are production." And these are over here are production." And these are over here are production." And they were very different. And we they were very different. And we they were very different. And we wouldn't typically swap. It would wouldn't typically swap. It would wouldn't typically swap. It would usually be a roll forward thing. And you usually be a roll forward thing. And you usually be a roll forward thing. And you don't usually get into swapping don't usually get into swapping don't usually get into swapping comfortably between two systems and comfortably between two systems and comfortably between two systems and calling them both production until calling them both production until calling them both production until you're a very mature organization. But you're a very mature organization. But you're a very mature organization. But it sounds like with the cloud you don't it sounds like with the cloud you don't it sounds like with the cloud you don't you don't really care, right? I mean you don't really care, right? I mean you don't really care, right? I mean they they literally production and they they literally production and they they literally production and staging can be the same thing because staging can be the same thing because staging can be the same thing because you are going to just swap back and you are going to just swap back and you are going to just swap back and forth.

forth. forth. >> Exactly. It's it's it's a very different >> Exactly. It's it's it's a very different >> Exactly. It's it's it's a very different deployment model and it's um it's also deployment model and it's um it's also deployment model and it's um it's also very interesting when you think about very interesting when you think about very interesting when you think about designing your services on top of this designing your services on top of this designing your services on top of this because you have to basically be super because you have to basically be super because you have to basically be super fault tolerant because we're taking down fault tolerant because we're taking down fault tolerant because we're taking down the production service and there may be the production service and there may be the production service and there may be data in it and we want that data to data in it and we want that data to data in it and we want that data to replay, right? We don't want to lose any replay, right? We don't want to lose any replay, right? We don't want to lose any data. Um but at the same time we don't data. Um but at the same time we don't data. Um but at the same time we don't want to have to like expert we don't want to have to like expert we don't want to have to like expert we don't expertly like time when we do our um our expertly like time when we do our um our expertly like time when we do our um our deployments. We can do them at any time deployments. We can do them at any time deployments. We can do them at any time and it just sort of works. Um and so and it just sort of works. Um and so and it just sort of works. Um and so there's a lot of things from you know there's a lot of things from you know there's a lot of things from you know like item potency to being super fault like item potency to being super fault like item potency to being super fault tolerant that are are helpful in doing tolerant that are are helpful in doing tolerant that are are helpful in doing that. that. that. >> When uh when we would have people on >> When uh when we would have people on >> When uh when we would have people on like a shopping cart uh in the apps that like a shopping cart uh in the apps that like a shopping cart uh in the apps that I worked on, we would start to kind of I worked on, we would start to kind of I worked on, we would start to kind of slowly drain them off. So we would slowly drain them off. So we would slowly drain them off. So we would slowly bring one production site down slowly bring one production site down slowly bring one production site down and slowly bring the other one back up. and slowly bring the other one back up. and slowly bring the other one back up. Mhm. Mhm. Mhm. >> Is that is that similar kind of thing or >> Is that is that similar kind of thing or >> Is that is that similar kind of thing or do you are you literally swapping the do you are you literally swapping the do you are you literally swapping the entire thing out from underneath them entire thing out from underneath them entire thing out from underneath them and they and and the cues keep it from and they and and the cues keep it from and they and and the cues keep it from um feeling like something's happened? um feeling like something's happened? um feeling like something's happened? >> Yep. Exactly. We just we just swap them >> Yep. Exactly. We just we just swap them >> Yep. Exactly. We just we just swap them in and out. Um we don't really do any in and out. Um we don't really do any in and out. Um we don't really do any drain off right now.

drain off right now. drain off right now. >> Wow. And then and you will you do that >> Wow. And then and you will you do that >> Wow. And then and you will you do that like on a random Tuesday afternoon or do like on a random Tuesday afternoon or do like on a random Tuesday afternoon or do you wait until like the middle of the you wait until like the middle of the you wait until like the middle of the night cuz someone's always playing Halo night cuz someone's always playing Halo night cuz someone's always playing Halo somewhere. somewhere. somewhere. >> Exactly. Yeah. So we do deployments. We >> Exactly. Yeah. So we do deployments. We >> Exactly. Yeah. So we do deployments. We try to do like really impactful ones um try to do like really impactful ones um try to do like really impactful ones um at night just not we don't so we don't at night just not we don't so we don't at night just not we don't so we don't get up at like 2 a.m. and do get up at like 2 a.m. and do get up at like 2 a.m. and do deployments. We do them generally during deployments. We do them generally during deployments. We do them generally during the day when we're here. Mhm. the day when we're here. Mhm. the day when we're here. Mhm. >> Um and then launch week was its own >> Um and then launch week was its own >> Um and then launch week was its own thing, special thing because um I guess thing, special thing because um I guess thing, special thing because um I guess it's it's notable in um game services. it's it's notable in um game services. it's it's notable in um game services. You have this very um different load You have this very um different load You have this very um different load pattern where on you know launch night pattern where on you know launch night pattern where on you know launch night or launch week in the first couple weeks or launch week in the first couple weeks or launch week in the first couple weeks you see your highest number of you see your highest number of you see your highest number of concurrent users and then it sort of concurrent users and then it sort of concurrent users and then it sort of tapers off and and and becomes more tapers off and and and becomes more tapers off and and and becomes more stable. Um whereas most services you stable. Um whereas most services you stable. Um whereas most services you sort of have like this gradual ramp up sort of have like this gradual ramp up sort of have like this gradual ramp up to your peak number of users. So to um to your peak number of users. So to um to your peak number of users. So to um sort of help with this on launch week, sort of help with this on launch week, sort of help with this on launch week, we had at least five people in the we had at least five people in the we had at least five people in the office around the clock. We were working office around the clock. We were working office around the clock. We were working like 8 to nine hour shifts and then you like 8 to nine hour shifts and then you like 8 to nine hour shifts and then you know addressing issues and monitoring know addressing issues and monitoring know addressing issues and monitoring when they came up. And so that week we when they came up. And so that week we when they came up. And so that week we were just like fixing things and and were just like fixing things and and were just like fixing things and and switching in things like you know at switching in things like you know at switching in things like you know at least once or twice a day.

least once or twice a day. least once or twice a day. >> Does it feel like helpless in that >> Does it feel like helpless in that >> Does it feel like helpless in that you're using this cloud service that you you're using this cloud service that you you're using this cloud service that you didn't make? Like you made what sits on didn't make? Like you made what sits on didn't make? Like you made what sits on top of it but you didn't make Azure. So top of it but you didn't make Azure. So top of it but you didn't make Azure. So like from from the perspective of of you like from from the perspective of of you like from from the perspective of of you guys, the Halo team that you know guys, the Halo team that you know guys, the Halo team that you know there's this company out there called there's this company out there called there's this company out there called Azure. I mean we happen to be Microsoft Azure. I mean we happen to be Microsoft Azure. I mean we happen to be Microsoft but you know it's still it's a thing you but you know it's still it's a thing you but you know it's still it's a thing you didn't make so you can't control it. I didn't make so you can't control it. I didn't make so you can't control it. I always thought about that like Netflix always thought about that like Netflix always thought about that like Netflix uses Amazon and like if Amazon's down or uses Amazon and like if Amazon's down or uses Amazon and like if Amazon's down or something happens something happens something happens >> the Netflix guys are like what do they >> the Netflix guys are like what do they >> the Netflix guys are like what do they do? Just call a 1-800 number or what do do? Just call a 1-800 number or what do do? Just call a 1-800 number or what do you how do you freak out when you can't you how do you freak out when you can't you how do you freak out when you can't like go and reboot the server yourself? like go and reboot the server yourself? like go and reboot the server yourself? >> So there's Right. So there is the Azure >> So there's Right. So there is the Azure >> So there's Right. So there is the Azure support team as well and then because of support team as well and then because of support team as well and then because of our scale um and size we had um you know our scale um and size we had um you know our scale um and size we had um you know some help from the Azure team and people some help from the Azure team and people some help from the Azure team and people that we could call directly um if that that we could call directly um if that that we could call directly um if that happened. I mean but in any system happened. I mean but in any system happened. I mean but in any system there's sort of like right you don't there's sort of like right you don't there's sort of like right you don't really even own your machines or in a really even own your machines or in a really even own your machines or in a data center somewhere and you have data center somewhere and you have data center somewhere and you have people who could go and like you know people who could go and like you know people who could go and like you know kick out a power cord or and and like kick out a power cord or and and like kick out a power cord or and and like obviously this doesn't happen all the obviously this doesn't happen all the obviously this doesn't happen all the time but there there are issues there as time but there there are issues there as time but there there are issues there as well whereas like a drive will just fail well whereas like a drive will just fail well whereas like a drive will just fail and like I can't control that either. So and like I can't control that either. So and like I can't control that either. So um Azure is just a different set of um Azure is just a different set of um Azure is just a different set of problems and and it helps that we problems and and it helps that we problems and and it helps that we basically rebuilt the service stack from basically rebuilt the service stack from basically rebuilt the service stack from the ground up to be on top of Azure. So the ground up to be on top of Azure. So the ground up to be on top of Azure. So we sort of addressed those issues like we sort of addressed those issues like we sort of addressed those issues like right we know from the beginning that right we know from the beginning that right we know from the beginning that they're going to pull they can pull any they're going to pull they can pull any they're going to pull they can pull any VM out from under us and restart it. So VM out from under us and restart it. So VM out from under us and restart it. So we went and built the system with that we went and built the system with that we went and built the system with that in mind.

in mind. in mind. >> That's interesting. So there are people >> That's interesting. So there are people >> That's interesting. So there are people who might move an existing system over who might move an existing system over who might move an existing system over and I think there's a lot of discussion and I think there's a lot of discussion and I think there's a lot of discussion around migration and you know moving around migration and you know moving around migration and you know moving this website or that website but and this website or that website but and this website or that website but and then there's also concerns about vendor then there's also concerns about vendor then there's also concerns about vendor portability. People are there there are portability. People are there there are portability. People are there there are those who are always paranoid about those who are always paranoid about those who are always paranoid about that. So then they'll build their system that. So then they'll build their system that. So then they'll build their system based entirely on VMs. But I'm hearing based entirely on VMs. But I'm hearing based entirely on VMs. But I'm hearing you say that you looked at the things you say that you looked at the things you say that you looked at the things that Azure offers like compute and that Azure offers like compute and that Azure offers like compute and storage and service bus particularly storage and service bus particularly storage and service bus particularly service bus service bus service bus >> and built it to sit on Azure and to use >> and built it to sit on Azure and to use >> and built it to sit on Azure and to use the Azure stuff and it it doesn't it the Azure stuff and it it doesn't it the Azure stuff and it it doesn't it doesn't have an identity that is not doesn't have an identity that is not doesn't have an identity that is not attached to cloud services. attached to cloud services. attached to cloud services. >> Exactly right. So the this is the first >> Exactly right. So the this is the first >> Exactly right. So the this is the first I guess it's interesting to point out I guess it's interesting to point out I guess it's interesting to point out this is the first game that is built this is the first game that is built this is the first game that is built entirely or Halo game that is built entirely or Halo game that is built entirely or Halo game that is built entirely on top of Azure. our backends entirely on top of Azure. our backends entirely on top of Azure. our backends before used to be in physical data before used to be in physical data before used to be in physical data centers. And so we took uh the our team centers. And so we took uh the our team centers. And so we took uh the our team took on the challenge of rebuilding took on the challenge of rebuilding took on the challenge of rebuilding importing it to Azure because for a importing it to Azure because for a importing it to Azure because for a number of reasons with the the load number of reasons with the the load number of reasons with the the load pattern it makes a lot of sense because pattern it makes a lot of sense because pattern it makes a lot of sense because you can start turning things off as you you can start turning things off as you you can start turning things off as you don't need them. You don't have to buy don't need them. You don't have to buy don't need them. You don't have to buy all this really expensive hardware that all this really expensive hardware that all this really expensive hardware that then goes out of warranty. Um right we then goes out of warranty. Um right we then goes out of warranty. Um right we get the nice elastic scale. Um and so get the nice elastic scale. Um and so get the nice elastic scale. Um and so that was really worthwhile to us as a that was really worthwhile to us as a that was really worthwhile to us as a service and we wanted to do that and it service and we wanted to do that and it service and we wanted to do that and it makes a lot of sense for you know future makes a lot of sense for you know future makes a lot of sense for you know future experiences and and um maintainability experiences and and um maintainability experiences and and um maintainability of the service and so we we did we just of the service and so we we did we just of the service and so we we did we just rebuilt everything on top of Azure and rebuilt everything on top of Azure and rebuilt everything on top of Azure and we rearchitected and re looked at every we rearchitected and re looked at every we rearchitected and re looked at every service even though a lot of them are service even though a lot of them are service even though a lot of them are the same as um have the same names and the same as um have the same names and the same as um have the same names and have the same functionality of what was have the same functionality of what was have the same functionality of what was in um previous Halo games. Um, but we we

in um previous Halo games. Um, but we we in um previous Halo games. Um, but we we went and built it for Azure went and built it for Azure went and built it for Azure specifically. specifically. specifically. >> Did were were there discussions where >> Did were were there discussions where >> Did were were there discussions where you had to say, "Well, do we know if if you had to say, "Well, do we know if if you had to say, "Well, do we know if if Azure is big enough for Halo? Like are Azure is big enough for Halo? Like are Azure is big enough for Halo? Like are you I don't know if you're the biggest you I don't know if you're the biggest you I don't know if you're the biggest client, but I'm just saying like there client, but I'm just saying like there client, but I'm just saying like there must have been a whenever you're must have been a whenever you're must have been a whenever you're evaluating a cloud service, you're like, evaluating a cloud service, you're like, evaluating a cloud service, you're like, is this going to be big enough?" Because is this going to be big enough?" Because is this going to be big enough?" Because we don't know how popular Halo 4 could we don't know how popular Halo 4 could we don't know how popular Halo 4 could have been. It could have been bigger have been. It could have been bigger have been. It could have been bigger than Azure maybe, than Azure maybe, than Azure maybe, >> right? So we definitely had a lot of >> right? So we definitely had a lot of >> right? So we definitely had a lot of conversations and worked with the Azure conversations and worked with the Azure conversations and worked with the Azure team to make sure um right that we could team to make sure um right that we could team to make sure um right that we could get the number of cores that we needed get the number of cores that we needed get the number of cores that we needed that that they were scaling out and so that that they were scaling out and so that that they were scaling out and so that was right great. I mean I think in that was right great. I mean I think in that was right great. I mean I think in we talked with Clemens um from service we talked with Clemens um from service we talked with Clemens um from service bus and you know during one of our Perf bus and you know during one of our Perf bus and you know during one of our Perf tests we actually took um production tests we actually took um production tests we actually took um production service bus cues down and so um but they service bus cues down and so um but they service bus cues down and so um but they worked with us to make sure to fix the worked with us to make sure to fix the worked with us to make sure to fix the problem and make sure that they were problem and make sure that they were problem and make sure that they were ready for launch. ready for launch. ready for launch. >> That's pretty cool. I I I had this >> That's pretty cool. I I I had this >> That's pretty cool. I I I had this conversation with a guy from Netflix. He conversation with a guy from Netflix. He conversation with a guy from Netflix. He was explaining how um they were um his was explaining how um they were um his was explaining how um they were um his cloud service provider was going from cloud service provider was going from cloud service provider was going from spinning Rust hard drives over to SSDs.

spinning Rust hard drives over to SSDs. spinning Rust hard drives over to SSDs. And this guy in the audience And this guy in the audience And this guy in the audience >> uh calls out, well, you know, SSDs fail >> uh calls out, well, you know, SSDs fail >> uh calls out, well, you know, SSDs fail and what about what about this and that and what about what about this and that and what about what about this and that and performance and trim and d and the and performance and trim and d and the and performance and trim and d and the guy from Netflix was like, well, that's guy from Netflix was like, well, that's guy from Netflix was like, well, that's not my problem. Like I'm renting them. not my problem. Like I'm renting them. not my problem. Like I'm renting them. >> Exactly. >> Exactly. >> Exactly. >> And I was like, whoa. That just totally >> And I was like, whoa. That just totally >> And I was like, whoa. That just totally blew my mind. like, "Oh my goodness, blew my mind. like, "Oh my goodness, blew my mind. like, "Oh my goodness, he's so enlightened. I'm renting them." he's so enlightened. I'm renting them." he's so enlightened. I'm renting them." Like, "Yeah, drives fail, but it's not Like, "Yeah, drives fail, but it's not Like, "Yeah, drives fail, but it's not my problem." Like, for for someone like my problem." Like, for for someone like my problem." Like, for for someone like Netflix where popular movie comes out or Netflix where popular movie comes out or Netflix where popular movie comes out or someone like Halo, hey, Halo 4 comes someone like Halo, hey, Halo 4 comes someone like Halo, hey, Halo 4 comes out, I'm going to rent a bunch of hard out, I'm going to rent a bunch of hard out, I'm going to rent a bunch of hard drives and when I'm done and the game drives and when I'm done and the game drives and when I'm done and the game becomes less popular and Halo 4 plus one becomes less popular and Halo 4 plus one becomes less popular and Halo 4 plus one comes out, you know, then I'll just rent comes out, you know, then I'll just rent comes out, you know, then I'll just rent some more. Uh, so like all those some more. Uh, so like all those some more. Uh, so like all those machines you had in data centers like machines you had in data centers like machines you had in data centers like before, you don't need those data before, you don't need those data before, you don't need those data centers. centers. centers. >> Exactly. Okay. Well, so we still do >> Exactly. Okay. Well, so we still do >> Exactly. Okay. Well, so we still do support like all of the old games. We support like all of the old games. We support like all of the old games. We haven't migrated to Azure. haven't migrated to Azure. haven't migrated to Azure. >> Um, so we still own some physical >> Um, so we still own some physical >> Um, so we still own some physical machines and some data centers, but um machines and some data centers, but um machines and some data centers, but um it's really nice for entertainment when it's really nice for entertainment when it's really nice for entertainment when you sort of look at the way that those you sort of look at the way that those you sort of look at the way that those uh the the users sort of like flock to uh the the users sort of like flock to uh the the users sort of like flock to one hot thing of the moment and then it one hot thing of the moment and then it one hot thing of the moment and then it it drops off. But you can still support it drops off. But you can still support it drops off. But you can still support the long tail of the the entertainment the long tail of the the entertainment the long tail of the the entertainment experience in the product because you experience in the product because you experience in the product because you don't have to maintain all these really don't have to maintain all these really don't have to maintain all these really expensive machines. You just keep up the expensive machines. You just keep up the expensive machines. You just keep up the number of cores you need running. Um, number of cores you need running. Um, number of cores you need running. Um, and it it is really nice. I kind of and it it is really nice. I kind of and it it is really nice. I kind of really enjoy like not getting woken up really enjoy like not getting woken up really enjoy like not getting woken up in the middle of the night because, you in the middle of the night because, you in the middle of the night because, you know, the SQL drive failed, right? Azure know, the SQL drive failed, right? Azure know, the SQL drive failed, right? Azure just deals with it and spins up a new just deals with it and spins up a new just deals with it and spins up a new instance of of a node that went down for instance of of a node that went down for instance of of a node that went down for me and I don't even get notified.

me and I don't even get notified. me and I don't even get notified. >> Yeah, exactly. And also, I I think as a >> Yeah, exactly. And also, I I think as a >> Yeah, exactly. And also, I I think as a user of games, I always hate when a game user of games, I always hate when a game user of games, I always hate when a game that I'm that I really dig, I I get a that I'm that I really dig, I I get a that I'm that I really dig, I I get a notification that like, oh yeah, the notification that like, oh yeah, the notification that like, oh yeah, the backend service for this game is going backend service for this game is going backend service for this game is going away now because we can't afford it. You away now because we can't afford it. You away now because we can't afford it. You know, it's like if if you're on like know, it's like if if you're on like know, it's like if if you're on like Halo 15 and I'm still playing Halo 4, uh Halo 15 and I'm still playing Halo 4, uh Halo 15 and I'm still playing Halo 4, uh yeah, you could go and run it on a yeah, you could go and run it on a yeah, you could go and run it on a machine and it's going to cost you guys machine and it's going to cost you guys machine and it's going to cost you guys five bucks a month and it doesn't really five bucks a month and it doesn't really five bucks a month and it doesn't really matter, right? You could keep it running matter, right? You could keep it running matter, right? You could keep it running forever as far as you're concerned. forever as far as you're concerned. forever as far as you're concerned. >> Exactly. So, and that's one of the other >> Exactly. So, and that's one of the other >> Exactly. So, and that's one of the other nice things about this, right, is like nice things about this, right, is like nice things about this, right, is like we we want our users to have a really we we want our users to have a really we we want our users to have a really great experience and we want them to great experience and we want them to great experience and we want them to play what they love. And so, this makes play what they love. And so, this makes play what they love. And so, this makes it really easy to do that. And it makes it really easy to do that. And it makes it really easy to do that. And it makes it really easy to keep around all this it really easy to keep around all this it really easy to keep around all this data and and whatnot. Like, I don't data and and whatnot. Like, I don't data and and whatnot. Like, I don't care. it's just out there and it's care. it's just out there and it's care. it's just out there and it's durable and Azure takes care of a lot of durable and Azure takes care of a lot of durable and Azure takes care of a lot of the the operating systems and the the the operating systems and the the the operating systems and the software upgrades for me. software upgrades for me. software upgrades for me. >> There's uh there is one framework that >> There's uh there is one framework that >> There's uh there is one framework that you're using that I have had not been you're using that I have had not been you're using that I have had not been familiar with that's called Orleans like familiar with that's called Orleans like familiar with that's called Orleans like New Orleans. What what is Orleans? So, New Orleans. What what is Orleans? So, New Orleans. What what is Orleans? So, Orleans is a Microsoft um research pro Orleans is a Microsoft um research pro Orleans is a Microsoft um research pro uh project and it's basically um a uh project and it's basically um a uh project and it's basically um a framework for building large reliable framework for building large reliable framework for building large reliable distributed systems and it's based on distributed systems and it's based on distributed systems and it's based on the actor model and so we partnered with the actor model and so we partnered with the actor model and so we partnered with them really early on and thought this them really early on and thought this them really early on and thought this was a really cool thing and would help was a really cool thing and would help was a really cool thing and would help us build um and write the code much us build um and write the code much us build um and write the code much easier. Um and so we've been working easier. Um and so we've been working easier. Um and so we've been working tightly with that group as well.

tightly with that group as well. tightly with that group as well. >> And I'm looking at this here. They have >> And I'm looking at this here. They have >> And I'm looking at this here. They have this thing called grains. this thing called grains. this thing called grains. What is a grain? A grain is sort of um a What is a grain? A grain is sort of um a What is a grain? A grain is sort of um a logical unit used to uh process data and logical unit used to uh process data and logical unit used to uh process data and all of the the anything that goes any of all of the the anything that goes any of all of the the anything that goes any of the calls that are in a grain run the calls that are in a grain run the calls that are in a grain run singlethreadedly. So everything is singlethreadedly. So everything is singlethreadedly. So everything is asynchronous and non-blocking but it it asynchronous and non-blocking but it it asynchronous and non-blocking but it it it abstracts away the idea of me having it abstracts away the idea of me having it abstracts away the idea of me having to worry about threading within a single to worry about threading within a single to worry about threading within a single grain. So I I can write code grain. So I I can write code grain. So I I can write code synchronously basically in a grain. synchronously basically in a grain. synchronously basically in a grain. >> I see. So, so writing code that is >> I see. So, so writing code that is >> I see. So, so writing code that is synchronously of course is is easier synchronously of course is is easier synchronously of course is is easier because you want to write code that because you want to write code that because you want to write code that focuses on business, focuses on business, focuses on business, >> right? >> right? >> right? >> And if you're writing code and you're >> And if you're writing code and you're >> And if you're writing code and you're sitting on a thread and you need data sitting on a thread and you need data sitting on a thread and you need data and from all sorts of different and from all sorts of different and from all sorts of different locations, you can get your business locations, you can get your business locations, you can get your business code kind of tangled up with the code kind of tangled up with the code kind of tangled up with the business of threading. business of threading. business of threading. >> Yeah. So, this simplifies a lot of the >> Yeah. So, this simplifies a lot of the >> Yeah. So, this simplifies a lot of the threading problems. Um, it also is nice threading problems. Um, it also is nice threading problems. Um, it also is nice because it scales out and does a lot of because it scales out and does a lot of because it scales out and does a lot of the addressing address addressing for the addressing address addressing for the addressing address addressing for us. So when I say, you know, like go us. So when I say, you know, like go us. So when I say, you know, like go post stats to um for Scott, it just goes post stats to um for Scott, it just goes post stats to um for Scott, it just goes I like do the lookup to find where in I like do the lookup to find where in I like do the lookup to find where in this Orleans does the lookup for me like this Orleans does the lookup for me like this Orleans does the lookup for me like where to go send this what machine to go where to go send this what machine to go where to go send this what machine to go send this message to.

send this message to. send this message to. >> Wow. I'm looking up here and there's no >> Wow. I'm looking up here and there's no >> Wow. I'm looking up here and there's no downloads. There's nothing. It's from downloads. There's nothing. It's from downloads. There's nothing. It's from the Extreme Computing Group, the Extreme Computing Group, the Extreme Computing Group, >> which is an awesome name if you're going >> which is an awesome name if you're going >> which is an awesome name if you're going to work for a group. to work for a group. to work for a group. >> Yeah. So, unfortunately, this isn't >> Yeah. So, unfortunately, this isn't >> Yeah. So, unfortunately, this isn't available publicly on Microsoft or for available publicly on Microsoft or for available publicly on Microsoft or for people outside of Microsoft to use, people outside of Microsoft to use, people outside of Microsoft to use, >> but um working with them is like we >> but um working with them is like we >> but um working with them is like we worked very closely with that team and worked very closely with that team and worked very closely with that team and they're really awesome and it's been um they're really awesome and it's been um they're really awesome and it's been um I thought it was a great experience I thought it was a great experience I thought it was a great experience working with it and made our lives a lot working with it and made our lives a lot working with it and made our lives a lot easier built on building on top of their easier built on building on top of their easier built on building on top of their framework. framework. framework. >> Wow. Hopefully that'll that'll come out >> Wow. Hopefully that'll that'll come out >> Wow. Hopefully that'll that'll come out or I'll hear more about that. That or I'll hear more about that. That or I'll hear more about that. That sounds actually pretty sweet. sounds actually pretty sweet. sounds actually pretty sweet. >> Yeah, it's very cool. >> Yeah, it's very cool. >> Yeah, it's very cool. >> That's cool. And and service bus. Um, I >> That's cool. And and service bus. Um, I >> That's cool. And and service bus. Um, I wrote a couple of basic uh service wrote a couple of basic uh service wrote a couple of basic uh service relays and some different stuff, but I relays and some different stuff, but I relays and some different stuff, but I haven't dug too much into service. And I haven't dug too much into service. And I haven't dug too much into service. And I do know that, like you said, Clemens do know that, like you said, Clemens do know that, like you said, Clemens Vasters from that team is always telling Vasters from that team is always telling Vasters from that team is always telling me to get on service bus. And there's me to get on service bus. And there's me to get on service bus. And there's actually a video uh where you're up actually a video uh where you're up actually a video uh where you're up there with Hoop and uh one of your there with Hoop and uh one of your there with Hoop and uh one of your teammates, Angus. teammates, Angus. teammates, Angus. >> Yep. >> Yep. >> Yep. >> Talking about service bus on channel 9, >> Talking about service bus on channel 9, >> Talking about service bus on channel 9, and I'll put a link to that in the uh in and I'll put a link to that in the uh in and I'll put a link to that in the uh in the show notes. How is service bus the show notes. How is service bus the show notes. How is service bus different from just using like MSMQ?

different from just using like MSMQ? different from just using like MSMQ? Um, so I'm not super familiar with MSMQ, Um, so I'm not super familiar with MSMQ, Um, so I'm not super familiar with MSMQ, like looking into that, but we we chose like looking into that, but we we chose like looking into that, but we we chose to use that because a it gives us the to use that because a it gives us the to use that because a it gives us the ability to scale out really easily cuz ability to scale out really easily cuz ability to scale out really easily cuz we wanted to be the main goal there is we wanted to be the main goal there is we wanted to be the main goal there is we wanted to be able to have the server we wanted to be able to have the server we wanted to be able to have the server go down just in case it did for whatever go down just in case it did for whatever go down just in case it did for whatever reason for um I think reason for um I think reason for um I think >> three two to three days and we don't >> three two to three days and we don't >> three two to three days and we don't lose any data. Um, so we use service bus lose any data. Um, so we use service bus lose any data. Um, so we use service bus to build in the reliability of the stat to build in the reliability of the stat to build in the reliability of the stat system and the scale because when on system and the scale because when on system and the scale because when on lunch night, you know, we get a ton of lunch night, you know, we get a ton of lunch night, you know, we get a ton of users coming in playing and perhaps we users coming in playing and perhaps we users coming in playing and perhaps we had issues with stats or um, right, we had issues with stats or um, right, we had issues with stats or um, right, we just like couldn't process it as many just like couldn't process it as many just like couldn't process it as many games as quickly as they were coming in. games as quickly as they were coming in. games as quickly as they were coming in. Service bus gave us the ability to uh, Service bus gave us the ability to uh, Service bus gave us the ability to uh, sort of cue that data, store it durably, sort of cue that data, store it durably, sort of cue that data, store it durably, and then process it as we had time. So, and then process it as we had time. So, and then process it as we had time. So, we didn't just slam our servers. M and we didn't just slam our servers. M and we didn't just slam our servers. M and service bus is is a is a service on service bus is is a is a service on service bus is is a is a service on Azure that is that you're abstracting Azure that is that you're abstracting Azure that is that you're abstracting away what's underneath it. Like you away what's underneath it. Like you away what's underneath it. Like you don't have to think about if there's VMs don't have to think about if there's VMs don't have to think about if there's VMs underneath the service bus, right? If underneath the service bus, right? If underneath the service bus, right? If you want more service buses, you just you want more service buses, you just you want more service buses, you just dial it up, dial it up, dial it up, >> right? So you just um basically say I >> right? So you just um basically say I >> right? So you just um basically say I want this many cues and that gives you want this many cues and that gives you want this many cues and that gives you uh a set amount of data that you can use uh a set amount of data that you can use uh a set amount of data that you can use and um and then you throw things into and um and then you throw things into and um and then you throw things into the queue and then you can pull them out the queue and then you can pull them out the queue and then you can pull them out and once they're in they're uh stored and once they're in they're uh stored and once they're in they're uh stored durably, right? Right. And so you don't durably, right? Right. And so you don't durably, right? Right. And so you don't have to worry about anything going down have to worry about anything going down have to worry about anything going down or getting lost. Um and and it's pretty or getting lost. Um and and it's pretty or getting lost. Um and and it's pretty simple, right? You just push things on simple, right? You just push things on simple, right? You just push things on and pull them off and and it's there for and pull them off and and it's there for and pull them off and and it's there for you. So it's a really nice solution for you. So it's a really nice solution for you. So it's a really nice solution for building out the scale out.

building out the scale out. building out the scale out. >> And and does this thing automatically >> And and does this thing automatically >> And and does this thing automatically scale? Like you're looking at your your scale? Like you're looking at your your scale? Like you're looking at your your database, you're looking at your message database, you're looking at your message database, you're looking at your message sender, you're looking at the receiver sender, you're looking at the receiver sender, you're looking at the receiver of the messages. These things are kind of the messages. These things are kind of the messages. These things are kind of ebbing and flowing as as traffic of ebbing and flowing as as traffic of ebbing and flowing as as traffic comes and goes, comes and goes, comes and goes, >> right? So it it's not elastic um in the >> right? So it it's not elastic um in the >> right? So it it's not elastic um in the sense that like right they they don't sense that like right they they don't sense that like right they they don't automatically spin up cues as the automatically spin up cues as the automatically spin up cues as the traffic uh like waines and fluxes right traffic uh like waines and fluxes right traffic uh like waines and fluxes right uh we just we have a set number that we uh we just we have a set number that we uh we just we have a set number that we use and it and it's it's also the same use and it and it's it's also the same use and it and it's it's also the same way for all of our um way for all of our um way for all of our um >> for all of our uh compute and storage >> for all of our uh compute and storage >> for all of our uh compute and storage instances those are we basically planned instances those are we basically planned instances those are we basically planned for amount of load and then as we go for amount of load and then as we go for amount of load and then as we go through uh when we do deployments we'll through uh when we do deployments we'll through uh when we do deployments we'll actually take down the number of uh actually take down the number of uh actually take down the number of uh cores we're using cores we're using cores we're using >> and make that decision. Um, we don't >> and make that decision. Um, we don't >> and make that decision. Um, we don't have anything built in right now to do have anything built in right now to do have anything built in right now to do that elastically and just determine that that elastically and just determine that that elastically and just determine that for us. for us. for us. >> And do you use Q's service bus Q's and >> And do you use Q's service bus Q's and >> And do you use Q's service bus Q's and uh service bus as a relay? uh service bus as a relay? uh service bus as a relay? >> What do you mean by >> What do you mean by >> What do you mean by >> there's service bus relay service? >> there's service bus relay service? >> there's service bus relay service? >> Oh, okay.

>> Oh, okay. >> Oh, okay. >> And the service bus Q. >> And the service bus Q. >> And the service bus Q. >> I believe we just use service bus Q's >> I believe we just use service bus Q's >> I believe we just use service bus Q's right now. right now. right now. >> Uhhuh. Okay, cool. And what do you do >> Uhhuh. Okay, cool. And what do you do >> Uhhuh. Okay, cool. And what do you do for for logging? Like if it you have to for for logging? Like if it you have to for for logging? Like if it you have to go and and uh debug all of this. What's go and and uh debug all of this. What's go and and uh debug all of this. What's what's your log? I hope you don't have what's your log? I hope you don't have what's your log? I hope you don't have like text files. No, no, no. So, we have like text files. No, no, no. So, we have like text files. No, no, no. So, we have a an an ETW event tracing for Windows uh a an an ETW event tracing for Windows uh a an an ETW event tracing for Windows uh logging system in place. logging system in place. logging system in place. >> ETW? >> ETW? >> ETW? >> Yep. >> Yep. >> Yep. >> What is that? >> What is that? >> What is that? >> So, we're um So, it's it's basically the >> So, we're um So, it's it's basically the >> So, we're um So, it's it's basically the it's a new product of .NET 45. Um it's it's a new product of .NET 45. Um it's it's a new product of .NET 45. Um it's sort of a model for sending events and sort of a model for sending events and sort of a model for sending events and and and tracking stuff like that. And it and and tracking stuff like that. And it and and tracking stuff like that. And it because right now when you go to Azure, because right now when you go to Azure, because right now when you go to Azure, you can write to um the WAD logs, which you can write to um the WAD logs, which you can write to um the WAD logs, which is the diagnostic table. uh but the is the diagnostic table. uh but the is the diagnostic table. uh but the number of logs that we actually wanted number of logs that we actually wanted number of logs that we actually wanted to write didn't scale right so ETW was to write didn't scale right so ETW was to write didn't scale right so ETW was this framework that we basically built this framework that we basically built this framework that we basically built on top of that allows us to um you know on top of that allows us to um you know on top of that allows us to um you know sort of ship the logs out and not have sort of ship the logs out and not have sort of ship the logs out and not have to store as much you don't store all the to store as much you don't store all the to store as much you don't store all the strings right you um sort of store the strings right you um sort of store the strings right you um sort of store the format once and then apply when you're format once and then apply when you're format once and then apply when you're looking at the logs it applies it for looking at the logs it applies it for looking at the logs it applies it for you you just store all the parameters you you just store all the parameters you you just store all the parameters that you're or all the the variables that you're or all the the variables that you're or all the the variables that you're popping in and out that you're popping in and out that you're popping in and out >> um and so and then we also use um MDS >> um and so and then we also use um MDS >> um and so and then we also use um MDS which is uh an internal which is uh an internal which is uh an internal Microsoft tool right now to uh view the Microsoft tool right now to uh view the Microsoft tool right now to uh view the logs and whatnot.

logs and whatnot. logs and whatnot. >> So, ETW for folks who are wondering >> So, ETW for folks who are wondering >> So, ETW for folks who are wondering means event tracing for Windows and it's means event tracing for Windows and it's means event tracing for Windows and it's a it's a kernel mode logging feature a it's a kernel mode logging feature a it's a kernel mode logging feature like you like you said, but this is like you like you said, but this is like you like you said, but this is something that was uh added in um in something that was uh added in um in something that was uh added in um in Windows Server 2008 Windows Server 2008 Windows Server 2008 >> and it's it's basically integrated into >> and it's it's basically integrated into >> and it's it's basically integrated into the kernel such that you could turn it the kernel such that you could turn it the kernel such that you could turn it on and off as you're going and it's very on and off as you're going and it's very on and off as you're going and it's very very low overhead. very low overhead. very low overhead. >> Yeah. >> Yeah. >> Yeah. >> Yeah. It's pretty nice because if you >> Yeah. It's pretty nice because if you >> Yeah. It's pretty nice because if you have a user mode application, but you have a user mode application, but you have a user mode application, but you want to get some really low-level uh want to get some really low-level uh want to get some really low-level uh debugging, including things like uh even debugging, including things like uh even debugging, including things like uh even device drivers, you can go all the way device drivers, you can go all the way device drivers, you can go all the way down to the bottom if you want to. It's down to the bottom if you want to. It's down to the bottom if you want to. It's it started actually in Windows Server it started actually in Windows Server it started actually in Windows Server 20. It's Windows, excuse me, Windows 20. It's Windows, excuse me, Windows 20. It's Windows, excuse me, Windows 2000, but it got really really 2000, but it got really really 2000, but it got really really interesting in 2008 Server uh when III interesting in 2008 Server uh when III interesting in 2008 Server uh when III and and different websites started using and and different websites started using and and different websites started using it. it. it. >> Interesting. Okay. And so then you view >> Interesting. Okay. And so then you view >> Interesting. Okay. And so then you view that in a large from a larger that in a large from a larger that in a large from a larger application, a larger viewer, a log application, a larger viewer, a log application, a larger viewer, a log viewer, viewer, viewer, >> right? that sort of aggregates all the >> right? that sort of aggregates all the >> right? that sort of aggregates all the logs from all our different machines and logs from all our different machines and logs from all our different machines and and buy our services so we can run and buy our services so we can run and buy our services so we can run queries on top of them and look at what queries on top of them and look at what queries on top of them and look at what we think is interesting. we think is interesting. we think is interesting. >> So I I heard that you said that you used >> So I I heard that you said that you used >> So I I heard that you said that you used a management uh a the management portal a management uh a the management portal a management uh a the management portal as well as a third party management as well as a third party management as well as a third party management portal, but do you ever do any portal, but do you ever do any portal, but do you ever do any scripting? Like I kind of imagined a scripting? Like I kind of imagined a scripting? Like I kind of imagined a large system like this would include large system like this would include large system like this would include PowerShell scripts to fire up and turn PowerShell scripts to fire up and turn PowerShell scripts to fire up and turn down things.

down things. down things. >> So we have some tools right now. That's >> So we have some tools right now. That's >> So we have some tools right now. That's one of the things that we're working on one of the things that we're working on one of the things that we're working on currently is building out stuff. So this currently is building out stuff. So this currently is building out stuff. So this a lot of the things that we do manually a lot of the things that we do manually a lot of the things that we do manually through the portal and whatnot are um through the portal and whatnot are um through the portal and whatnot are um are done via just you know oneclick are done via just you know oneclick are done via just you know oneclick deployments type is the scenario that we deployments type is the scenario that we deployments type is the scenario that we want to get to. Um we're just not there want to get to. Um we're just not there want to get to. Um we're just not there yet because we just didn't have time. yet because we just didn't have time. yet because we just didn't have time. >> Mhm. And when you uh like when I log >> Mhm. And when you uh like when I log >> Mhm. And when you uh like when I log into Azure I have like a live ID and a into Azure I have like a live ID and a into Azure I have like a live ID and a credit card. Is there like some this is credit card. Is there like some this is credit card. Is there like some this is just a totally random question because I just a totally random question because I just a totally random question because I got to ask a random one at the end. Is got to ask a random one at the end. Is got to ask a random one at the end. Is there like some dude's credit card being there like some dude's credit card being there like some dude's credit card being charged? Uh I am actually not [laughter] charged? Uh I am actually not [laughter] charged? Uh I am actually not [laughter] sure how our billing is set up. I know sure how our billing is set up. I know sure how our billing is set up. I know um one of the funny things is we all um one of the funny things is we all um one of the funny things is we all used to be on like one developer account used to be on like one developer account used to be on like one developer account in Azure and because we [laughter] we in Azure and because we [laughter] we in Azure and because we [laughter] we were new to Azure, everyone like wasn't were new to Azure, everyone like wasn't were new to Azure, everyone like wasn't shutting down their cores and their shutting down their cores and their shutting down their cores and their deployments every night because you just deployments every night because you just deployments every night because you just don't think about that coming from a don't think about that coming from a don't think about that coming from a traditional service background and so traditional service background and so traditional service background and so they gave every developer we now have they gave every developer we now have they gave every developer we now have our own accounts so that they can see our own accounts so that they can see our own accounts so that they can see who's like, you know, costing them a lot who's like, you know, costing them a lot who's like, you know, costing them a lot of money. of money. of money. >> Oh, that's awesome. Okay, cuz that's >> Oh, that's awesome. Okay, cuz that's >> Oh, that's awesome. Okay, cuz that's that is the fear. I mean, that's funny that is the fear. I mean, that's funny that is the fear. I mean, that's funny when once you have a whole lot of stuff when once you have a whole lot of stuff when once you have a whole lot of stuff running on Azure, you do have to think running on Azure, you do have to think running on Azure, you do have to think about like, you know, does this need to about like, you know, does this need to about like, you know, does this need to be running and does this need and it's be running and does this need and it's be running and does this need and it's kind of cool. It's almost like running kind of cool. It's almost like running kind of cool. It's almost like running your website in Azure is almost like the your website in Azure is almost like the your website in Azure is almost like the way that you drive a Prius. If you've way that you drive a Prius. If you've way that you drive a Prius. If you've ever got a you have a Prius or anything ever got a you have a Prius or anything ever got a you have a Prius or anything like that, you ever driven a hybrid car?

like that, you ever driven a hybrid car? like that, you ever driven a hybrid car? >> Yeah. Yeah. Yeah. >> Yeah. Yeah. Yeah. >> Yeah. Yeah. Yeah. >> It's there's a gamification when you get >> It's there's a gamification when you get >> It's there's a gamification when you get a hybrid car and it's like, okay, I can a hybrid car and it's like, okay, I can a hybrid car and it's like, okay, I can do 44 miles per gallon, right? I'm going do 44 miles per gallon, right? I'm going do 44 miles per gallon, right? I'm going to save like 5 cents. And it changes the to save like 5 cents. And it changes the to save like 5 cents. And it changes the way you drive. and cloud services change way you drive. and cloud services change way you drive. and cloud services change the way you write code in that you want the way you write code in that you want the way you write code in that you want to squeeze that last penny, you know, to squeeze that last penny, you know, to squeeze that last penny, you know, >> right? So, it's just like an interesting >> right? So, it's just like an interesting >> right? So, it's just like an interesting note. Um, like obviously the big I would note. Um, like obviously the big I would note. Um, like obviously the big I would never say just take code out of a server never say just take code out of a server never say just take code out of a server and like port it into Azure. You're and like port it into Azure. You're and like port it into Azure. You're probably you could do it and it will probably you could do it and it will probably you could do it and it will probably work, but it's not going to be probably work, but it's not going to be probably work, but it's not going to be super cost effective or efficient. M super cost effective or efficient. M super cost effective or efficient. M >> um I think like really thinking about >> um I think like really thinking about >> um I think like really thinking about how you design your service for Azure so how you design your service for Azure so how you design your service for Azure so that a you're getting the most out of that a you're getting the most out of that a you're getting the most out of the pricing model and um you know you're the pricing model and um you know you're the pricing model and um you know you're building it for this sort of it's it's building it for this sort of it's it's building it for this sort of it's it's kind of a chaotic environment where you kind of a chaotic environment where you kind of a chaotic environment where you you you assume things fail uh is super you you assume things fail uh is super you you assume things fail uh is super helpful and and make your services more helpful and and make your services more helpful and and make your services more reliable and it'll make them scale out reliable and it'll make them scale out reliable and it'll make them scale out better in the as you move forward and better in the as you move forward and better in the as you move forward and get more users as well. get more users as well. get more users as well. >> Very cool. Well, thanks so much for >> Very cool. Well, thanks so much for >> Very cool. Well, thanks so much for chatting with me today, Katie. chatting with me today, Katie. chatting with me today, Katie. >> No problem. >> No problem. >> No problem. >> Cool. Next uh next couple weeks, we're >> Cool. Next uh next couple weeks, we're >> Cool. Next uh next couple weeks, we're actually going to have the Waypoint, the actually going to have the Waypoint, the actually going to have the Waypoint, the client side people who consume and use client side people who consume and use client side people who consume and use your services, and we're going to talk your services, and we're going to talk your services, and we're going to talk to those folks, too. So, we're going to to those folks, too. So, we're going to to those folks, too. So, we're going to see how Halo works on the back and the see how Halo works on the back and the see how Halo works on the back and the front end.

front end. front end. >> Cool. >> Cool. >> Cool. >> This has been another episode of Hansel >> This has been another episode of Hansel >> This has been another episode of Hansel Minutes, and we'll see you again next Minutes, and we'll see you again next Minutes, and we'll see you again next week. [music]