When I take the Xbox gamepad to start playing Halo: Master Chief Collection, I say something I’ve said approximately 17,000 times in my life: “Let me just invert the Y-axis.” I always think of the right thumbstick, which controls the game view, like a pilot's yoke, so when I want to look down, I push the thumbstick up.
Some people…don't agree. “Interview over,” Kareem Choudhry says, laughing.
Choudhry’s colleague, Kevin La Chapelle, jumps in to defend me. “I’ve found another one!” he says gleefully. “We’re unicorns.” Even inside the Studio D building on Microsoft’s campus, home of all things Xbox, the thumbstick-preference wars rage on.
Not that fixing the controller layout makes me any better at the game; trying to drive a Warthog, I crash the all-terrain vehicle into a tree. Repeatedly. It’s not my fault, though—I’m just not used to playing Halo on a phone.
Yes, on a phone.
Nearly a year ago, Microsoft executives acknowledged that the company was actively pursuing a cloud gaming service that would allow users to play Xbox games without an actual console. It wasn’t the only company racing toward the technology; everyone from chipmaker Nvidia to game publisher Electronic Arts to Sony’s PlayStation division has been working on the ability to stream games directly to customers. No downloads, no storage, no brawny processor requirements. Cloud gaming could untether games from the hardware we use to run them, and in doing so untether people from needing to buy ever-stronger, ever-pricer machines.
Now, on an ironically cloudless early October day, I'm visiting those same Microsoft execs at their headquarters in Redmond, Washington, to see the progress they’ve made. That progress includes a lot of caveats, and more than a little secrecy—but it also makes clear that Microsoft is aiming higher, and achieving more, than what some have thought.
Project xCloud, as the effort is known internally, is a company-wide push that leverages the expertise of a multitude of different teams. It tapped the secretive Microsoft Research division, which works on everything from quantum computing to AR/VR to genomics, to crack thresholds that probably wouldn’t have been cracked otherwise. It harnesses the power and ubiquity of Microsoft’s globe-spanning network of Azure data centers. And perhaps more importantly, it turns the living-room console—an immovable fixture of game culture since 1972—from an anchor into a hub.
According to market intelligence firm NewZoo, the games industry is set to bring in $138 billion in revenue this year, from 2.3 billion players—and for the first time, more than half of that money is coming from mobile games. Fortnite alone, which is available on mobile as well as consoles and PCs, has made publisher Epic Games more than $1 billion. Console gaming may represent the second-strongest sector in the industry, but it’s still missing out on a lot of those 2.3 billion players. We’ve gotten used to playing games anywhere we have a screen; Microsoft wants to make sure all those games are Xbox’s.
Finding Gamers Where They Are
Phil Spencer has been head of Xbox for years. If you play games, you know his face and voice well—he’s the guy who gets onstage at E3 or Gamescom, invariably in a t-shirt and sneakers, to pump the newest games and features coming to the Xbox One or its predecessor, the Xbox 360. Last year, though, Spencer’s role changed; Microsoft CEO Satya Nadella added him to the company’s Senior Leadership Team, a group of 15 executives that meets weekly with Nadella. That came with a new title, “executive vice president of gaming,” but it also signaled a shift inside Microsoft, and inside Nadella’s head: gaming mattered. A lot.
>
By this summer, that signal became a shout. Over a span from April through June, Xbox revenue was up 35 percent over the previous year. And on a conference call with investors, Nadella explicitly said what Xbox owners hadn’t heard in a long time: the company was all-in on gaming. “We're pursuing our expansive opportunity from the way games are created and distributed, to how they're played and viewed,” Nadella said. “We're investing aggressively in content, community, and cloud services across every end-point to expand usage and deepen engagement with gamers.”
What he meant wasn’t any single decision, but a larger shift in the company’s approach. And for Spencer, who has long considered himself an advocate of the player above all else, the shift is simple: “Let’s put our user at the center of everything.”
Sitting today in a meeting room in Studio D, Spencer—who's wearing long sleeves, though they’re pushed up his forearms—explains that this shift has been part of Xbox’s thinking for some time now. He cites the backward-compatibility project that allowed Xbox One owners to play the games they’d bought on older Xbox consoles; the ability to play first-party Xbox games on your Windows PC and console interchangeably; the move to Bluetooth controllers, so that gamers could use their controllers easily on PCs and other devices; even Game Pass, a subscription service Xbox launched last year that gave customers access to an ever-growing library of games they could download and play on demand.
>
Yet, while all of those measures were steps toward player freedom, they all kept Xbox squarely in the home, and squarely on Microsoft and Windows devices. There was one giant leap that had yet to be taken, and that’s exactly the leap Spencer wanted to take. He spun up a new cloud gaming division, and asked Kareem Choudhry—the aforementioned Y-axis purist—to run it. A 20-year Microsoft veteran, Choudhry had gone from Outlook to Windows DirectX to a 13-year stint working on Xbox. And now, Choudhry would try to pull off what had long been near-impossible: doing what Spotify had done for music, and what Netflix and Hulu had done for video. Making games available anywhere, on any device. No download necessary.
Partly Cloudy With a Chance of Latency
“Where do I watch Netflix?” Spencer asks me—before he answers his own question. “Yes, right? If I'm at the dentist office waiting. I'm on my phone. If I happen to be sitting in an airport I might be on my laptop. When I get home I'm probably on my television because it's the biggest screen that I have—and I don't even really think about those choices.”
That’s exactly what Project xCloud wants to do with games. There’s just one problem: from a technological standpoint, streaming games isn’t exactly easy. A video or music file is a static piece of content; as long as you’ve got the distribution handled, you can stream those to anyone with a 4G connection. Games, though, are interactive—and thus exponentially more complicated.
“Each frame,” says Choudhry, “you have to ask yourself: What did the player do? What did all of the multiplayer actors do and how did they impact the world? What are the AI agents in the world doing? What's happening from a physics, from an audio, from a rendering perspective? And then put all that together, package it up, and send it back down to the user on the other side of the streaming pipe.”
If the game runs at 30 frames per second, you've got 33.3 milliseconds to make all those things happen; for a 60fps game, you've got half that. Try it on a 4G connection with a graphic-intensive game like Red Dead Redemption or Overwatch, and you’d be looking at … well, not either of those games, that’s for sure. So while game-streaming does exist in some forms—Sony’s PlayStation Now service lets you stream games to your PS4 console without downloading the whole thing—no one’s figured out how to do it reliably with AAA games on non-AAA devices.
But not everyone has a company that’s already invested billions of dollars in the server power you’d need to pull it off.
“Our expectation would be if you fast-forward five years, there will be multiple videogame streaming aggregators,” says Mike Olson, a managing director and senior research analyst at Piper Jaffray. “The ones that are best positioned are probably those that already have a good brand in gaming, and a robust cloud platform. That’s a pretty short list.” According to Olson, it’s a list of three: Amazon, which owns Twitch and server behemoth AWS; Google, which has Google Play and Google Cloud Platform (and just last week announced their own cloud-gaming initiative, Project Stream); and Microsoft, with Xbox and its Azure cloud infrastructure.
Azure has been lurking in the Xbox One ecosystem since the console was released in 2013, handling various background processes; Microsoft’s first-party games like Forza and Minecraft rely on it, as do games like Hitman and Rainbow Six Siege. By now, there are Azure data centers in 54 global regions—and each one of those data centers will become key to being able to stream a game experience to a connected device.
See, the chief bugaboo in cloud gaming is latency: the delay between a game input like pressing a button or moving a thumbstick, and seeing it reflected on the screen. And latency gets worse over distance. “The speed of light is the speed of light,” Choudhry says. “I'm not aware of anyone at Microsoft has found a way to change those physics just yet.” So the closer a gamer is to a cloud server, the faster the signal can travel back and forth. There’s more than just latency at play here, though; there’s visual quality, how the signal is encoded and decoded, the crispness and color of the image. And all of it has to be processed in an Azure data center.
So the Cloud Gaming division’s first real challenge was figuring out what would go into the Azure centers. This wasn’t just a matter of handling cloud saves or helping out with multiplayer functions; whatever went into the server racks needed to be able to handle 100% of gameplay offsite, then send it back to the player with minimal latency. (Choudhry says that there’s a target number the team is using—latency is generally measured in milliseconds— but he won’t share what it is.)
Choudhry and his gaming cloud colleagues bring out an early prototype of one of those server units to show me how it works. It’s a “1u blade,” containing the internal componentry of four separate Xbox One consoles—two facing up, two facing down, to maximize airflow—along with a row of cooling units, a power management board, and networking jacks. Whenever a gamer connects to the server, their account gets linked to one of the four Xbox-ish units in one of the blades in one of the racks in one of the Azure data centers that’s as close as possible to the gamer’s geographic location.
Now comes the tough part. First, the responsible Xbox-ish unit encodes a frame of the game that the user is playing, compressing it down and sending it over the network to the user’s device. Next, it sends any input from the user back from the device through the network—rationalizing it against any multiplayer activity that may be coming from other users’ devices—then renders the next frame, calculating the physics and lighting and audio and any other dynamic system that might change as a result of users’ actions. (How an object bounces, or what happens if the sun becomes visible through a window.) But: how do you do that as quickly as possible, with the highest quality possible, using as little internet bandwidth as possible?
That’s where Microsoft Research comes in. The research group was instrumental in helping make the Xbox One backward-compatible; now, it’s trying to make Project xCloud efficient enough to run on a 10 megabits-per-second internet connection, which means you’d be able to play using a 4G LTE connection in virtually every major market in the US. (A high-def Netflix video requires about 5 Mbps; Google's recently announced cloud-gaming project recommends 25.) That encompasses a host of challenges, but many of them boil down to trying to improve existing compression methods so that encoding and decoding can be accomplished using fewer bits. Fewer bits = less information to move = lower latency. One example: since most video games have a heads-up display, researchers are using machine learning to train the encoder on those less variable areas of the screen.
Of course, given that Microsoft wants to start testing the service publicly sometime in 2019, you’d figure they have something running now. They do, though no one outside the company has tried it. Until now.
An Xbox In Your Pocket
Kevin La Chapelle is another Microsoft veteran. In 1999, he was the founding engineer for Windows Movie Maker, and went on to create video streaming encoders, ultimately working on the team that made it possible for NBC to stream the 2008 Beijing Games around the clock. A decade ago, people streaming live video to their phones was a triumph (even if it required Silverlight); in 2018, the stakes are higher. So when La Chapelle, the general manager of Xbox game streaming, hands me the Xbox controller, he seems almost relieved that an outsider is going to experience what he's been living.
Attached to the controller, via a cheapo Amazon clip, is a Samsung Galaxy phone. (Microsoft started putting Bluetooth in its Xbox controllers in 2016, so people could use the controllers with their Windows computers.) Halo: Master Chief is on this one, but there are four Xbox titles total, all running on phones or tablets. “They don't know they're running in the cloud,” La Chapelle says. “As they're concerned, they're on a console that could be in your house.” The consoles that they’re on, each nestled among three of its siblings, are in a data center in eastern Washington, about 200 miles away.
Could I tell? Honestly, no. On that phone, I drove the Warthog like I’d just staggered out of a bar; on another, I played a little Gears of War 4. None were high-speed multiplayer environments, but the controls were responsive and the graphics were smooth, and I never felt appreciable latency. Aside from a single instance of blocky graphics on Gears, it was like playing at home, albeit on a much smaller screen—and with phone-speaker sound. (You’re gonna want headphones. And maybe reading glasses.) The games were … the games. “We didn't crack the bits,” Choudhry says. “They didn't repackage it. They didn't reopen it. They didn't put cloud-specific restrictions on. It is the same experience.”
But not everyone wants to bring their controller anywhere. To demonstrate how folks might play Xbox games on the go and accessory-free, Bill Stillwell, the gaming cloud team’s director of product planning and strategy, hands me a tablet. It’s playing—or, rather, it’s channeling—Forza Horizon 4, a game that came out that very day. In the lower right-hand corner of the screen, I see touch versions of the buttons that festoon every Xbox gamepad: green A, red B, blue X, orange moon, yellow Y. In the lower left corner, a four-directional steering interface.
But then Stillwell changes a setting, and gives it back. “This isn’t a game that would be super easy with this interface, given all the button presses you need to do,” he says. “So we’re building different types of templates.” Now the tablet I just switched over to the driving template. The Xbox controller buttons have turned into icons for gas, brake, and rewind (because it’s Forza). Even better, the steering mechanism is now a simple horizontal slider, and there’s even a tiny little haptic bump sensation when you reach the midpoint. This is Forza Horizon 4, designed to be played with two thumbs.
Choudhry refuses to put a date on when Project xCloud, whatever it ends up being called, will become an official cloud gaming service. He’s adamant about three things, though. One is the ultimate goal of being able to “lift and shift” every single title playable on the Xbox One—more than 3,000 games, from the original Xbox all the way through present day—to the cloud. That’s something Microsoft is handling, no developer work necessary. (Just as with backward compatibility, game studios only needed to agree to add their game to the library; the technical process becomes a Microsoft problem.)
Second is making sure that cloud gaming isn’t a halfway deal. Choudhry doesn’t want Project xCloud to feel like a separate silo through which you access an Xbox game; he wants it to be indistinguishable from any other game you’re playing. So: Save files that synchronize across devices. Or a friend network and multiplayer matchmaking that remains consistent from cloud to conventional gaming. “We're really proud of what we've been able to do in our console ecosystem,” he says, “and for us to make a cloud offering that fell short of everything that we have there just wasn't going to be good enough for us.”
But most of all, Choudhry, like Phil Spencer, wants to make clear that cloud gaming isn’t coming to replace your game discs and digital downloads. “I’m not a fan of the tyranny of ‘or,’” he says. “We’re going to continue playing in the console space as you know it today.” He won’t cop to what form the final product might come in; some reporting points to a console with streaming capabilities included, as well as a digital-only console that can download or stream games. But with a solid cloud-gaming platform, Microsoft stands to offer something new to existing customers—and bring in a generation of younger players whose primary gaming device fits in their pocket.
That’s not a conversation for now, though. For now, Choudhry’s just proud of what they’ve managed to build so far. And how crucial Project xCloud is to today’s always-on digital world. “My children don’t understand what it means to listen to the radio, waiting for your favorite song to come on,” he says.
So why should they wait until they get home to play Forza?