Good Luck Escaping AI: At Build 2024, Microsoft Leans Hard Into Copilot

The biggest news at Microsoft’s Build 2024 developer conference came before today's keynote even started with a press-only reveal of new Surface computers and Copilot AI features. But Tuesday's code-focused event, headlined by CEO Satya Nadella, had plenty of product news, nearly all of which centers on Copilot and its Azure cloud development platform.

Microsoft has "added 30 times the supercomputing power to Azure," according to Nadella, who says Microsoft is "on track to meet our goal to have our data centers powered by 100% renewable energy by next year" despite all the heavy AI processing on its servers.

Nadella stressed Redmond's close partnership with Nvidia, which supplies most of the world's AI chips. But he didn't leave out AMD, noting that Microsoft would support its ND MI300X V5, which is optimized for Azure workloads. Not to be outdone, Microsoft announced a public preview of its Azure Cobalt Arm-based data-center processor.

Team Copilot

New on the Copilot front for Microsoft 365 is Team Copilot, which can act as a meeting facilitator, taking notes, running and summarizing meetings, and even managing projects. It will be available in preview later this year for customers with Copilot for Microsoft 365 licenses.

Edge Gets Video Translation and Screenshot Prevention

Windows’ built-in web browser, Microsoft Edge, will be able to translate web video across sites like YouTube, LinkedIn, Reuters, CNBC News, Bloomberg, Coursera, and more. Currently, “real-time video translation will be available from Spanish to English and from English to German, Hindi, Italian, Russian and Spanish," with languages to follow.

The Business edition of Edge is getting new security features, such as screenshot prevention and automatic browser updating on managed PCs.

Windows May Finally Get Arm Right

Another major theme of the conference is Windows running on Arm processors, and on the new Snapdragon X Elite chips in particular. Microsoft’s aim is to finally beat—or at least equal—Apple’s performance success with its Arm-based silicon on macOS. And indeed, the company claims to have done this with its new Surfaces.

Microsoft also announced Prism, an emulator (similar to Apple’s Rosetta 2) that allows any Windows application to run with good performance on an Arm-based PC.

The AI PC Is Coming Soon

Hands On: Microsoft's 2024 Surfaces Level Up With Copilot AI, Arm Silicon

Just as important for Microsoft at this Build is to create a true AI PC that uses on-board hardware for processing at least some of the data, rather than sending prompts to Microsoft’s servers. The new Surfaces and Copilot+ PCs from partners include neural processing units (NPUs) to handle this onboard AI processing.

At the pre-Build press conference, Microsoft announced PCs that do just this and new AI-powered experiences that leverage the Copilot+ PCs' NPUs. One of the most interesting of these is Recall, which lets you snap back to anything you were doing on your PC, with full context.

Also announced were Cocreator, an art generation tool; Live Caption, a speech-to-text feature that includes translation from 40+ languages; and video call enhancements called Windows Studio Effects that include Portrait light and other effects.

On the heels of Microsoft’s Surface news, Qualcomm announced its Snapdragon Dev Kit for Windows, a test computer for developers powered by the Snapdragon X Elite SoC.

Microsoft's New AI Chief Speaks

Prior to the conference, the press was privy to a recorded interview with Microsoft’s newly minted CEO of AI, Mustafa Suleyman. Suleyman cofounded DeepMind, an AI startup acquired by Google in 2014. He then founded Inflection AI, a generative AI and machine-learning startup that released Pi.ai, a ChatGPT-like product intended to have more empathy and kindness for the user.

In a conversation moderated by Microsoft’s PR chief, Frank X. Shaw, Suleyman noted that there are 135 Copilot surfaces across the Microsoft portfolio, an impressive number given that Copilot's only been around for just over a year. He claimed Microsoft AI is “focused on the safety and ethics of AI," and a goal for Microsoft’s AI involves “experiences that are emotionally intelligent, not just factual, feeling, tone, energy.” This can provide “fluency and smoothness of the interactions.”

Suleyman didn’t mention Google’s AI blunders by name, but he said AI “models are too scared to touch sensitive issues. If you shut down contrarian views, it’s one of the worst things you can do.” Stepping away from "a hot topic" is the wrong approach. (Google's AI image creator currently won’t let you create any images that include any humans.)

The Microsoft exec argued that "AI lowered the floor and raised the ceiling for everybody," which stands in contrast to popular fears of AI taking over jobs and calling the shots, though you might expect that answer from someone running Microsoft's AI operations. Ultimately, "we have to rethink the entire foundation of computer interaction," he said.

Copilot Developer Tools

New Copilot-related dev tools announced today include Windows Copilot Runtime and Windows Copilot Library, which can be used to create new AI features.

According to Pavan Davuluri, Corporate VP for Windows & Devices Developers, coders can tap over 40 on-device AI models that ship with Windows, adding AI features like Studio Effects, Live Caption Translations, OCR, Recall with User Activity, and Phi Silica, a small language model (SLM) that will be available to developers in June.

Davuluri also announced Windows Semantic Index, “a new OS capability which redefines search on Windows and powers new experiences like Recall.” Later, this capability will be available for developers with Vector Embeddings API to build their own vector store and RAG (Retrieval Augmented Generation) within their applications and with their app data.

Recommended by Our Editors

Why Are Intel Processors AWOL in Microsoft’s New Surfaces?

Hands On: Microsoft's 2024 Surfaces Level Up With Copilot AI, Arm Silicon

At Microsoft Build, Did We Just See the Spark of a PC-Laptop Revolution?

Other new capabilities include Phi Silica, which, according to Davuluri, “is built from the Phi series of models and is designed specifically for the NPUs in Copilot+ PCs.” He noted that “Windows is the first platform to have a state-of-the-art small language model custom built for the NPU and shipping inbox.”

Also new from Microsoft is native support for PyTorch, which allows thousands of Hugging Face models to work on Windows, as well as Web Neural Network, which will allow web developers to build AI into web applications.

Microsoft also talked about DirectML, an intermediary that accelerates code using GPU or NPU hardware. This works with an ONNX Runtime to speed up generative AI models including Phi, Llama, Mistral, and Stable Diffusion.

GitHub is getting Copilot updates, too. GitHub Copilot Extensions will let users “build and deploy to the cloud in their natural language with their preferred tools and services,” according to GitHub’s announcement. Extensions from DataStax, Docker, LambdaTest, LaunchDarkly, McKinsey & Company, Microsoft Azure and Teams, MongoDB, Octopus Deploy, Pangea, Pinecone, Product Science, ReadMe, Sentry, and Stripe are up first. These extensions work in GitHub Copilot Chat, Visual Studio, as well as VS Code.

AI 3D Experiences with Microsoft Mesh

In Microsoft’s tech portfolio, the metaverse lives on in the form of Microsoft Mesh, which is intended as a 3D environment for distributed business workforces. New for Mesh as of this Build conference is the ability for developers to add AI to immersive experiences using Microsoft Azure OpenAI Service to access large language models.

Khan Academy Gets Microsoft AI Tutor

Education is another focus of the conference. Microsoft is collaborating with Khan Academy and using its new Phi-3 small SLMs to improve AI tutoring and make it accessible for free to K-12 students without using their data for training. Khanmigo for Teachers is an AI-powered teaching assistant that will also use Phi-3 SLMs.

Microsoft Fabric

Fabric is Microsoft’s data analytics tool. This, too, is getting an AI infusion as well as Real-Time Intelligence. That means giving customers, according to Microsoft press materials, the ability to “act on high volume, time-sensitive and highly granular data in a proactive and timely fashion to make faster and more-informed business decisions.”

Visual Studio

A new AI Toolkit for Visual Studio Code was announced at the show, now in preview. It integrates AI development tools and models into VS in order to run various language models, use local and cloud compute to optimize and fine-tune models, and deploy models to Microsoft Azure AI Studio using container images.