The custom hardware in the Xbox Series X to support application of artificial intelligence (AI) through machine learning hasn't gotten enough attention in my opinion, particularly because Sony has not cited anything similar yet regarding AI support in the hardware. We can all agree that more pixels doesn't change how games play or are designed, but this type of technology can really impact various aspects of visual quality and things like smarter NPCs. One real world example of integration into the rendering pipeline would be smart upscale like the Forza Horizon 3 super resolution demo at GDC last year so that fewer pixels are being rendered by the GPU, which of course means that more of the 12 TF in the Series X can be used for some awesome graphical effects or ray tracing for example.
Here is the technical glossary reference for their Series X hardware support of the DirectML (Direct Machine Learning) API - https://news.xbox.com/en-us/2020/03/16/xbox-series-x-glossary/
The really exciting information comes for the Eurogamer full Xbox Series X specs reveal (https://www.eurogamer.net/articles/digitalfoundry-2020-inside-xbox-series-x-full-specs) where the Microsoft engineers talked further about the custom hardware support that they developed working with AMD to support machine learning applications. The GDC talk referenced later mentioned how the Nvidia Turing RTX cards of course have the tensor cores that provide FP16 support to improve machine learning performance, and Microsoft went further than that in the custom hardware.
This is a Microsoft talk at GDC last year first introducing DirectML which will now be supported directly by the hardware in the Series X. The entire talk is interesting if you are into API details, but the Forza Horizon 3 super resolution demo to show an upscale from 540p to 1080p at the 23:44 mark is particularly interesting and the Unity demo at 37:26 just shows how they used machine learning to provide animation in their demo based on the physics at play.
Here is the technical glossary reference for their Series X hardware support of the DirectML (Direct Machine Learning) API - https://news.xbox.com/en-us/2020/03/16/xbox-series-x-glossary/
DirectML
– Xbox Series X supports Machine Learning for games with DirectML, a component of DirectX. DirectML leverages unprecedented hardware performance in a console, benefiting from over 24 TFLOPS of 16-bit float performance and over 97 TOPS (trillion operations per second) of 4-bit integer performance on Xbox Series X. Machine Learning can improve a wide range of areas, such as making NPCs much smarter, providing vastly more lifelike animation, and greatly improving visual quality.
The really exciting information comes for the Eurogamer full Xbox Series X specs reveal (https://www.eurogamer.net/articles/digitalfoundry-2020-inside-xbox-series-x-full-specs) where the Microsoft engineers talked further about the custom hardware support that they developed working with AMD to support machine learning applications. The GDC talk referenced later mentioned how the Nvidia Turing RTX cards of course have the tensor cores that provide FP16 support to improve machine learning performance, and Microsoft went further than that in the custom hardware.
Machine learning is a feature we've discussed in the past, most notably with Nvidia's Turing architecture and the firm's DLSS AI upscaling. The RDNA 2 architecture used in Series X does not have tensor core equivalents, but Microsoft and AMD have come up with a novel, efficient solution based on the standard shader cores. With over 12 teraflops of FP32 compute, RDNA 2 also allows for double that with FP16 (yes, rapid-packed math is back). However, machine learning workloads often use much lower precision than that, so the RDNA 2 shaders were adapted still further.
"We knew that many inference algorithms need only 8-bit and 4-bit integer positions for weights and the math operations involving those weights comprise the bulk of the performance overhead for those algorithms," says Andrew Goossen. "So we added special hardware support for this specific scenario. The result is that Series X offers 49 TOPS for 8-bit integer operations and 97 TOPS for 4-bit integer operations. Note that the weights are integers, so those are TOPS and not TFLOPs. The net result is that Series X offers unparalleled intelligence for machine learning."
This is a Microsoft talk at GDC last year first introducing DirectML which will now be supported directly by the hardware in the Series X. The entire talk is interesting if you are into API details, but the Forza Horizon 3 super resolution demo to show an upscale from 540p to 1080p at the 23:44 mark is particularly interesting and the Unity demo at 37:26 just shows how they used machine learning to provide animation in their demo based on the physics at play.