• Light/Dark
  • Ever wanted an RSS feed of all your favorite gaming news sites? Go check out our new Gaming Headlines feed! Read more about it here.

Nintendo Filed Two Patents for AI Upscaling Methods

ILikeFeet

ILikeFeet

DF Deet Master
Banned
Oct 25, 2017
61,987
two patents for "SYSTEMS AND METHODS FOR MACHINE LEARNED IMAGE CONVERSION". essentially AI upscaling akin to Nvidia's DLSS and Intel's XeSS.

these patents were filed on March 25, 2020 and published today. the inventors are all listed from France, so this is probably the work of NERD. the patents mention the methods can be accelerated by matrix multiplication hardware, like Nvidia's Tensor Cores

also note that they were not granted the patents.

FAjrNlBX0AULp3W


patents are viewable here

SYSTEMS AND METHODS FOR MACHINE LEARNED IMAGE CONVERSION - NINTENDO CO., LTD.

<div p-id="p-0001">A computer system is provided for converting images through use of a trained neural network. A source image is divided into blocks and context data is added to each pixel block. The
www.freepatentsonline.com www.freepatentsonline.com

SYSTEMS AND METHODS FOR MACHINE LEARNED IMAGE CONVERSION - NINTENDO CO., LTD.

<div p-id="p-0001">A computer system is provided for converting images through use of a trained neural network. A source image is divided into blocks and context data is added to each pixel block. The
www.freepatentsonline.com www.freepatentsonline.com

thanks to Disorientator for finding this
 
Maple

Maple

Member
Oct 27, 2017
11,719
Interesting. Why would Nintendo be developing their own upscaling methods if the next Switch is presumably using DLSS?

Wouldn't these implementations be handled by Nvidia?
 
Skittzo

Skittzo

Member
Oct 25, 2017
41,037
*Not granted yet.

Also notable is that this was filed right around when DLSS 2.0 actually came out. And they were clearly working on it months prior, since it's hard to file a patent application for something you're not working on.
 
RadioJoNES

RadioJoNES

Prophet of Truth - One Winged Slayer
Avenger
Oct 25, 2017
1,877
Ok neat. I actually did scan the documents a little bit before posting but I'm glad that's the case! Website is acting weird but I am replying to the people who quoted me.
 
Instro

Instro

Member
Oct 25, 2017
15,001
Maple said:
Interesting. Why would Nintendo be developing their own upscaling methods if the next Switch is presumably using DLSS?

Wouldn't these implementations be handled by Nvidia?
Click to expand...
Click to shrink...
Given that it mentions tensor cores, this may be for some Nintendo/Switch specific version, built or tweaked for their hardware and/or their game engines.
 
jj70

jj70

Member
Oct 27, 2017
273
these are technologies made to increase resolution to bring 3DS games to switch with some sort of retro console, imho...
 
Skittzo

Skittzo

Member
Oct 25, 2017
41,037
The timing of these pgpubs is impeccable, considering the Bloomberg article. Nintendo does not control when these are published BTW, unless they file a non-publication request which they did not in these cases.

So it was all completely random that they were published today.
 
BloodHound

BloodHound

Member
Oct 27, 2017
8,997
Opting to use their own AI temporal upscaling technique instead of using what's available?

I understand why someone like insomniac would use proprietary upscaling techniques cuz there were no options available and now FSR is trash, but DLSS is best in the business.
 

Deleted member 93062

Account closed at user request
Banned
Mar 4, 2021
24,767
jj70 said:
these are technologies made to increase resolution to bring 3DS games to switch with some sort of retro console, imho...
Click to expand...
Click to shrink...
Not sure if you're being sarcastic, but these do say that the game engine needs to work with it. If you need to add ML upscaling to your engine... you might as well just raise the resolution.
 
Dyle

Dyle

One Winged Slayer
The Fallen
Oct 25, 2017
29,890
So if I'm reading it right this would break the image up into smaller parts, upscale each one individually, and then put them back together again, is that different from most AI upscaling? I could imagine such a method being faster but maybe not quite as clean, perhaps resulting in issues around the seams between pieces.
 
ArchedThunder

ArchedThunder

Uncle Beerus
Member
Oct 25, 2017
18,994
BloodHound said:
Opting to use their own AI temporal upscaling technique instead of using what's available?

I understand why someone like insomniac would use proprietary upscaling techniques cuz there were no options available and now FSR is trash, but DLSS is best in the business.
Click to expand...
Click to shrink...
It'd probably be a fork of DLSS tailor made to the Switch 2/Pro's mobile hardware, not something entirely new.
 
Aardvark Barber

Aardvark Barber

Member
Oct 26, 2017
1,206
Pittsburgh, PA
hey we are definitely not working on any new switch pro or anything like that okay don't even think about it we would not lie to you i mean mario works for us and he trusts us so you should too
 
808s & Villainy

808s & Villainy

Banned
Oct 27, 2017
42,700
Aardvark Barber said:
hey we are definitely not working on any new switch pro or anything like that okay don't even think about it we would not lie to you i mean mario works for us and he trusts us so you should too
Click to expand...
Click to shrink...
Well patents don't always necessarily manifest themselves in HW. It's just that these particular ones have too much smoke around them for them not to
 
Skittzo

Skittzo

Member
Oct 25, 2017
41,037
Maple said:
Right, but isn't that something Nvidia would develop and implement, not Nintendo?
Click to expand...
Click to shrink...

It is curious that they worked on their own solution, but yeah it appears NERD does have their own proprietary deep learning engine. I guess it's sufficiently different enough or specifically tooled towards upressing older games (like they did with Sunshine's cutscenes).
 
PlayBee

PlayBee

One Winged Slayer
Member
Nov 8, 2017
5,528
Crazy how they're working on stuff like this when they have absolutely no plans to introduce new hardware
 
Rösti

Rösti

Member
Oct 25, 2017
801
I was going through this myself. Yes, the inventors are from NERD (Nintendo European Research and Development). Alexandre Delattre is the managing director and Theo Charvet is an R&D engineer. As for the scope of this, this may lend some more understanding:

For example, a neural network for upconverting to 1080p from 540p will be different than one upconverting from 1080p to 1440p (e.g., 2560.times.1440). It will also be appreciated that while the example shown in FIGS. 3-7 relates to transforming a 540p image to a 1080p image, the techniques herein may be applied to other image sizes (e.g., 720p to 1080p; 480p to 1080p, 1080p to 1440p, 1080p to 4k/3840.times.2160, 720p to 4k, etc.).
Click to expand...
Click to shrink...
 
OP
OP
ILikeFeet

ILikeFeet

DF Deet Master
Banned
Oct 25, 2017
61,987
some other things to note

the games implements upscaling natively, so there's no system level scaling it seems

FAj5VGJWYAEP3bp

The game engine 110 includes program structure for generating images that are to be output to the display 116. For example, the game engine 110 may include program structure for managing and updating the position of an object(s) in a virtual space based on inputs provided from the input device 114. The provided data is the used to render an image of the virtual space by using, for example, a virtual camera. This image may be a source image that is generated in a first resolution (e.g., 540p). The source image is applied to the neural network 112 that converts the source image into an upconverted image (e.g., an upconverted image is generated based on application of the source image to the neural network 112) that is at a higher resolution (e.g., 1080p) than the original source image. That upconverted image is then output to the display device 116 for display thereon. Further description of how the neural network is generated is provided in connection with FIG. 9.
Click to expand...
Click to shrink...


Nintendo can continue training and then distribute updates to systems. this is similar to how Nvidia can distribute DLLs for updated dlss

FAj5k_wXsAIeqbD


there can be multiple neural networks that are tailored towards game type and even processing speed
As discussed herein different types of neural networks may be generated and distributed to the various game devices. Thus, for example, game device 1 may receive and use a neural network that is different than a neural network that is received and used by game devices 2, 3, 4, and 5. In certain example embodiments, each game (or more generally each application) may have a corresponding neural network (or multiple neural networks) that has been generated (e.g., by system 900) for that game. Accordingly, for example, game devices may store multiple different neural networks and use different such networks based on the game (or type of game) that is being played on the corresponding game device. In certain example embodiments, multiple games may share or use the same neural network. For example, one neural network may be generated for first person shooter type games, another for strategy games, etc. Accordingly, game may be group based on their "type." Such type classifications may be based on the genre of game or may be based on another criteria, such as the type of rendering engine that the game uses to generate images therein.

In certain example embodiments, the game engine (or other service that is providing converting functionality to the game engine) may dynamically decide to select one neural network among a selection of them depending on the remaining time available to "prepare the current video frame". If the frame was rendered fast, it may have more time to be upscaled with a high quality and slow neural network (e.g., one that includes additional layers), but if the frame used up more of the typically available 16 ms (for both rendering of the frame and subsequently upscaling images at 60 frames per second), the engine could select a faster neural network (e.g., one with less layers), although not providing as high image quality as the slower ones. Such a determination may be made through a "testing" phase of a video game application program (e.g., where the game engine produces a number of example images) and/or may be performed during normal gameplay.
Click to expand...
Click to shrink...
 
808s & Villainy

808s & Villainy

Banned
Oct 27, 2017
42,700
Skittzo said:
It is curious that they worked on their own solution, but yeah it appears NERD does have their own proprietary deep learning engine. I guess it's sufficiently different enough or specifically tooled towards upressing older games (like they did with Sunshine's cutscenes).
Click to expand...
Click to shrink...
NERD was such a good purchase for Nintendo. They're also the reason we don't have the janky emulator + ROM bundles that use to be how Nintendo handled their retro titles pre-Switch and why none of them are portable
 
Josh5890

Josh5890

I'm Your Favorite Poster's Favorite Poster
The Fallen
Oct 25, 2017
23,136
TheGreatMightyPoo said:
It's 5 times worse at the Nintendolife comments, lol.

They act like Nate ran over their dogs with a Monster Truck and that Bloomberg made a smoothie out of the remains.
Click to expand...
Click to shrink...

Oh they can't be that bad....

If I were a betting man, what this is.. is Bloomberg acting a bit spiteful over this Switch Pro that never transpired, more so since they've been the ones championing those rumors since day-one. Coincidentally (NOT!), they publish this somewhat speculative article a week before Switch OLED drops? No, doesn't sound petty AT ALL.
Click to expand...
Click to shrink...

Wow
 
Skittzo

Skittzo

Member
Oct 25, 2017
41,037
DLSS as it is may have been too "general" for Nintendo to use in a future ML upscaling device. Nintendo's games typically are much more stylized and colorful, so they may have been able to create a more optimal deep learning engine to do basically the same thing DLSS does but for their games specifically.

Possibly also tooled to better upres legacy content.
 
Instro

Instro

Member
Oct 25, 2017
15,001
Maple said:
Right, but isn't that something Nvidia would develop and implement, not Nintendo?
Click to expand...
Click to shrink...
Could be something they are developing with Nvidia, but Nintendo gets the patent. Could also be that this is specifically for their own games, whereas standard DLSS is for 3rd parties. Probably a lot of variations here, but the patent seems to basically be DLSS, unless I'm missing something.

Edit
They may also need this separate patented version if the relationship with Nvidia ever dissolves.
 
You must log in or register to reply here.