• iOS/Safari issues when launching the site from mobile home screens should now be fixed! A2HS button compatibility on Android Firefox has also been fixed! Add some purple to your home screen today.

AI Neural Networks being used to generate HQ textures for older games (You can do it yourself!)

Is this going to change game dev?

  • Yeah

    Votes: 876 70.0%
  • Naw

    Votes: 376 30.0%

  • Total voters
    1,252
Dec 28, 2017
3,892
#1
Credit to Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, Wenzhe Shi/Arxiv on their paper about Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. I suggest you give it a read if you're interested in this kind of thing.

https://arxiv.org/pdf/1609.04802.pdf

Credit to Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, Xiaoou Tang for their paper on ESRGAN and for providing the ESRGAN repo with pretrained models

https://arxiv.org/pdf/1809.00219.pdf
https://github.com/xinntao/ESRGAN


Guide if ya wanna try it yourself:
Credit to kingdomakrillic their amazing work!


https://kingdomakrillic.tumblr.com/post/178254875891/i-figured-out-how-to-get-esrgan-and-sftgan

(also you don't need an NVIDIA card for this just go into test.py and change “device = torch.device(‘cuda’)“ to “ device = torch.device(‘cpu’)”. No AMD/Intel though)
(if you want to try this, make sure you have the image at its native resolution!)
(for the best results, don't use compressed shit off the internet)


NVIDIA has their own Generative Adverserial Network but you have to sign up to use it as it is still in beta: https://developer.nvidia.com/gwmt

Please read through the whole thread! I won't be updating this OP with newer images and instead'll be posting newer stuff within the thread! There's some cool stuff below!

Explanation:

Long story short, Enhanced Super Resolution Generative Adverserial Network, or ESRGAN, is an upscaling method that is capable of generating realistic textures during single image super-resolution. Basically it's a machine learning technique that uses a generative adverserial network to upres smaller images. By doing it over several passes, it will usually produce an image with more fidelity than methods such as SRCNN and SRGAN. In fact, ESRGAN is based off SRGAN. The difference between the two is that ESRGAN improves on SRGAN's network architecture, adversarial loss and perceptual loss. Furthermore ESRGAN

  • adopts a deeper model using Residual-in-Residual Dense Block (RRDB) without batch normalization layers.
  • employs Relativistic average GAN instead of the vanilla GAN.

Obviously this isn't going to make every image look amazing but it's worth giving a shot. There are some genuinely great stuff out there.



ESRGAN has been used to improve the textures of older games such as Doom and Morrowind. In fact, there's a DOOM texture pack that was released recently using this method.
https://www.doomworld.com/forum/topic/99021-v-0-95-doom-neural-upscale-2x/

For reference, Waifu2x uses Deep Convolutional Neural Networks as opposed to Adverserial Networks to scale 2x the original size.


With NVIDIA's GWMT tools and their own GAN, expect to see some crazier stuff down the line. Any game dev looking into super-resolution will definitely be using NVIDIA's stuff. This is only the precipice!





































 
Last edited:
Oct 27, 2017
5,114
#5
I want to see FFIX's background art improved through this

Like really really badly. More than anything. I'd drop every other game to replay Final Fantasy 9 this way.
 
Last edited:
Oct 25, 2017
1,360
#7
So, think this can be used to create Upscales worth a damn for FF7-9 instead of the blurry mess that current "HD Background" mods for them are?
 
May 17, 2018
1,442
#9
Impvessive!

But the first thing that came to my mind is that years-long Resident Evil 4 HD Project....
I was thinking of those Xbox One X enhanced games, like Red Dead Redemption, and wondering if Microsoft is developing a technology that would allow their console to do that work automatically instead of by hand like now. It'd make a lot of sense, in my opinion.
 
Oct 25, 2017
9,599
#16
im not sure if i like the effect on things that already had a lot of detail, its streaky and contrasty a bit like a photoshop sharpen filter.
Other than that some nice results
 
Oct 27, 2017
5,114
#17
Here's some FF7 stuff

definately less authentic than the blurry upscales we're used to but I'm sure with the right tweaks the results would end up surpassing them. Hyped now for the inevitability that is a truly good looking FFIX upscale
 
Nov 23, 2017
1,737
#19
can someone explain the difference between this and smoothing algorithms? Does this convert the textures as a pack or is it done when the game is run?

Is the difference that it has an input set to look at 10000s etc higher resolution versions of any of these and then apply what it learns, so to speak?
 
OP
OP
vestan
Dec 28, 2017
3,892
#21
definately less authentic than the blurry upscales we're used to but I'm sure with the right tweaks the results would end up surpassing them. Hyped now for the inevitability that is a truly good looking FFIX upscale
im not sure if i like the effect on things that already had a lot of detail, its streaky and contrasty a bit like a photoshop sharpen filter.
Other than that some nice results
tbh this isn't really the best thing for pre-rendered CGs

it's way better for textures imo as demonstrated by morrowind
 
Oct 27, 2017
5,114
#23
tbh this isn't really the best thing for pre-rendered CGs

it's way better for textures imo
it'd be a lot of work but I wonder if better results could be had by breaking up each pre-rendered CG image into parts (landscape, buildings, pathways, etc), running them through separately, and then recombining them all
 
Nov 14, 2017
844
#26
I was thinking of those Xbox One X enhanced games, like Red Dead Redemption, and wondering if Microsoft is developing a technology that would allow their console to do that work automatically instead of by hand like now. It'd make a lot of sense, in my opinion.
This isn't the same thing; texture resolution and rendering resolution are different. Increasing rendering resolution is already "automatic"; the manual adjustments made to X enhanced games are to test for performance, compatibility etc... not create new assets.

Disregard if you're just talking about using an AI to create textures for remasters (though that would most likely still be something done at the production level, not by the console).
 
OP
OP
vestan
Dec 28, 2017
3,892
#27
it'd be a lot of work but I wonder if better results could be had by breaking up each pre-rendered CG image into parts (landscape, buildings, pathways, etc), running them through separately, and then recombining them all
it would take years to do FF7 alone lmao
 
OP
OP
vestan
Dec 28, 2017
3,892
#33
I've already seen enough, that looks mighty impressive and I want to see many games enhanced with this.
i know dolphin lets you dump textures for games so i might mess around a bit with that

also this is VERY hit and miss, should really put that out there. when it works it looks amazing tho.

I enjoy the painterly look of some of these.
same, it's a nice little reimagining
 
Oct 27, 2017
956
Pennsylvania
#34
I enjoy the painterly look of some of these... though I'm not nostalgic about old school pixel-looking art like some are. I imagine this might rub those folks the wrong way.
 
OP
OP
vestan
Dec 28, 2017
3,892
#35
I want to see FFIX's background art improved through this

Like really really badly. More than anything. I'd drop every other game to replay Final Fantasy 9 this way.
Also, here ya go :)

What do you think?

A lot of artifacts but this was just a quick pass. I imagine if someone was serious they could polish this up in Photoshop or something and make it look nicer.

 
May 17, 2018
1,442
#37
This isn't the same thing; texture resolution and rendering resolution are different. Increasing rendering resolution is already "automatic"; the manual adjustments made to X enhanced games are to test for performance, compatibility etc... not create new assets.

Disregard if you're just talking about using an AI to create textures for remasters (though that would most likely still be something done at the production level, not by the console).
I know they are different, but I was thinking in more vague terms. They are investing so heavily in AI and machine learning that I was theorizing that they might use it for something like this; improve old games in new ways.

Imagine a console compatible with games from all its predecessors that automatically upgrades resolution, textures, framerates, lighting... I wonder what the possibilities are now and what will be possible in the near future, but it doesn't sound like sci-fi that much anymore.
 
Oct 25, 2017
2,707
#39
Also, here ya go :)

What do you think?

A lot of artifacts but this was just a quick pass. I imagine if someone was serious they could polish this up in Photoshop or something and make it look nicer.

Wow! It looks like a painting and it looks very pretty.
 
OP
OP
vestan
Dec 28, 2017
3,892
#41
Here's the market from OoT blown up 4x

Original:



ESRGAN:



So is there a way to stick these back into their respective games?
Yes. It has already been done with DOOM and Morrowind like I linked in the OP.

https://www.nexusmods.com/morrowind/mods/46221?tab=files

Holy fucking shit that looks good. So this works with every game? Like say BG1?
Yup but results will vary. Also each and every texture has to be accounted for which isn't easy.
 
Oct 25, 2017
3,011
#42
REmake HD uses upscaled Gamecube assets because Capcom lost original assets. Can’t wait to see someone replace them with these AI takes.
 
Oct 27, 2017
2,606
Connecticut
#50
Impvessive!

But the first thing that came to my mind is that years-long Resident Evil 4 HD Project....
There is no way this could achieve anything close to that Re4 HD Project. They found many of the original real life areas that spots were based off and corrected the textures to correctly look the way they were meant to. Its not simply better texture. The geometry work they have fixed is amazing.

This is super impressive though and I'm loving what I'm seeing.