AI Neural Networks being used to generate HQ textures for older games (You can do it yourself!)

Is this going to change game dev?

  • Yeah

    Votes: 1,217 70.8%
  • Naw

    Votes: 501 29.2%

  • Total voters
    1,718

vestan

Member
Dec 28, 2017
9,236
Credit to Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, Wenzhe Shi/Arxiv on their paper about Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. I suggest you give it a read if you're interested in this kind of thing.

https://arxiv.org/pdf/1609.04802.pdf

Credit to Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, Xiaoou Tang for their paper on ESRGAN and for providing the ESRGAN repo with pretrained models

https://arxiv.org/pdf/1809.00219.pdf
https://github.com/xinntao/ESRGAN


Guide if ya wanna try it yourself:
Credit to kingdomakrillic their amazing work!


https://kingdomakrillic.tumblr.com/post/178254875891/i-figured-out-how-to-get-esrgan-and-sftgan

(also you don't need an NVIDIA card for this just go into test.py and change “device = torch.device(‘cuda’)“ to “ device = torch.device(‘cpu’)”. No AMD/Intel though)
(if you want to try this, make sure you have the image at its native resolution!)
(for the best results, don't use compressed shit off the internet)


NVIDIA has their own Generative Adverserial Network but you have to sign up to use it as it is still in beta: https://developer.nvidia.com/gwmt

Please read through the whole thread! I won't be updating this OP with newer images and instead'll be posting newer stuff within the thread! There's some cool stuff below!

Explanation:

Long story short, Enhanced Super Resolution Generative Adverserial Network, or ESRGAN, is an upscaling method that is capable of generating realistic textures during single image super-resolution. Basically it's a machine learning technique that uses a generative adverserial network to upres smaller images. By doing it over several passes, it will usually produce an image with more fidelity than methods such as SRCNN and SRGAN. In fact, ESRGAN is based off SRGAN. The difference between the two is that ESRGAN improves on SRGAN's network architecture, adversarial loss and perceptual loss. Furthermore ESRGAN

  • adopts a deeper model using Residual-in-Residual Dense Block (RRDB) without batch normalization layers.
  • employs Relativistic average GAN instead of the vanilla GAN.

Obviously this isn't going to make every image look amazing but it's worth giving a shot. There are some genuinely great stuff out there.



ESRGAN has been used to improve the textures of older games such as Doom and Morrowind. In fact, there's a DOOM texture pack that was released recently using this method.
https://www.doomworld.com/forum/topic/99021-v-0-95-doom-neural-upscale-2x/

For reference, Waifu2x uses Deep Convolutional Neural Networks as opposed to Adverserial Networks to scale 2x the original size.


With NVIDIA's GWMT tools and their own GAN, expect to see some crazier stuff down the line. Any game dev looking into super-resolution will definitely be using NVIDIA's stuff. This is only the precipice!





































 
Last edited:

J75

Member
Sep 29, 2018
1,080
I wonder how the pre rendered Resident Evil backgrounds will fare with this.
 

Solid

Member
Oct 27, 2017
6,665
I want to see FFIX's background art improved through this

Like really really badly. More than anything. I'd drop every other game to replay Final Fantasy 9 this way.
 
Last edited:

SaberVS7

Member
Oct 25, 2017
1,671
So, think this can be used to create Upscales worth a damn for FF7-9 instead of the blurry mess that current "HD Background" mods for them are?
 

Tetrinski

Member
May 17, 2018
2,329
Impvessive!

But the first thing that came to my mind is that years-long Resident Evil 4 HD Project....
I was thinking of those Xbox One X enhanced games, like Red Dead Redemption, and wondering if Microsoft is developing a technology that would allow their console to do that work automatically instead of by hand like now. It'd make a lot of sense, in my opinion.
 

lvl 99 Pixel

Member
Oct 25, 2017
12,788
im not sure if i like the effect on things that already had a lot of detail, its streaky and contrasty a bit like a photoshop sharpen filter.
Other than that some nice results
 
Nov 23, 2017
2,694
can someone explain the difference between this and smoothing algorithms? Does this convert the textures as a pack or is it done when the game is run?

Is the difference that it has an input set to look at 10000s etc higher resolution versions of any of these and then apply what it learns, so to speak?
 
OP
OP
vestan

vestan

Member
Dec 28, 2017
9,236
definately less authentic than the blurry upscales we're used to but I'm sure with the right tweaks the results would end up surpassing them. Hyped now for the inevitability that is a truly good looking FFIX upscale
im not sure if i like the effect on things that already had a lot of detail, its streaky and contrasty a bit like a photoshop sharpen filter.
Other than that some nice results
tbh this isn't really the best thing for pre-rendered CGs

it's way better for textures imo as demonstrated by morrowind
 

Solid

Member
Oct 27, 2017
6,665
tbh this isn't really the best thing for pre-rendered CGs

it's way better for textures imo
it'd be a lot of work but I wonder if better results could be had by breaking up each pre-rendered CG image into parts (landscape, buildings, pathways, etc), running them through separately, and then recombining them all
 

Poimandres

Member
Oct 26, 2017
2,288
I'm truly gobsmacked by how effective this is. Can't wait to see results for this applied to full games, bless you PC gaming!
 
Nov 14, 2017
985
I was thinking of those Xbox One X enhanced games, like Red Dead Redemption, and wondering if Microsoft is developing a technology that would allow their console to do that work automatically instead of by hand like now. It'd make a lot of sense, in my opinion.
This isn't the same thing; texture resolution and rendering resolution are different. Increasing rendering resolution is already "automatic"; the manual adjustments made to X enhanced games are to test for performance, compatibility etc... not create new assets.

Disregard if you're just talking about using an AI to create textures for remasters (though that would most likely still be something done at the production level, not by the console).
 
OP
OP
vestan

vestan

Member
Dec 28, 2017
9,236
it'd be a lot of work but I wonder if better results could be had by breaking up each pre-rendered CG image into parts (landscape, buildings, pathways, etc), running them through separately, and then recombining them all
it would take years to do FF7 alone lmao
 

c0de

Banned
Oct 28, 2017
3,202
I've already seen enough, that looks mighty impressive and I want to see many games enhanced with this.
 
OP
OP
vestan

vestan

Member
Dec 28, 2017
9,236
I've already seen enough, that looks mighty impressive and I want to see many games enhanced with this.
i know dolphin lets you dump textures for games so i might mess around a bit with that

also this is VERY hit and miss, should really put that out there. when it works it looks amazing tho.

I enjoy the painterly look of some of these.
same, it's a nice little reimagining
 

Evolved1

Member
Oct 27, 2017
1,241
Pennsylvania
I enjoy the painterly look of some of these... though I'm not nostalgic about old school pixel-looking art like some are. I imagine this might rub those folks the wrong way.
 
OP
OP
vestan

vestan

Member
Dec 28, 2017
9,236
I want to see FFIX's background art improved through this

Like really really badly. More than anything. I'd drop every other game to replay Final Fantasy 9 this way.
Also, here ya go :)

What do you think?

A lot of artifacts but this was just a quick pass. I imagine if someone was serious they could polish this up in Photoshop or something and make it look nicer.

 

Tetrinski

Member
May 17, 2018
2,329
This isn't the same thing; texture resolution and rendering resolution are different. Increasing rendering resolution is already "automatic"; the manual adjustments made to X enhanced games are to test for performance, compatibility etc... not create new assets.

Disregard if you're just talking about using an AI to create textures for remasters (though that would most likely still be something done at the production level, not by the console).
I know they are different, but I was thinking in more vague terms. They are investing so heavily in AI and machine learning that I was theorizing that they might use it for something like this; improve old games in new ways.

Imagine a console compatible with games from all its predecessors that automatically upgrades resolution, textures, framerates, lighting... I wonder what the possibilities are now and what will be possible in the near future, but it doesn't sound like sci-fi that much anymore.
 
OP
OP
vestan

vestan

Member
Dec 28, 2017
9,236
Here's the market from OoT blown up 4x

Original:



ESRGAN:



So is there a way to stick these back into their respective games?
Yes. It has already been done with DOOM and Morrowind like I linked in the OP.

https://www.nexusmods.com/morrowind/mods/46221?tab=files

Holy fucking shit that looks good. So this works with every game? Like say BG1?
Yup but results will vary. Also each and every texture has to be accounted for which isn't easy.
 

Televator

Member
Oct 25, 2017
4,430
REmake HD uses upscaled Gamecube assets because Capcom lost original assets. Can’t wait to see someone replace them with these AI takes.
 

Drain You

Member
Oct 27, 2017
3,301
Connecticut
Impvessive!

But the first thing that came to my mind is that years-long Resident Evil 4 HD Project....
There is no way this could achieve anything close to that Re4 HD Project. They found many of the original real life areas that spots were based off and corrected the textures to correctly look the way they were meant to. Its not simply better texture. The geometry work they have fixed is amazing.

This is super impressive though and I'm loving what I'm seeing.