• iOS/Safari issues when launching the site from mobile home screens should now be fixed! A2HS button compatibility on Android Firefox has also been fixed! Add some purple to your home screen today.

AI Neural Networks being used to generate HQ textures for older games (You can do it yourself!)

Is this going to change game dev?

  • Yeah

    Votes: 876 70.0%
  • Naw

    Votes: 376 30.0%

  • Total voters
    1,252
Oct 26, 2017
1,324
Manga model ran through 1080 (CUDA). Really low res pictures (VGA - like DOS SCUMM games) takes less then 1 second, it`s almost instant - that`s way I was thinking of implementing this in ScummVM "on the fly". This ones took a little longer (2~3 seconds). Card is going at full blast when processing.
I've been trying some other images in CPU mode (I have a cuda card but wanted to try a the non cuda process) and it can take forever (literally minutes) on a 3.7 Ghz i7... depending on the image of course.
 

jett

Community Resettler
Member
Oct 25, 2017
11,857
These Grim Fandango pictures are absurd. I bet plenty of people that worked on remasters this generation sure wish this had been available before. :P
 
Oct 20, 2018
126
Brazil
Seeing these Grim Fandango pictures make me wonder how Day of the Tentacle would look. Who knows, it might even look better than the official remaster :P
 
Dec 7, 2017
886
Hey guys, not super related but has someone found a model trained to actually upres images that are LR from the internet? Like, real life photos you would get from google that would be like 400x400.
I have some photos from my dad's band (that we don't have the source at all) and I'm trying to use ESRGAN but it really doesnt work that well.
 
Oct 27, 2017
198
I'm trying to follow the instructions in the blog, but am stuck on the PyTorch step.

I have the latest Python (3.7.2) and CUDA (10.0.130). Following the QuickStart instructions on pytorch.org, I'm trying to execute this from command line:
Code:
pip3 install https://download.pytorch.org/whl/cu100/torch-1.0.0-cp37-cp37m-win_amd64.whl
pip3 install torchvision
The first pip3 install fails with "torch-1.0.0-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform." message.

Advice from the blog:
If it tells you “x is not a supported wheel on this platform”, make absolutely sure your options match and that your Python installation is 64-bit, or try different versions of Python or CUDA.
is not helpful - my versions match, and choosing different versions gives same error.

Any ideas?

EDIT: Nevermind, I think I see the problem - I needed 64-bit Python installer.
 
Last edited:
Sep 14, 2018
14
Are you using the sprites from spriter's resource?
I'm hoping to get it from the game image. As far as I researched it should be pretty easy.
If not, I'll probably get it from a sprite resource. That should be a good alternative I hadn't thought about. :)
 
Oct 25, 2017
1,297
Used the upscaler as a baseline on the Skyrim mountain texture and added some more manual touchups on top. This texture was already upscaled from 1K to 4K in Skyrim SE but it was done so badly that you can hardly even tell the difference. This is with only the diffuse as I haven't even gotten around to editing the normal and specular materials yet.

http://screenshotcomparison.com/comparison/126992
 
Oct 26, 2017
1,324
I need an open source Jpeg artefact removal algo (there was a link to a project earlier in the thread but that wasn't actual code I believe)... ESRGAN works really good with non compressed images, or high quality low res images. As soon as Jpeg artefacting comes into play it has problems. Photoshop can help a little the Jpeg removal in the filters is very brutal on details when used on low res and highly compressed images. You can smudge and stamp and paint and use noise and blur and and and ... but at some point it is too much work for average results.

I've seen other images online (there is a low res bear image) which can get very detailed by using other ai upscalers... I wasn't able to reproduce the same results they have online with ESRGAN, maybe I didn't use the original image they've been using, I'm not sure.
 

Coreda

SVG Wizard
Member
Oct 26, 2017
874
I need an open source Jpeg artefact removal algo.
waifu2x-caffe has a decent denoiser that can be used without the upscaling. Mind you it won't add things like additional texture detail like ESRGAN but for cleaning sources I use it from time to time.

Here's a photo saved at a very low JPEG quality, and here's the waifu2x-caffe denoise. Comparison crops below, scaled 200% nearest neighbor in PS afterward to better see:



For the above used the Photo model preset, TTA mode and Level 2 denoise.
 
Oct 26, 2017
1,051
How does one make those comparison gif without loosing in color because of the 256 color limitation ? I always end up with bad result when using gimp.
 

Dio

Member
Oct 25, 2017
1,523
Used the upscaler as a baseline on the Skyrim mountain texture and added some more manual touchups on top. This texture was already upscaled from 1K to 4K in Skyrim SE but it was done so badly that you can hardly even tell the difference. This is with only the diffuse as I haven't even gotten around to editing the normal and specular materials yet.

http://screenshotcomparison.com/comparison/126992
holy shit
 
Nov 6, 2017
76
Yes please do more Metroid! The applications of this for video encoding will be spectacular I guess. An AI specifically trained for a certain movie should allow for drastically smaller file sizes. Or are these AI containers large in size? How large are these tools?

Edit: They could even switch AI training sets/models on a scene by scene basis. The possibilities are grand...
 
Last edited:
OP
OP
vestan
Dec 28, 2017
3,892
Yes please do more Metroid! The applications of this for video encoding will be spectacular I guess. An AI specifically trained for a certain movie should allow for drastically smaller file sizes. Or are these AI containers large in size? How large are these tools?
Mines around 700MB. That's with 12 models though. Can easily get it smaller.
 

Vire

Member
Oct 27, 2017
3,058
Wow this is so cool, thank you for bringing this to light. I realize it's not perfect, but that is impressive.
 
Oct 31, 2017
2,804
Soul Calibur 2. Per texture, not post-process.

http://screenshotcomparison.com/comparison/127037

Before:

After:


I couldn't get mipmaps working properly with the modded textures, so I had to disable texture filtering. Looks great regardless though. Does anyone know if you can force anisotropic filtering in Dolphin through the Nvidia control panel?
 
Last edited:
Oct 26, 2017
1,324
waifu2x-caffe has a decent denoiser that can be used without the upscaling. Mind you it won't add things like additional texture detail like ESRGAN but for cleaning sources I use it from time to time.

Here's a photo saved at a very low JPEG quality, and here's the waifu2x-caffe denoise. Comparison crops below, scaled 200% nearest neighbor in PS afterward to better see:



For the above used the Photo model preset, TTA mode and Level 2 denoise.
Interesting, thanks! (seems to be Windows only tho?)

Mines around 700MB. That's with 12 models though. Can easily get it smaller.
Where did you get those other models from? I'm looking for some more to try!
 
Oct 27, 2017
162
I was ready to call those Max Payne textures impossible until I saw the text on Reddit itself:

Textures was made by ESRGAN(Manga109 model) with combined hand-made textures.
Which I think will be the method going forward, really. Filter them but then clean up the stuff the filter can't quite understand.
 
Oct 31, 2017
2,804
Yes please do more Metroid! The applications of this for video encoding will be spectacular I guess. An AI specifically trained for a certain movie should allow for drastically smaller file sizes. Or are these AI containers large in size? How large are these tools?

Edit: They could even switch AI training sets/models on a scene by scene basis. The possibilities are grand...
Metroid you say????? I gotchu covered.



http://screenshotcomparison.com/comparison/127113/picture:0
http://screenshotcomparison.com/comparison/127113/picture:1
http://screenshotcomparison.com/comparison/127113/picture:2
http://screenshotcomparison.com/comparison/127113/picture:3
http://screenshotcomparison.com/comparison/127113/picture:4
 
Nov 6, 2017
76
Very nice! I hope all developers take note and implement this in their remasters with manual touch-ups as suggested.
 
Nov 6, 2017
76
Could someone upres this picture for me with the toon AI?
I only have it in this low resolution, but have no access to this program. Thanks a lot!
 
Nov 6, 2017
76
Does it work on Donkey Kong Country or is it too low res
Wow, is this one actually real?
I've a pretty good idea how this stuff works but that's still striking. You really have to marvel at just how few pixels the network needs to find some really fitting higher-res shapes and textures in a lot of cases.
Yeah it really is surprising how well it knows there have been sharp lines hidden in those pixels, if you look at the ridges between the woodpanels in the table for example. It just looks „right“ most of the time. Just stunning!
 
Oct 25, 2017
4,649
There will be a time where AI can make 3D games visually cohesive in real time, and perhaps even radically changing the visual style.

I'm even wondering if the semi-distant future of game graphics are crude visuals that you tell an AI "futuristic city of style XYZ" and the AI will construct the entire scene (with comparitively little work being done on the rasterization / raytracing side).
 
Oct 25, 2017
1,117
I was gonna run some Dark Souls textures through this but it takes too damn long for larger images to be processed by CPU. No chance of AMD GPU support?
 
Nov 11, 2017
116
Wow, is this one actually real?
I've a pretty good idea how this stuff works but that's still striking. You really have to marvel at just how few pixels the network needs to find some really fitting higher-res shapes and textures in a lot of cases.
Yes, all original image (240p) but 400% resize without filter for comparaison below.
other comparison : http://screenshotcomparison.com/comparison/127161/picture:0




Indeed it's impressive, the operation to find the most identical render. Even if it's "fake", in some cases, it works very well. (Especially the slightly dark images)

In other cases, the rendering is too "sharp" and "crisp"
The fact that 8/16 bit (dithering) colors can also break the result.

Like this :


Could someone upres this picture for me with the toon AI? I only have it in this low resolution, but have no access to this program. Thanks a lot!
Here, I downscale /4, otherwise, full pixelated


What happens if your run the output through it again? Do you get the same thing or does it enhance further?
More calculation, very long and especially very greedy. I try 720p picture and it's too long and my PC freezing.
I have a 3770k i7, 16 GB RAM and a SSD.
I have a GTX 770 but not supported even if could run CUDA, so it use RAM + SSD
 
Oct 28, 2017
22
Australia
What happens if your run the output through it again? Do you get the same thing or does it enhance further?
Edges continue to improve somewhat, but fine details don't gain much relative to the amount of artefacts created.

Shadowrun Returns (crop from 8xSGSSAA screenshot)
Original, 1600% Nearest

Manga109, 400% Nearest
Manga109, Manga109

Interp_08, 400% Nearest
Interp_08, Interp_08
(interp_08 from ESRGAN_x4 and PSNR_x4 models)

It might look better with some extra processing between steps (reducing halos in particular).
 

Durante

Dark Souls Man
Member
Oct 24, 2017
4,309
In other cases, the rendering is too "sharp" and "crisp"
The fact that 8/16 bit (dithering) colors can also break the result.
Yeah, I think it would be interesting to train a model specifically for dithered pictures (e.g. by downsampling and dithering images in the training set).
 
Oct 25, 2017
1,297
In Retroarch there are shaders that detect dither patterns and blend them together without blurring out the rest of the image. There ought to be some software that can do the same thing to image files so you can pre-process them for upscaling.
 
Oct 26, 2017
1,083
I ran Dark Souls Remastered UI icons through this and the results are pretty good. Also put them in proper files for use. It would be nice if someone who has a fully loaded char could test it cause I don't and I don't know if one of the item sheets is messed up maybe.

https://www.nexusmods.com/darksoulsremastered/mods/114/


It unfortunately only works with the remaster because they already use 2k texture files, while DS original messes up when you upscale the UI textures.
 
Last edited: