AI Neural Networks being used to generate HQ textures for older games (You can do it yourself!)

Is this going to change game dev?

  • Yeah

    Votes: 1,109 70.9%
  • Naw

    Votes: 456 29.1%

  • Total voters
    1,565

Durante

Dark Souls Man
Member
Oct 24, 2017
4,664
I'm not shooting the messenger at all, I'm just again surprised at just how entirely dysfunctional the patent process seems to be.

(Even the details are the same -- e.g. taking a hash of the texture and using that to optionally reload and replace it after an artist provided a better one is pretty much exactly what I implemented in DSfix 7 or so years ago, and I don't think it was entirely novel even then)
 
Oct 31, 2017
3,327
I'd love to see what Ultimate Mortal Kombat 3 could look like using these tools, especially since it's got digitised actors.
I think there's a way to get texture dumpingloading working in Project 64, I'll add MK Trilogy to my list of stuff to try once I finish with my current batch of Gamecube games.
 
Oct 25, 2017
217
F-Zero GX HD Texture pack v1.0 is now available for download:
https://mega.nz/#!VJUCCSqa!O85Cd5dK2yK-TmNB8qSaWUUa1fT1QH_V5qbiazErHRU (585 MB, 6367 textures in DDS format)
Original textures were dumped from the NA version of the game (GFZE01). The zip is a resource pack which can be loaded If you're using a somewhat recent release of Dolphin that has support for it.
On Windows you can drop the zip into %UserProfile%\Dolphin Emulator\ResourcePacks. Go into ResourcePack manager and install from there. Otherwise, you can copy the GFZE01 folder inside the zip to %UserProfile%\Dolphin Emulator\Load\Textures
 
Feb 2, 2018
1,349
The Viewtiful Joe mod is here! HENSHIN A GO GO BABY!

The video uses a combination of ESRGAN and a handmade pack that re-does UI elements, effects, items, and a handful of characters. All of the environments are ESRGAN.

Download links:
https://github.com/rapka/dolphin-textures/blob/master/PACKS.md
http://onthegreatsea.tumblr.com/post/133505457500/hypatias-viewtiful-joe-mod-download-link

Screenshots comparison:
http://screenshotcomparison.com/comparison/129069/picture:0
http://screenshotcomparison.com/comparison/129069/picture:1
http://screenshotcomparison.com/comparison/129069/picture:2

Bonus comparison of ESRGAN vs handmade hi-res textures:
http://screenshotcomparison.com/comparison/129071

While obviously the handmade ones are way better, I think it's cool that an algorithm is even comparable to the work done by a professional texture artist.

Coming soon: Tony Hawk's Pro Skater 3!
F-Zero GX HD Texture pack v1.0 is now available for download:
https://mega.nz/#!VJUCCSqa!O85Cd5dK2yK-TmNB8qSaWUUa1fT1QH_V5qbiazErHRU (585 MB, 6367 textures in DDS format)
Original textures were dumped from the NA version of the game (GFZE01). The zip is a resource pack which can be loaded If you're using a somewhat recent release of Dolphin that has support for it.
On Windows you can drop the zip into %UserProfile%\Dolphin Emulator\ResourcePacks. Go into ResourcePack manager and install from there. Otherwise, you can copy the GFZE01 folder inside the zip to %UserProfile%\Dolphin Emulator\Load\Textures
Big thanks to both of you for your tremendous efforts, these have turned out amazingly!

Viewtiful Joe said:
F-Zero GX said:
Album here in case of 'bandwidth exceeded' error

The choice to combine ESRGAN with Hypatias' excellent UI mod has given Viewtiful Joe a pretty stunning unofficial remaster, while I was surprised how noticeably pleasing the GX work was even with the high speed. Again, thank you both so much :D
 
Last edited:
Oct 26, 2017
1,530
Castlevania SotN sprite upscale tests

http://screenshotcomparison.com/comparison/129147

A mix of several methods. Some observations, Reduced Colors model seems to be the best with pixelart of low resolution, GMIC Anisotropic Smoothing applied before upscaling can change the outcome positively (colorbanding etc, reduce harshness of ESRGAN), applied after the upscaling process it can remove noise that was introduced into the image. GMIC smoothig creates a halo in uniform color areas, maybe could be avoided by adjusting the settings, not sure. Otherwise would have to be manually edited out (visible around the Dracula sprite in the blue areas)
Reduced Colors can be more precise than ESRGAN, it's just blurrier, combining two or more upscaled results and masking out the good and bad parts can be neccessary for more truthfulness to the original image.
ReducedColors also gave the most smooth edges but not the most detailed textures, combining the edges and textures from another upscale method can also increase quality.
Very difficult to keep quality and "ground truth" of low res pixelart with ESRGAN, the Nvidia upscaler maybe will be better for this (maybe even a one click upscale with good results?). Didn't try any Waifu2x here, might be even better or worse, dunno (also didn't try different small step upscale or antialiasing methods, too much to try out)
 
Oct 27, 2017
876
I'm not shooting the messenger at all, I'm just again surprised at just how entirely dysfunctional the patent process seems to be.

(Even the details are the same -- e.g. taking a hash of the texture and using that to optionally reload and replace it after an artist provided a better one is pretty much exactly what I implemented in DSfix 7 or so years ago, and I don't think it was entirely novel even then)
That could just mean it's vulnerable to attack by third parties at the USPTO if they try to enforce it against others, meaning they probably won't.

(Although more likely they'll have argued their method is different enough to be novel, the devil being in the detail)
 
Last edited:
Oct 25, 2017
4,702
Oct 26, 2017
1,530
The GMIC Anisotropic Smoothing Filter can reduce blotchy, low res compression artefacting quite well, here is a Twilight Princess Midna texture comparion:

http://screenshotcomparison.com/comparison/129244

The filter was applied before upscaling the textures but the setting/strength of the filter was reduced a bit so it wouldn't kill too much detail (which this is a bad comparison for as there is very little detail, only here and there). The models used were Reduced Color - Esrgan 04 interpolation, I also used that for the alpha textures, which was a mistake looking at the eyes, there is now a transparent ring and I also didn't up/downscale the alpha textures beforehand so they are still quite jaggy.
 
Oct 25, 2017
3,471
Y'know what I wanna see now? Final Fantasy XII.

The way they did the textures in the HD version never completely sat well with me.

Oh, and the PC version of Tales of Symphonia.
 
Oct 29, 2017
56
So a user named Shiryu over at the RE modding forums actually went ahead and used AI Gigapixel + manual touchups to improve REmaster HD's BGs. The results are pretty damn impressive:




More pictures + download link here:
https://reupscaleproject.blogspot.com/p/screenshots.html

The aqua ring renders actually looks pretty decent now, compared to the muddy/noisy vanilla upscale that Capcom did:


As much as I love the upscaling tech on other screens in this thread, on these REmake screens the original looks more detailed. Just look at the loss of detail on the wallpaper designs in the upscale. Even in that first screenshot the grime on the wall near the ladder is smoothed way out and almost removed.
 
Jan 27, 2018
2,214
As much as I love the upscaling tech on other screens in this thread, on these REmake screens the original looks more detailed. Just look at the loss of detail on the wallpaper designs in the upscale. Even in that first screenshot the grime on the wall near the ladder is smoothed way out and almost removed.
The "original" screens are the HD upscale, the "detail" is a crappy edge enhancement & sharpening fitler. He shouldn't have named them the original, the original Gamecube backgrounds didn't look like that.
 
Oct 29, 2017
56
The "original" screens are the HD upscale, the "detail" is a crappy edge enhancement & sharpening fitler. He shouldn't have named them the original, the original Gamecube backgrounds didn't look like that.
Interesting. I guess the best way for them to do this REmake upscale would be to upscale the Gamecube versions.
 

Mato

Member
Oct 27, 2017
438
The GMIC Anisotropic Smoothing Filter can reduce blotchy, low res compression artefacting quite well, here is a Twilight Princess Midna texture comparion:

http://screenshotcomparison.com/comparison/129244

The filter was applied before upscaling the textures but the setting/strength of the filter was reduced a bit so it wouldn't kill too much detail (which this is a bad comparison for as there is very little detail, only here and there). The models used were Reduced Color - Esrgan 04 interpolation, I also used that for the alpha textures, which was a mistake looking at the eyes, there is now a transparent ring and I also didn't up/downscale the alpha textures beforehand so they are still quite jaggy.
Looks great. This is the kind of high res textures I've always wanted for old 3d games. Not the the noisy crap modders usually do.
 
Oct 25, 2017
1,991
Australia
I'm really interested in looking at Riven, unfortunately using SFTGAN (which is what I'm guessing vestan used) I don't get anywhere near the same quality. What am I doing wrong? Tested many values in ESRGAN network interpolation and that yielded nothing close to the level of clarity vestan achieved.

Original Image:



My result:



vestans result:

The bottom one here really is spectacular; something about this process seems incredible well-suited to mid 90s - early 00s pre-rendered CG. I hope the process for setting up the software becomes a bit simpler sometime soon, I'd love to run some of the art from the Zork games through it.
 

J75

Member
Sep 29, 2018
710
Finally figured out how to make this work. Is there a way you can upscale higher than 4X times the original resolution?
I'm assuming upscaling an already upscale image twice gives you bad results?
 
Oct 26, 2017
1,530
Finally figured out how to make this work. Is there a way you can upscale higher than 4X times the original resolution?
I'm assuming upscaling an already upscale image twice gives you bad results?
It doesn't give you bad results it just doesn't do all that much to the overall appearance and just takes forever or you run out of RAM, you might as well scale your image the regular way once it is large enough so the algorithm doesn't have anything to latch onto anymore.

As far as I understand it, the entire process needs to be in a specific range of a low res pixelated image as a source image for the upscaling to have a visible effect. If your image is already too big or too "defined" in it's pixel matrix, then after you've upscaled that image you ask yourself if anything actually improved or even has changed?
That goes so far that you can sometimes trick this system into generating more fine detail by heavily scaling down the source image before upscaling it again to it's original size. So you didn't really upscale that image, you just generated more fine detail out it. I'll try to post an example of this tomorrow but you can often see it right in the source image when you open it up in an image editor / image viewer. If the contrast of fine detail already has several pixels of a smooth anti aliased halo area around it then the upscaling process will just scale that up as is with very little change. The source image has to be inside a certain range of size and pixel structure to give good upscaling results, if you have an image that looks exactly the same after upscaling it but is now just 4 times bigger, try scaling the source image down (to 25% of it's original size) and upscale that again back up to it's original size. You can lose authenticity but you can gain artificial detail, depends on the source image of course.
 
Oct 25, 2017
1,507
The "original" screens are the HD upscale, the "detail" is a crappy edge enhancement & sharpening fitler. He shouldn't have named them the original, the original Gamecube backgrounds didn't look like that.
If people are worried about detail, I think layering the images and messing around with opacity could help bring back some "texture". Or even selectively applying it to parts of images.

There's clearly room for refinement with this kind of tool.
 
Jan 27, 2018
2,214
If people are worried about detail, I think layering the images and messing around with opacity could help bring back some "texture". Or even selectively applying it to parts of images.

There's clearly room for refinement with this kind of tool.
You could slightly sharpen them i suppose, but just not go overboard like Capcom did.
 
Oct 26, 2017
1,530
Here is a comparison of just downscaling the textures 25% and rescaling them up to their original size with ESRGAN, so no real upscaling if you will as the images are the same size after doing this, the pixels are just rearranged (I didn't care much for denoising and color or even consistency here):

Final Fantasy 12 - Fran Rescale Comparison

I assume those textures to be the original but I am not 100% sure as I didn't rip them and maybe someone messed with them. The thing with them was that just upscaling them as is, didn't do anything. No increase in details as their size and pixelstructure was not usable for the algo (although I didn't test several models, so maybe it is possible?).
As previously mentioned, you lose authenticity and detail with this method but you gain artificially created detail too. In theory you could upscale from that result and get a bigger difference out of it.
 
Oct 26, 2017
2,561
I'm not shooting the messenger at all, I'm just again surprised at just how entirely dysfunctional the patent process seems to be.

(Even the details are the same -- e.g. taking a hash of the texture and using that to optionally reload and replace it after an artist provided a better one is pretty much exactly what I implemented in DSfix 7 or so years ago, and I don't think it was entirely novel even then)
Just FYI: The validity of a patent can be challenged even after that patent has been granted.
 
Jan 16, 2019
13
For my FF IX mod, I end up adding a lot of pre and post processing (de-dithering, sharpening, denoising) that took a long time to process.
So I'm retraining a ESGRAN network with the original background (no process) as input and the current version of my mod as output, and now, I will have a network that can get rid of dithering artefacts and have more details than the original process :

https://twitter.com/Ze_PilOt/status/1092709418276438016

I'm not sure that network will be usuable with any other kind of texture, but as many people use manga109 for stuff that are not drawed, it might work (probably for PSX textures)
 
Oct 26, 2017
1,530
For my FF IX mod, I end up adding a lot of pre and post processing (de-dithering, sharpening, denoising) that took a long time to process.
So I'm retraining a ESGRAN network with the original background (no process) as input and the current version of my mod as output, and now, I will have a network that can get rid of dithering artefacts and have more details than the original process :

https://twitter.com/Ze_PilOt/status/1092709418276438016

I'm not sure that network will be usuable with any other kind of texture, but as many people use manga109 for stuff that are not drawed, it might work (probably for PSX textures)
No matter the result, I would like to try that model when finished (if you want to share of course)
 
Feb 2, 2018
1,349
For my FF IX mod, I end up adding a lot of pre and post processing (de-dithering, sharpening, denoising) that took a long time to process.
So I'm retraining a ESGRAN network with the original background (no process) as input and the current version of my mod as output, and now, I will have a network that can get rid of dithering artefacts and have more details than the original process :

https://twitter.com/Ze_PilOt/status/1092709418276438016

I'm not sure that network will be usuable with any other kind of texture, but as many people use manga109 for stuff that are not drawed, it might work (probably for PSX textures)
Thank you for your continued amazing work :) Exciting to see further opportunities and experimentation continue with this already amazing tech
 
Jan 16, 2019
13
I wouldn't say that enough : training your dataset accordingly to what you are upscaling is the most important thing.

Here is direct comparaison of pre-existing techniques/dataset VS custom training. Upscaled from PSX, no pre or post process.

For some reasons, Gigapixels is offsetting the image slightly. That could be a huge issue when doing games upscales.

http://www.framecompare.com/image-compare/screenshotcomparison/199JNNNU

Noise management/training is one of the important part when dealing with compressed/altered image (in our case : dithering to a reduced color palette). Here is another exemple of the different solutions dealing with a noisy image :

http://www.framecompare.com/image-compare/screenshotcomparison/Y777NNNX
 
Last edited:
Oct 26, 2017
1,530
I wouldn't say that enough : training your dataset accordingly to what you are upscaling is the most important thing.

Here is direct comparaison of pre-existing techniques/dataset VS custom training. Upscaled from PSX, no pre or post process.

For some reasons, Gigapixels is offsetting the image slightly. That could be a huge issue when doing games upscales.

http://www.framecompare.com/image-compare/screenshotcomparison/199JNNNU

Noise management/training is one of the important part when dealing with compressed/altered image (in our case : dithering to a reduced color palette). Here is another exemple of the different solutions dealing with a noisy image :

http://www.framecompare.com/image-compare/screenshotcomparison/Y777NNNX

I always assumed that the noise reduction was as seperate process from the upscaling part in software like Waifu2x/Gigapixel etc. because they offer 3-4 steps of noise reduction intensity, did they train that into the models? (is your model training finished?)
 
Jan 16, 2019
13
I always assumed that the noise reduction was as seperate process from the upscaling part in software like Waifu2x/Gigapixel etc. because they offer 3-4 steps of noise reduction intensity, did they train that into the models? (is your model training finished?)
Yes, Waifu is using different models / parameters depending of what you are asking.

For the training, it just reached epoch 30 and I still see some improvements over epoch 25, so no, it's not finished :)