• An old favorite feature returns: Q&ERA is back! This time we'll be collecting questions for Remedy Entertainment, makers of Max Payne, Alan Wake, Quantum Break, and Control. Members can submit questions for the next 12 hours, 50 minutes, 15 seconds. Submissions will close on Dec 12, 2019 at 12:00 AM.

AI Neural Networks being used to generate HQ textures for older games (You can do it yourself!)

Is this going to change game dev?

  • Yeah

    Votes: 1,234 71.0%
  • Naw

    Votes: 505 29.0%

  • Total voters
    1,739

Lkr

Member
Oct 28, 2017
1,286
That's exactly what this method was created for, yes. You want a network trained on natural images, not illustrations, though (so, skip the manga109 one).
Cool I’ll look into this when i get home from visiting the family. I have some really old pics from a 20 year old digital camera that I would like to see if I can improve
 

Gifmaker

Member
Oct 29, 2017
732
Next thing you know they'll be able to detect where a light source is coming from and improve the lighting to modern levels in a retro game.
Some time after they'll be able to auto generate levels that are intelligently and intricately designed from being fed examples of levels in games.

Think of this applied to everything. First improved, then ability to create from scratch. Animations, levels, and then entire fucking games. Just like they're already able to do now with fake celebs and fake photos of any scenario after being trained on sets of those images. But it's not gonna stop there.
It's like automatization of creativity. It's equally scary and impressive/amazing. It will be interesting to see where this is headed - AI generated entertainment sounds both exciting and potentially devastating for entire industries.
 

Coreda

SVG Wizard
Member
Oct 26, 2017
886
Does this allow for taking a low res, compressed photo and enlarging it? I know photoshop will do this with a bicubic sharpener already but that doesn’t create a great image
I'll just add that the models linked on the Github repo were trained with non-lossy images so they expect a source without too many artifacts. For various photos I've tested with a decent resolution/fidelity to begin with this isn't an issue but it's kind of a coin flip for some others. Best to do some tests.
 

JoshuaJSlone

Member
Dec 27, 2017
202
Anyone do Yoshi's Island yet?
What happens if you do pc98ish stuff
My understanding of this is it compares images to downsampled versions of the same images, and from it tries to learn the methods to best reconstruct the large images from the small images. Once it's been trained to do this, the same techniques can be applied to other images.

However, old sprite stuff with things like harsh edges and color dithering isn't going to be helped much because a high res image isn't going to look like that when scaled down, so it's not the sort of thing it's trained to know what to do with. More basic things like EAGLE or HQ3x are better suited for that type of input.
 
Oct 26, 2017
1,336
New Orleans
This is incredible. And I love that the technique maintains the original art style and color palette, so that there aren't questionable changes made like the Majora's Mask Moon when remastered by people.
 

Necrocorpse

Member
Nov 17, 2018
149
London, UK
What is this sorcery? Getting excited about computing for first time in years. Couple of years from now we will have something incredible especially when the big software developers will step in and implement similar way into their applications - I can imagine having an AI batch conversion in Photoshop and Nuke etc... It all comes down to the training sets to match the original art in question to get acceptable results. I feel bit sorry for the Resident Evil 4 guy though (maybe not yet!).
 
Last edited:

bulletyen

Member
Nov 12, 2017
1,309
It's like automatization of creativity. It's equally scary and impressive/amazing. It will be interesting to see where this is headed - AI generated entertainment sounds both exciting and potentially devastating for entire industries.
Yeah what a time we live in. The images in the OP shows a glimpse of what machine imagination looks like, filling in the blanks, the void with precise detail. And what a fool I was to think automation would never displace creative people in creative industries. It is exciting indeed because this can speed things up and create things that weren't possible due to time or manpower constraints when used to supplement traditional methods, but also at some point it could be doing the majority of the workload and yes, that could be devastating.
 
Oct 27, 2017
4,130
Somewhere South
My understanding of this is it compares images to downsampled versions of the same images, and from it tries to learn the methods to best reconstruct the large images from the small images. Once it's been trained to do this, the same techniques can be applied to other images.
Yup, basically. The learning process develops an internal degradation model - a mapping form high-resolution to low-resolution - that it will then reverse and apply to LR images to get their HR equivalents.

However, old sprite stuff with things like harsh edges and color dithering isn't going to be helped much because a high res image isn't going to look like that when scaled down, so it's not the sort of thing it's trained to know what to do with.
The degradation model learned from natural images doesn't apply to pixel art because this isn't how LR pixel art was "degraded" in the first place. In fact, ESRGAN suffers from a related problem since it is learning its degradation model from synthetic LR images - achieved through algorithms like bilinear, bicubic or nearest neighbor downsampling. That's not how you get natural LR images, though, and that's one source for the artifacts you see on images it superesolves, since it doesn't deal well with noise, for instance.

One thing that can be done is enhancing the dataset it learns on to include image pairs with the degradation patterns we want it to learn. Dithering is an easy one, just a matter of making a bunch of gradient patches with 2 and more colors, in various orientations and gradient shapes, and then index the colors of these patches using the most common dithering patterns (diffusion, pattern, noise, etc).

One could use vector illustrations to teach it a model for hard edges, dark outlines, etc, you usually find in character sprites.
 

JoshuaJSlone

Member
Dec 27, 2017
202
Yeah what a time we live in. The images in the OP shows a glimpse of what machine imagination looks like, filling in the blanks, the void with precise detail. And what a fool I was to think automation would never displace creative people in creative industries.
It's harder to imagine when machines will be doing stuff we consider REALLY creative, but enough work is "creative", that a machine can be taught a format and come up with a million competent results by the time a regular person has made one. Data in Star Trek TNG had an awful time with humor--and while maybe he should've had trouble understanding why something is funny, he'd be able to think of a billion different shiny things to compare Picard's head to that would entertain his crewmates.
 

bulletyen

Member
Nov 12, 2017
1,309
It's harder to imagine when machines will be doing stuff we consider REALLY creative, but enough work is "creative", that a machine can be taught a format and come up with a million competent results by the time a regular person has made one. Data in Star Trek TNG had an awful time with humor--and while maybe he should've had trouble understanding why something is funny, he'd be able to think of a billion different shiny things to compare Picard's head to that would entertain his crewmates.
Yep, they might never come up with truly original ideas or the best jokes, but then again neither do a lot of creatives. Much of our of entertainment is mediocre and derivative, after all. With enough statistical and situational information to draw from, machines will begin to understand what gets a human to react.

Working in the TV animation industry, I can tell you that so much of what we do can be done by automation in the future given where it shockingly already is. So many things are rote and formulaic, including stories, design, and action, that a machine will eventually be able to grasp the formulas and principles at every level and come up with ones that are similar but different to what we've done so many times before.

I agree, truly and wildly creative works and ideas at the moment genuinely require human wit and twists, as I sure as hell hope a machine will never be able to make something like a Lars Von Trier or Sono Sion film in its entirety. But I sure as hell can imagine them someday pumping out Marvel movies with the littlest of human input. Sooner or later just plug some Joseph Campbell and Robert McKee into its learning along with the millions of examples which follow these rules of story, introduce an element of randomness, and bam you've got AI generated films.
 
OP
OP
FGH

FGH

Member
Dec 28, 2017
12,800
Ended up running the majority of Sonic Colors through Gigapixel:



Haven't watched the video yet but I think you'd be able to do a better job showcasing the effect with some before and afters in the vid itself.

Other than that, not really notcing that much of a difference? It's there but I wouldn't say it's substantial or anything.

To be fair, Sonic Colors looks decent enough as it is.
 

Kyle Cross

Member
Oct 25, 2017
4,259
The guy has confirmed on Twitter that he is looking on how to mod the game with the results:

Can't wait...
Is he likely to succeed, how hard is it to mod FFIX? It's on sale right now and this would be the deciding factor on me buying it.

I also want someone to do a texture mod of Final Fantasy XII: The Zodiac Age with this, as all the devs of that did was make a photoshop action of resizing the images and applying a couple filters to a folder containing the games textures. I really don't like the vaseline look they have.
 

Tagg

Member
Oct 25, 2017
1,361
Here's some more from Chrono Cross. They certainly aren't perfect (just look at the sky) but still very impressive..








 

Necrocorpse

Member
Nov 17, 2018
149
London, UK
Not too shabby at all. Just thinking, what would happen if you process the bottom (with white buildings) source image first with degrain (or some waifu2x..?) to get rid of the initial graininess and then run the AI pass?
Maybe I should install the software on my machine and do some testing. It's more fruitful than browsing the web (dailymail :) all the time.
 
Last edited:

YannStone

Member
Oct 27, 2017
15
Is he likely to succeed, how hard is it to mod FFIX? It's on sale right now and this would be the deciding factor on me buying it.

I also want someone to do a texture mod of Final Fantasy XII: The Zodiac Age with this, as all the devs of that did was make a photoshop action of resizing the images and applying a couple filters to a folder containing the games textures. I really don't like the vaseline look they have.
I can't guarantee for him that he will be able to do it...but this seems to be feasible since someone already did it (updating the background with Waifu...which results are.."meh"):
https://steamcommunity.com/sharedfiles/filedetails/?id=927391278
 

elyetis

Member
Oct 26, 2017
2,122
I can't guarantee for him that he will be able to do it...but this seems to be feasible since someone already did it (updating the background with Waifu...which results are.."meh"):
https://steamcommunity.com/sharedfiles/filedetails/?id=927391278
I nver took the time to try, but before that fan patch, imho the best visual was to play with a CRT shader ( cf : https://steamcommunity.com/sharedfiles/filedetails/?id=668831565 ) :



Maybe combining both ( or even better background using in part the AI tool ) could be the best solution ( thought from the little I've seen, Ze PilOt's background seem good enough not to need any crt shader to hide imperfections ).
 

prarts

Member
Oct 30, 2017
135
Extracted all backgrounds from Monkey Island 4 (EfMI) and fed them to AI. They go beautifully from 640x480 to... 2560x1920! That`s more than full HD, insane insanity.
Take a look:



I mean, second screen looks like original rendered background (take a look at those details) and the first one like its downgraded version.
 

Laser Man

Member
Oct 26, 2017
2,580
Extracted all backgrounds from Monkey Island 4 (EfMI) and fed them to AI. They go beautifully from 640x480 to... 2560x1920! That`s more than full HD, insane insanity.
Take a look:



I mean, second screen looks like original rendered background (take a look at those details) and the first one like its downgraded version.
You were using the Manga model, right? (did you run on CPU or GPU and how long did one image take?)
 

prarts

Member
Oct 30, 2017
135
You were using the Manga model, right? (did you run on CPU or GPU and how long did one image take?)
Manga model ran through 1080 (CUDA). Really low res pictures (VGA - like DOS SCUMM games) takes less then 1 second, it`s almost instant - that`s way I was thinking of implementing this in ScummVM "on the fly". This ones took a little longer (2~3 seconds). Card is going at full blast when processing.
 

Kyle Cross

Member
Oct 25, 2017
4,259
Extracted all backgrounds from Monkey Island 4 (EfMI) and fed them to AI. They go beautifully from 640x480 to... 2560x1920! That`s more than full HD, insane insanity.
Take a look:



I mean, second screen looks like original rendered background (take a look at those details) and the first one like its downgraded version.
Damn, that does look nice! I wish Curse looked this nice when fed through. I recently bought Curse and can't wait to sit down and finally play it, but the aliasing is quite high even with the smoothing on.
 
Dec 28, 2018
9
Extracted all backgrounds from Monkey Island 4 (EfMI) and fed them to AI. They go beautifully from 640x480 to... 2560x1920! That`s more than full HD, insane insanity.
Take a look:



I mean, second screen looks like original rendered background (take a look at those details) and the first one like its downgraded version.
Problem is that the manga models changes the contrast of the picture. I noticed the same when I trained NeuralEnhancer (based on the og SRGAN) back in the days. I wonder why.
 
Oct 29, 2017
737
Extracted all backgrounds from Monkey Island 4 (EfMI) and fed them to AI. They go beautifully from 640x480 to... 2560x1920! That`s more than full HD, insane insanity.
Take a look:



I mean, second screen looks like original rendered background (take a look at those details) and the first one like its downgraded version.
This is incredible
 

thetrin

Content Acquisition at DANGEN Entertainment
Verified
Oct 26, 2017
2,205
Osaka Japan
This is incredible stuff. I’d love to see a combination of this tech and hand-done patch ups being used to re-release old games. Titles that use prerendered backgrounds clearly benefit a ton from this.
 

Coreda

SVG Wizard
Member
Oct 26, 2017
886
Problem is that the manga models changes the contrast of the picture. I noticed the same when I trained NeuralEnhancer (based on the og SRGAN) back in the days. I wonder why.
The Manga109 model introduces scanned textures to the output, since it was trained with images that contained them (manga illustrations on scanned pages). If you upscale the original image then compare them both it's clearer to see how the textures and halftone color noise are affecting the surfaces in subtle ways which has the effect of changing the contrast in places subtly.

It seems pretty much that where it wants to fill in details it's implementing surfaces that approximate what it's been trained with, such as in the below crop where we can see patches of halftone texture and color noise where there was none to begin with. It's not necessarily something you'd want for every type of upscale but is a distinctive characteristic of its training.

 

Kyle Cross

Member
Oct 25, 2017
4,259
So wait, this Manga AI was trained specifically on just manga? Then yeah, it has no business be used on non-black and white 2D drawings.
 

Coreda

SVG Wizard
Member
Oct 26, 2017
886
So wait, this Manga AI was trained specifically on just manga? Then yeah, it has no business be used on non-black and white 2D drawings.
The actual Manga109 dataset includes full color covers (and possibly pages) not only black and white illustrations. However I'm not sure if the person who trained the model used the full dataset since afaict it requires some approval process to gain access to it, unlike the color covers-only dataset which the author of ESRGAN linked on the repo (which is far more limited in scope, being just 109 images of the covers of the manga from the full dataset).

Put briefly, it's possible the Manga109 model was trained exclusively on color manga covers.
 

Solace

Member
Oct 29, 2017
1,005
What the actual fucking fuck?!! How is this possible, shit, game devs had a good run!
 

Ladioss

Banned
Oct 29, 2017
847
Just a thought, probably idiotic, but could we use the tech for destructive, highly-compression schemes of image/sounds ?
 

Barrel Cannon

The Fallen
Oct 25, 2017
5,052
This is fucking mind blowing. I never thought the applications of big data could be used like this. I'm interested in seeing how fast this runs in real time though
 

JoshuaJSlone

Member
Dec 27, 2017
202
Just a thought, probably idiotic, but could we use the tech for destructive, highly-compression schemes of image/sounds ?
Well, that is pretty much what it is already. It tries to learn how to reconstruct the original version of an image from a smaller one that's lost a lot of the information. It's just that the same learned techniques can then be applied to generic input.
 

Greebman

Member
Oct 26, 2017
2,260
Man, I finally remembered the game I was thinking of that had 2D backgrounds that could really use a high res version. It's Grim Fandango. Got the remaster on PC, extracted some backgrounds and ran them through with the Manga model and man these results are insane. They can just stick these in with an update and you get a good presentation even on a 4K display.

Here are some comparison gifs:








Album with the full-res before and after pics:
https://imgur.com/a/uZJPXvq
 

starfox

Banned
Oct 28, 2017
1,341
Portugal
Man, I finally remembered the game I was thinking of that had 2D backgrounds that could really use a high res version. It's Grim Fandango. Got the remaster on PC, extracted some backgrounds and ran them through with the Manga model and man these results are insane. They can just stick these in with an update and you get a good presentation even on a 4K display.
Should I play the remastered version or wait for someone to mod the game with this INSANITY!

Incredible.
 

Raster

Member
Feb 28, 2018
2,164
On a slightly unrelated note, I think next-gen we're going to see a lot more AI being used for rendering techniques and optimization. Really looking forward to this.
 

Coreda

SVG Wizard
Member
Oct 26, 2017
886
Man, I finally remembered the game I was thinking of that had 2D backgrounds that could really use a high res version. It's Grim Fandango. Got the remaster on PC, extracted some backgrounds and ran them through with the Manga model and man these results are insane. They can just stick these in with an update and you get a good presentation even on a 4K display.
Among the better examples in this thread (apart from office blinds, office corridor bas relief detail in parts, and the kitchen collage shot which could use some extra refinements). The limited color palette patterned dither in the originals lends itself well to the texture of the Manga109 model upscaling. More than probably any other examples in this thread beside the Monkey Island 4 shot I'd say these are the most consistently solid upscales. I'd absolutely play the game modded like this.

Edit: the limited palette is in part caused by the GIF though looking at the gallery the dither pattern is also in the source. This thread would actually be a decent place to showcase SVG animated image comparisons using PNGs and JPEGs.
 
Last edited:

jett

Community Resettler
Member
Oct 25, 2017
20,287
Here's some more from Chrono Cross. They certainly aren't perfect (just look at the sky) but still very impressive..


Despite some of the bugs this is one of the most impressive pictures in the thread. It's like the AI managed to guess that the intended objective of Chrono Cross was to look painterly, which barely comes across in the original image.

Man, I finally remembered the game I was thinking of that had 2D backgrounds that could really use a high res version. It's Grim Fandango. Got the remaster on PC, extracted some backgrounds and ran them through with the Manga model and man these results are insane. They can just stick these in with an update and you get a good presentation even on a 4K display.

Here are some comparison gifs:








Album with the full-res before and after pics:
https://imgur.com/a/uZJPXvq
Oh man this is amazing too.

What's even more amazing is that this tool is free.