That's exactly what this method was created for, yes. You want a network trained on natural images, not illustrations, though (so, skip the manga109 one).
-
Ever wanted an RSS feed of all your favorite gaming news sites? Go check out our new Gaming Headlines feed! Read more about it here.
AI Neural Networks being used to generate HQ textures for older games (You can do it yourself!)
- Thread starter vestan
- Start date
That's exactly what this method was created for, yes. You want a network trained on natural images, not illustrations, though (so, skip the manga109 one).
Cool I'll look into this when i get home from visiting the family. I have some really old pics from a 20 year old digital camera that I would like to see if I can improve
OP
OP
It's like automatization of creativity. It's equally scary and impressive/amazing. It will be interesting to see where this is headed - AI generated entertainment sounds both exciting and potentially devastating for entire industries.
I'll just add that the models linked on the Github repo were trained with non-lossy images so they expect a source without too many artifacts. For various photos I've tested with a decent resolution/fidelity to begin with this isn't an issue but it's kind of a coin flip for some others. Best to do some tests.
My understanding of this is it compares images to downsampled versions of the same images, and from it tries to learn the methods to best reconstruct the large images from the small images. Once it's been trained to do this, the same techniques can be applied to other images.
However, old sprite stuff with things like harsh edges and color dithering isn't going to be helped much because a high res image isn't going to look like that when scaled down, so it's not the sort of thing it's trained to know what to do with. More basic things like EAGLE or HQ3x are better suited for that type of input.
This is incredible. And I love that the technique maintains the original art style and color palette, so that there aren't questionable changes made like the Majora's Mask Moon when remastered by people.
What is this sorcery? Getting excited about computing for first time in years. Couple of years from now we will have something incredible especially when the big software developers will step in and implement similar way into their applications - I can imagine having an AI batch conversion in Photoshop and Nuke etc... It all comes down to the training sets to match the original art in question to get acceptable results. I feel bit sorry for the Resident Evil 4 guy though (maybe not yet!).
Last edited:
The guy has confirmed on Twitter that he is looking on how to mod the game with the results:
Can't wait...
Yeah what a time we live in. The images in the OP shows a glimpse of what machine imagination looks like, filling in the blanks, the void with precise detail. And what a fool I was to think automation would never displace creative people in creative industries. It is exciting indeed because this can speed things up and create things that weren't possible due to time or manpower constraints when used to supplement traditional methods, but also at some point it could be doing the majority of the workload and yes, that could be devastating.
Yup, basically. The learning process develops an internal degradation model - a mapping form high-resolution to low-resolution - that it will then reverse and apply to LR images to get their HR equivalents.
The degradation model learned from natural images doesn't apply to pixel art because this isn't how LR pixel art was "degraded" in the first place. In fact, ESRGAN suffers from a related problem since it is learning its degradation model from synthetic LR images - achieved through algorithms like bilinear, bicubic or nearest neighbor downsampling. That's not how you get natural LR images, though, and that's one source for the artifacts you see on images it superesolves, since it doesn't deal well with noise, for instance.
One thing that can be done is enhancing the dataset it learns on to include image pairs with the degradation patterns we want it to learn. Dithering is an easy one, just a matter of making a bunch of gradient patches with 2 and more colors, in various orientations and gradient shapes, and then index the colors of these patches using the most common dithering patterns (diffusion, pattern, noise, etc).
One could use vector illustrations to teach it a model for hard edges, dark outlines, etc, you usually find in character sprites.
It's harder to imagine when machines will be doing stuff we consider REALLY creative, but enough work is "creative", that a machine can be taught a format and come up with a million competent results by the time a regular person has made one. Data in Star Trek TNG had an awful time with humor--and while maybe he should've had trouble understanding why something is funny, he'd be able to think of a billion different shiny things to compare Picard's head to that would entertain his crewmates.
Yep, they might never come up with truly original ideas or the best jokes, but then again neither do a lot of creatives. Much of our of entertainment is mediocre and derivative, after all. With enough statistical and situational information to draw from, machines will begin to understand what gets a human to react.
Working in the TV animation industry, I can tell you that so much of what we do can be done by automation in the future given where it shockingly already is. So many things are rote and formulaic, including stories, design, and action, that a machine will eventually be able to grasp the formulas and principles at every level and come up with ones that are similar but different to what we've done so many times before.
I agree, truly and wildly creative works and ideas at the moment genuinely require human wit and twists, as I sure as hell hope a machine will never be able to make something like a Lars Von Trier or Sono Sion film in its entirety. But I sure as hell can imagine them someday pumping out Marvel movies with the littlest of human input. Sooner or later just plug some Joseph Campbell and Robert McKee into its learning along with the millions of examples which follow these rules of story, introduce an element of randomness, and bam you've got AI generated films.
Since someone asked about upscaling video (and someone else tried it with ESRGAN) there's this one called TecoGAN.
Not seeing a source code around yet but the paper's here:
https://arxiv.org/pdf/1811.09393.pdf
Not seeing a source code around yet but the paper's here:
https://arxiv.org/pdf/1811.09393.pdf
Thanks, it's encouraging to see progress in video upscaling. There's a LOT of PS1-era FMV that could hopefully someday benefit from this sort of thing!
OP
OP
Haven't watched the video yet but I think you'd be able to do a better job showcasing the effect with some before and afters in the vid itself.
Other than that, not really notcing that much of a difference? It's there but I wouldn't say it's substantial or anything.
To be fair, Sonic Colors looks decent enough as it is.
Is he likely to succeed, how hard is it to mod FFIX? It's on sale right now and this would be the deciding factor on me buying it.
I also want someone to do a texture mod of Final Fantasy XII: The Zodiac Age with this, as all the devs of that did was make a photoshop action of resizing the images and applying a couple filters to a folder containing the games textures. I really don't like the vaseline look they have.
Not too shabby at all. Just thinking, what would happen if you process the bottom (with white buildings) source image first with degrain (or some waifu2x..?) to get rid of the initial graininess and then run the AI pass?
Maybe I should install the software on my machine and do some testing. It's more fruitful than browsing the web (dailymail :) all the time.
Maybe I should install the software on my machine and do some testing. It's more fruitful than browsing the web (dailymail :) all the time.
Last edited:
I can't guarantee for him that he will be able to do it...but this seems to be feasible since someone already did it (updating the background with Waifu...which results are.."meh"):
https://steamcommunity.com/sharedfiles/filedetails/?id=927391278
I nver took the time to try, but before that fan patch, imho the best visual was to play with a CRT shader ( cf : https://steamcommunity.com/sharedfiles/filedetails/?id=668831565 ) :
Maybe combining both ( or even better background using in part the AI tool ) could be the best solution ( thought from the little I've seen, Ze PilOt's background seem good enough not to need any crt shader to hide imperfections ).
Interesting. The modder even knows about it in the comments of that too. Guess I'll buy it.
Extracted all backgrounds from Monkey Island 4 (EfMI) and fed them to AI. They go beautifully from 640x480 to... 2560x1920! That`s more than full HD, insane insanity.
Take a look:
I mean, second screen looks like original rendered background (take a look at those details) and the first one like its downgraded version.
Take a look:
I mean, second screen looks like original rendered background (take a look at those details) and the first one like its downgraded version.
Extracted all backgrounds from Monkey Island 4 (EfMI) and fed them to AI. They go beautifully from 640x480 to... 2560x1920! That`s more than full HD, insane insanity.
Take a look:
I mean, second screen looks like original rendered background (take a look at those details) and the first one like its downgraded version.
You were using the Manga model, right? (did you run on CPU or GPU and how long did one image take?)
Manga model ran through 1080 (CUDA). Really low res pictures (VGA - like DOS SCUMM games) takes less then 1 second, it`s almost instant - that`s way I was thinking of implementing this in ScummVM "on the fly". This ones took a little longer (2~3 seconds). Card is going at full blast when processing.
Damn, that does look nice! I wish Curse looked this nice when fed through. I recently bought Curse and can't wait to sit down and finally play it, but the aliasing is quite high even with the smoothing on.Extracted all backgrounds from Monkey Island 4 (EfMI) and fed them to AI. They go beautifully from 640x480 to... 2560x1920! That`s more than full HD, insane insanity.
Take a look:
I mean, second screen looks like original rendered background (take a look at those details) and the first one like its downgraded version.
Problem is that the manga models changes the contrast of the picture. I noticed the same when I trained NeuralEnhancer (based on the og SRGAN) back in the days. I wonder why.Extracted all backgrounds from Monkey Island 4 (EfMI) and fed them to AI. They go beautifully from 640x480 to... 2560x1920! That`s more than full HD, insane insanity.
Take a look:
I mean, second screen looks like original rendered background (take a look at those details) and the first one like its downgraded version.
This is incredibleExtracted all backgrounds from Monkey Island 4 (EfMI) and fed them to AI. They go beautifully from 640x480 to... 2560x1920! That`s more than full HD, insane insanity.
Take a look:
I mean, second screen looks like original rendered background (take a look at those details) and the first one like its downgraded version.
This is incredible stuff. I'd love to see a combination of this tech and hand-done patch ups being used to re-release old games. Titles that use prerendered backgrounds clearly benefit a ton from this.
The Manga109 model introduces scanned textures to the output, since it was trained with images that contained them (manga illustrations on scanned pages). If you upscale the original image then compare them both it's clearer to see how the textures and halftone color noise are affecting the surfaces in subtle ways which has the effect of changing the contrast in places subtly.
It seems pretty much that where it wants to fill in details it's implementing surfaces that approximate what it's been trained with, such as in the below crop where we can see patches of halftone texture and color noise where there was none to begin with. It's not necessarily something you'd want for every type of upscale but is a distinctive characteristic of its training.
So wait, this Manga AI was trained specifically on just manga? Then yeah, it has no business be used on non-black and white 2D drawings.
The actual Manga109 dataset includes full color covers (and possibly pages) not only black and white illustrations. However I'm not sure if the person who trained the model used the full dataset since afaict it requires some approval process to gain access to it, unlike the color covers-only dataset which the author of ESRGAN linked on the repo (which is far more limited in scope, being just 109 images of the covers of the manga from the full dataset).
Put briefly, it's possible the Manga109 model was trained exclusively on color manga covers.
It seems like it does OK with B/W manga until you look closely.
Here's a page I just did to test.
Before:
I wonder what it was trained on?
Here's a page I just did to test.
Before:
After:
I wonder what it was trained on?
This is fucking mind blowing. I never thought the applications of big data could be used like this. I'm interested in seeing how fast this runs in real time though
Well, that is pretty much what it is already. It tries to learn how to reconstruct the original version of an image from a smaller one that's lost a lot of the information. It's just that the same learned techniques can then be applied to generic input.
Man, I finally remembered the game I was thinking of that had 2D backgrounds that could really use a high res version. It's Grim Fandango. Got the remaster on PC, extracted some backgrounds and ran them through with the Manga model and man these results are insane. They can just stick these in with an update and you get a good presentation even on a 4K display.
Here are some comparison gifs:
Album with the full-res before and after pics:
https://imgur.com/a/uZJPXvq
Here are some comparison gifs:
Album with the full-res before and after pics:
https://imgur.com/a/uZJPXvq
Should I play the remastered version or wait for someone to mod the game with this INSANITY!
Incredible.
I was hoping for it to be easy to do myself but it seems like the backgrounds are in a weird format and I only found a tool to extract them to PNG, not modify them.
Among the better examples in this thread (apart from office blinds, office corridor bas relief detail in parts, and the kitchen collage shot which could use some extra refinements). The limited color palette patterned dither in the originals lends itself well to the texture of the Manga109 model upscaling. More than probably any other examples in this thread beside the Monkey Island 4 shot I'd say these are the most consistently solid upscales. I'd absolutely play the game modded like this.
Edit: the limited palette is in part caused by the GIF though looking at the gallery the dither pattern is also in the source. This thread would actually be a decent place to showcase SVG animated image comparisons using PNGs and JPEGs.
Last edited:
Very nice!
Too bad backgrounds are stored in a super obtuse way in FF9 for PC, not sure it'd be possible to replace them.
Here's some more from Chrono Cross. They certainly aren't perfect (just look at the sky) but still very impressive..
Despite some of the bugs this is one of the most impressive pictures in the thread. It's like the AI managed to guess that the intended objective of Chrono Cross was to look painterly, which barely comes across in the original image.
Man, I finally remembered the game I was thinking of that had 2D backgrounds that could really use a high res version. It's Grim Fandango. Got the remaster on PC, extracted some backgrounds and ran them through with the Manga model and man these results are insane. They can just stick these in with an update and you get a good presentation even on a 4K display.
Here are some comparison gifs:
Album with the full-res before and after pics:
https://imgur.com/a/uZJPXvq
Oh man this is amazing too.
What's even more amazing is that this tool is free.