1. Durante

    Durante
    Dark Souls Man Member OP

    I just learned about (and immediately tried) this, and it's pretty amazing. And a genius idea, even though the process seems rather obvious in retrospect (like many great ideas). I hope I didn't miss a thread about this!

    Some background information:
    • Due to various factors, input lag increases when emulating a system have been an issue for a while -- generally more for older platforms (ie. 16 bit and earlier, maybe 32) that had short processing loops/pipelines
    • Over the past few years, many improvements were made to popular emulators (or emulation platforms such as retroarch) to reduce or even eliminate that increase in input lag (ie. hard GPU sync, reducing buffering, late-latching inputs, etc)
    • At this point, you can get close to (or even, in some cases, match) the original latency for some older systems
    • This generally takes more performance since many options to reduce latency trade efficiency for minimizing delay -- this is usually not an issue on PC, but might be on other platforms

    A new innovation in Retroarch takes this (trading performance for decreased input latency) one step further: below real hardware.

    Basically, most games, even on older hardware, have some built-in amount of input delay. On many NES games it's 1 frame (16 ms), on a lot of SNES games it's 2 frames (33 ms) and so on.
    What the new options in Retroarch allow is this:
    1. Poll the input.
    2. Run one frame (frame 1) without output (video/audio).
    3. Create a save state.
    4. Emulate N-2 (configurable parameter) additional frames without output.
    5. Render a final frame (frame N) with output -- this is what you see.
    6. Restore the state saved in 3.
    This process repeats for every "original" frame.
    In effect, what this does from your perspective is move your input backwards in time, since you are reacting to output which happened N frames later. It's so neat.

    (Note: there is an alternative mode which runs another instance of the emulator core "normally" to generate audio, since the audio output of some emulation cores gets confused/choppy using the method above)

    Obviously, this has a high performance cost (i.e. it increases the CPU emulation overhead by a factor of N compared to normal, plus the additional save state overhead) but for fast cores and older systems it works perfectly.

    Here's the thread discussing the feature on the libretro forums, with a download link.

    Let me tag a few people who I remember from emulator latency discussions back in the old world Tain Dark1x
     
  2. Tain

    Tain
    Member

    This... sounds really neat. Maybe I'm misunderstanding, but isn't this kind of similar in concept to what GGPO does? As in, at worst you'd get "rollback" effects? I should check it out.

    I was kind of dismissive of Retroarch a few years back (I don't think I liked the results of running it without any form of v-sync, didn't see much reason not to use MAME/UME/MESS), but have warmed up to it a lot since.
     
  3. SG-17

    SG-17
    Community Resettler Member

    If the input lag was native to the consoles wouldn't the games be designed with it in mind? Seems like this could create unintended issues like removing the sprite line limit in NES games can cause weirdness.
     
  4. OnionPowder

    OnionPowder
    Member

    This is amazing. Something I would love to test out.

    Time to start dumping my collection and move away from CRT at some point in the future I guess.
     
  5. Durante

    Durante
    Dark Souls Man Member OP

    It absolutely is. However, you won't get any visible rollback effects as long as you don't run ahead for more frames than the original game's internal lag.

    You can easily try this for yourself, the current build allows running ahead up to 5 or 6 frames, which no game on those older platforms really approaches in terms of lag (as far as I know).

    Note that to effectively test this you need a fast core, e.g. for SNES snes9x is preferable to bsnes for this purpose (and really, outside of corner cases it's hard to distinguish the two these days).
     
  6. harry the spy

    harry the spy
    Member

    It cannot I think - its closer to you reacting from frames from the future.
     
  7. Durante

    Durante
    Dark Souls Man Member OP

    From the perspective of emulation accuracy, it is in some ways less "dangerous" than e.g. removing the sprite limit: on the "reference time stream" that actually determines what happens nothing changes. The game doesn't know that you are looking into the future ;)

    Of course, that's not to say that there can never be any problems with it, but as long as you keep below the internal delay people have had excellent experiences with it so far.
     
  8. Nitpicker_Red

    Nitpicker_Red
    Member

    The lag is between the physical input and when the button is detected. not between two things in the software. So it shouldn't make the software run differently, but it could mess a smidge with your muscle memory if you were used to pressing the button a frame in advance to compensate.
     
  9. alr1ght

    alr1ght
    Member

    Genius. Now to output these in 240p to a CRT. :)
     
  10. 1-D_FE

    1-D_FE
    Member

    Is there really no sound difference between them? Because I can't tell if I'm just suffering from bias and placebo effect, but I feel like bsnes has better audio than SNES.

    Amazing regardless. Retroarch already was only like .2 frames behind actual SNES hardware (when configured properly), but this means you could play SNES on a TV with 32ms lag and still have it play as responsive as native hardware did on CRT. That's awesome!
     
  11. Durante

    Durante
    Dark Souls Man Member OP

    Oh, one more thing I wanted to mention: a potential argument I see against this is that it's not "accurate" or "authentic". This is, of course, true. However:
    • You probably played those games on a CRT with basically 0 display hardware based input lag. Conversely, your current TV might have ~20 ms if it's cream-of-the-crop in terms of latency, or more otherwise.
      So e.g. compensating 2 frames of latency with this method will make the gameplay feel as responsive as it would be on a 0-latency CRT on a 33 ms latency LCD display.
    • Or if you have a 3 ms latency G-sync monitor, you can still use this method to eliminate other emulator sources of latency, likely with better compatibility or less side-effects than previous methods.
    • Or you might just not care and enjoy the extremely responsive feeling of less-than-original latency ;)
    Edit:
    Heh!
     
  12. Eila

    Eila
    Member

    Too bad I don't have a low latency screen to really enjoy something like that, but every bit helps, I guess.
    My laptop screen isn't so bad, much better than my TV, in any case.
     
  13. exodus

    exodus
    Member

    This is amazing.

    How is the frame skipping situation these days? I've never been able to successfully play in the past without either frame skipping or juddering of some kind. Does G-Sync help alleviate this? I recall BSNES had issues in the past even working properly with G-Sync due to how it rendered to the screen.

    My experience with emulators is quite out of date.
     
  14. Remij

    Remij
    Member

    I really want to like and use Retroarch, but I can never seem to get things set up properly. There's almost too many options and it's daunting to get it all set up, and the cores I used were all terribly laggy, leading me to assume that I had things set up wrong. I freaking love the idea of it though.

    Can anyone point to any decent tutorials out there?
     
  15. Toad King

    Toad King
    Member

    Already done my dude.

     
  16. Wowfunhappy

    Wowfunhappy
    Member

    This is one of the big advantages of Retroarch—it already has systems to eliminate this problem on non-variable refresh screens.
     
  17. Bomblord

    Bomblord
    Member

    I gotta say I'm not quite understanding what you are saying. It almost sounds like you are saying the game is actually reacting to what I input before I actually hit the button.
     
  18. Durante

    Durante
    Dark Souls Man Member OP

    As per above, this is just as (or maybe even more) useful if you have a display with some input lag, since it will get the experience closer to what was originally intended by (at least partially) compensating for that source of lag.

    Honestly, the whole presentation framework and the amount of options to customize it is on another level in Retroarch compared to most stand-alone emulators.
    (And for good reason, it pools the individual efforts of tons of people and makes them easily available to lots of emulator cores -- that was the idea)

    You can configure the output API used, the type of synchronization (in detail), how many frames to render ahead (in the traditional GPU sense, not this feature!), when input is polled, and so on.
    And now of course this additional magic ;)
     
  19. Ultima_5

    Ultima_5
    Member

    This. Ive tried once or twice to get it going, but it's such a mess of options and none of them mean anything to me.
     
  20. Knurek

    Knurek
    Member

    As long as you're okay with slightly slower audio. :)
     
  21. iswasdoes

    iswasdoes
    Member

    Could this technique be used to resolve lag in streaming services?
     
  22. Wowfunhappy

    Wowfunhappy
    Member

    Not really. It would be more effective (less resource intensive) for the games being streamed to eliminate any internal sources of input lag in their code.
     
  23. Skyfireblaze

    Skyfireblaze
    Member

    Okay this is the first time I read one of your posts and have a hard time grasping it Durante! :P Say I want to try it out now, do I just have to download the latest beta or nightly build?
     
  24. Durante

    Durante
    Dark Souls Man Member OP

    In a way it is like that.

    Of course, that's not how causality works, so instead, the emulator makes you "see into the future". You see what will happen, press the button, and then that button press influences the game at the point in time where it "really" is, which is one or several frames earlier than what you see.

    I can see that, it's the danger of any really powerful tool.
    But retroarch now is honestly far easier to use out of the box than it was a few years back. The UI is slick, controls are plug and play and auto-configured, you can download and install emulation cores straight out of the menu, and so on. There are still lots of options but you need only touch those that you are interested in, and you generally see the result immediately.
     
  25. jmga

    jmga
    Member

    Every open source emulator should implement libretro API at least as an option.
     
  26. Dark1x

    Dark1x
    Digital Foundry Verified User

    Wow, that's brilliant. It makes perfect sense but I wouldn't have thought of it, that's for certain.

    The next step here is coupling Retroarch with a proper 15khz CRT monitor using the correct resolution. That should reduce latency to its absolute lowest point.
     
  27. Nitpicker_Red

    Nitpicker_Red
    Member

    In a way it "does the work in advance" and only actually uses the work done if you pressed it.
     
  28. Yoshi

    Yoshi
    Member

    Talk about your TV, my 1998 tube has super low latency. Do the emulators only do speculative code execution or do actual rollbacks happen, i.e. does a frame appear where Mario dies and the next frame corrects this, because you actually jumped in time according to the low latency emulator?
     
  29. Zutrax

    Zutrax
    Member

    I'm also trying to wrap my head around this. So basically in laymans terms if I'm understanding this right, it saves a state of every single possible outcome based on all button presses/combinations, and once the button press is read it then rolls back instantly to the state that's associated with that press?
     
  30. Wowfunhappy

    Wowfunhappy
    Member

    I saw this method on reddit about a week ago, and it took me several days to wrap my head around what is actually happening. Here's how I'm currently thinking about it. If my understanding is flawed, people more knowledgable than I am can feel free to correct me:

    *******

    Under normal circumstances, you press A, and the game's code begins reacting to that input on the next frame. Super Mario World's code does not actually begin Mario's jump animation until one frame after it receives the input.

    With this method, you press A, and in the time it would normally take to generate a single frame, the emulator quickly runs through two frames, but only actually outputs the second frame to your screen.

    You could keep increasing the number of "skipped" frames infinitely as long as your computer is fast enough to generate them in the time it would normally take to generate a single frame. However, if you rendered 3+ frames ahead in Super Mario World, for instance, you would press A, and Mario would "teleport" to a higher position on the screen, because you would have skipped the beginning part of the animation.
     
  31. Durante

    Durante
    Dark Souls Man Member OP

    No, that would take far too much computing effort, and is also not necessary. Let me try to illustrate it.

    Code:
     Real    Emulated time (frames)       Run ahead frame 1          Run ahead frame 2         ...
     Time     (where the game is)                                      (what you see)
      ||
      ||    |                     |
      ||    |      ...            |
      ||    +---------------------+
      ||    |  "Actual" Frame     |    +---------------------+    +---------------------+
      ||    |  at time N          | -> |  "Preview" for N+1  | -> |  "Preview" for N+2  |
      ||    |                     |    +---------------------+    +---------------------+
      ||    +---------------------+
      ||    |  "Actual" Frame     |    +---------------------+    +---------------------+
      ||    |  at time N+1        | -> |  "Preview" for N+2  | -> |  "Preview" for N+3  |
      ||    |                     |    +---------------------+    +---------------------+
      ||    +---------------------+
      ||    |  "Actual" Frame     |    +---------------------+    +---------------------+
      ||    |  at time N+2        | -> |  "Preview" for N+3  | -> |  "Preview" for N+4  |
      ||    |                     |    +---------------------+    +---------------------+
      ||    +---------------------+
      ||    |  "Actual" Frame     |    +---------------------+    +---------------------+
      ||    |  at time N+3        | -> |  "Preview" for N+4  | -> |  "Preview" for N+5  |
      ||    |                     |    +---------------------+    +---------------------+
      ||    +---------------------+
      ||    |      ...            |
      ||    |                     |
      ||
      \/
    For every "real" frame, the emulator renders ahead a few frames and shows you the result of that, rather than the "real" current frame.
     
  32. Mzo

    Mzo
    Member

    I don't think so, that's crazy.

    It's more like you're running towards a pit and press jump when Mario is at the edge. The latency would take a frame or 2 to process and you would normally die from input lag. With this, you're seeing the future (Mario at the edge) while the game is actually in the past (a step or so away from the edge.) You press the button based on what you see now and it gets pressed at the earlier, actual time the game is running on so you jump and don't die.
     
  33. Durante

    Durante
    Dark Souls Man Member OP

    This will not happen as long as you are not running ahead more than the internal input lag of the game (as long as you are not, the game won't have reacted to your input yet anyway in the time frame we are previewing, so there will be no visible rollback).
     
  34. Jazzem

    Jazzem
    Member

    Oh man that's incredible =O

    I have a love/hate relationship with Retroarch and its unwieldy yet powerful interface, certainly going to have to tinker with it more and try this out.
     
  35. oni-link

    oni-link
    Member

    Maybe I'm way off here but does this mean I can update my Retro Pi and kiss goodbye to input latency?
     
  36. Tain

    Tain
    Member

    I don't think this is part of RetroArch officially, yet.

    edit: oh it's in the nightly builds, so
     
  37. Burai

    Burai
    Member

    Can't wait to try this. The 8bitdo controllers add a fair bit of latency over Bluetooth so it would nice to dial some of that out.
     
  38. Wowfunhappy

    Wowfunhappy
    Member

    I know this doesn't actually help usability, but my solution (of a sort) has been to just hide almost every config option from the UI. I don't need to touch those options anyway, and now my copy of Retroarch has the single cleanest interface of any emulator I've ever used. :D

    And then if I read about something cool like this, I can go and enable it in the config file.
     
  39. Aeana

    Aeana
    Member

    I'd love to try this out with Mother 3, as that is the one emulated game I have personally felt suffers massively from emulator input lag. If this can solve that issue, I'd be ecstatic.
     
  40. Aeana

    Aeana
    Member

    BSNES/Higan and snes9x literally have the same sound core and have for a couple of years.
     
  41. Bomblord

    Bomblord
    Member

    I think I understand now so the emulator renders other possibilities and then shows the one that is corresponding to the button press instead of starting the rendering at the button press?
     
  42. Astral/H3X

    Astral/H3X
    Member

    This is a very CPU intensive task; I'm not sure a rPI would be able to handle this.
     
  43. Mechanized

    Mechanized
    Member

    This is kinda insane.

    We're going back in time to input before the input is registered so there is less latency and this actually takes less time than the normal way wat.
     
  44. Astral/H3X

    Astral/H3X
    Member

    Running games in real hardware has a certain amount of lag between the inputs... Say one frame, or two frames.
    This essentially 'skips' those one or two frames, so that you would see your input on the correct frame, reducing that latency.
     
  45. How intensive are we talking? I5-2500k enough to use this for ps1games?
     
  46. oni-link

    oni-link
    Member

    I'm not sure what this second part means

    I managed to build a Pi but I don't really know much about this kind of thing, so if it's not possible that's cool, but it's my one issue with the Pi
     
  47. wondermagenta

    wondermagenta
    Member

    Anyone know if this is gonna work with the Wii U/PS3 versions of Retroarch using the latest Nightly build?
     
  48. Astral/H3X

    Astral/H3X
    Member

    If I'm reading this right, you'd essentially need to be able to run the game 3 times over at full speed to be able to use this.
     
  49. scitek

    scitek
    Member

    It sounds similar to how AVRs allow you to delay audio by a certain number of milliseconds in order to sync it up with video.
     
  50. Barrel Cannon

    Barrel Cannon
    Member

    I think I'll need a video to understand this better. I can't visualize what the OP means due to my technologial incompetence :(