A.I. Used to Upscale Scenes Shot in SD to 4K

With the emergence of artificial intelligence and machine learning in our industry, we’re seeing some very interesting applications for it, including editing, color grading, casting and even replacing actors in iconic scenes. But can anyone use A.I. for this purpose? Well, one sci-fan found out, and used machine learning to upscale one of his favorite TV shows to full HD.

I thought. With tools like AI Gigapixel, I knew it might be possible the low definition frames of DS9 can be scaled up to a higher definition such as 1080p or 4K. It would never be the same as proper remastering, but it would a step in the good direction. So I tried my hand at frame or two, to see what it could do… it did amazing upscaling real-life footage and the bigger budget CGI effects of DS9. – captrobau

The technique was tried by a blogger named CaptRobau who has been doing remastering the backgrounds, textures, and cut scenes of the video game Final Fantasy 7 using an app called Remako Mod.  The blogger was able to do it using still frames, and then graduated up to the short video clips.

Then, as a side project, he decided to give it a try with one of his favorite sci-fi shows … Star Trek: DS9. While the Original Trek Series, and the Next Generation was shot on film, by the mid 90s, most of the visual effects were relegated to digital video, and as such, the cost of upscaling them to HD or even 4K would be cost prohibitive. This meant that DS9 would likely never be remastered for fans to enjoy.

This was likely going to be too much for the Remake Mod to handle, so he graduated to AI GigaPixel, a $99 app from Topaz Labs. According to the Topaz website, using machine learning AI Gigapixel can enlarge and remaster still images, upscaling them by as much as 600% without losing any real image quality. But to do it with video, and to go from 48p to 4K might be quite a stretch for your basic home computer power.

But the experiment was about what it could do, not what it couldn’t, so he started with 1080p full HD. “It took me about two days to get everything extracted, upscaled and put it back together in a way that was pleasing, Robau writes. “This resulted only in the first five minutes of the episode being done (the episode recap, the opening scene, and the intro). The result left me pretty awestruck. It looked better than I had hoped. No weird issues or anything … pretty good for a mid-to-high end PC with software that isn’t just available to professionals.”

Robeau goes on to say that trying to push the remastering to 4K may be a phyrric victory, at best. He gave it a try with a very short clip, but in doing so, he said it “nearly melted my computer, as it is a lot more intense to upscale than 1080p. So, I’ll stick to this single video for 4K examples:”

Robeau also pointed out that this is likely the end of the experiment for him, since CBS/Paramount is very protective of the Star Trek copyright, and anything beyond a proof of concept may result in legal action. But it goes to show just what A.I. can do to future proof our cinematic heritage.

With Apple freezing out 32 bit videos in the next version of macOS, a technique like this could be used to give some of your old footage some legs.

About James DeRuvo 801 Articles
Editor in Chief at doddleNEWS. James has been a writer and editor at doddleNEWS for nearly a decade. As a producer/director/writer James won a Telly Award in 2005 for his Short Film "Searching for Inspiration. James is a recovering talk show producer from KABC in Los Angeles, and a weekly guest on the Digital Production Buzz with Larry Jordan.

Be the first to comment

Leave a Reply

Your email address will not be published.