Intro
Text to image and text to video tools inside of Runway are revolutionizing the way that we're going to make videos and the speed at which we can perform certain visual effects.
Runway, what is it?
For example, a few videos back, I demoed some of their AI magic tools, for example, removing backgrounds in painting to remove objects from your scene, text to color grade where you can type in a description of what that color should look like and it will change your video based on the text prompt. And there's a whole bunch more, but the next step forward is Gen 1.
What is Gen-1?
And you can see here in the corner, you need to request access. This is what the main Gen 1 website looks like. And you can also request access here, but essentially Gen 1 is what it says right here. Use words and images to generate new videos out of existing ones. For example, me talking right now, I could export a snippet and then type in a text prompt in the style of Vincent Van Gogh, and then we'll have a new style of the same video, but with Vincent Van Gogh applied. As we scroll down on the page, you can see here another example of the subway shot that's now in kind of this paper cutout animation. So in a way, it's almost like Gen 1 is an extension of our imagination. We can kind of think about what type of style that we want, and then using the existing images and videos that we have access to, to generate storyboards and new scenes from this tool. You can also upload a reference image to apply the color and style from this driving image to that existing video clip. So let's go ahead and start testing them out with a variety of clips that I already have here.
Test Parameters
But there are a few limitations in the research mode. One, I can only upload three second clips because that's the current limit at this stage. And two, I'm not allowed to share the UI of how to generate it, but I can share with you my reference images, the text prompts that I use, and the results.
Testing Shots: Stylization
All right, let's go ahead and get testing.
Here I have this martial arts dude. It's a stock video clip from Envato Elements. I'm going to download it as our first test. Here I have the clip in my timeline and this is the reference image that I uploaded to Gen 1 which Gen 1 then took and applied this style to that clip.
Isn't that awesome? And I love that it took his red hair from the reference image and applied it to the guy. And it even animated the background behind him. And I think my favorite part is that the AI knew that Vincent van Gogh cut off one of his ears 'cause he doesn't have an ear right here. So here it is side by side, the Vincent van Gogh driving image. On the left is the original and on the right is the exported result of the style. So I also tried it one more time, but instead of using the reference image, I put in a text prompt of in style of Vincent van Gogh. And this was the result this time. You can see it didn't add the red hair from the image, but it still put that famous starry night in the background. So you can play around between the text prompts and uploading images to get a variety of different results. It's never gonna be the same each time. And that's kind of the beauty of it. Also, once you get a result that you like, if you want to upscale it, there is a command inside of Gen 1 that will allow you to upscale to 4K, for example, because the first generated version is more low res. So there is that capability, but right now you can't export to a specific resolution. It just kind of spits out the best quality version that it can. Another style I tried out was claymation. And the first one didn't actually work out that great. You can see here, it actually looks like something from a horror film because it didn't quite define the end of the dog's mouth. So here I uploaded a reference of Wallace and Gromit, one of the most famous claymations out there, and it didn't do a very good job at translating this style in this particular scene. But instead I tried a text prompt instead of a image in style of Wallace and Gromit, and I got this result. So it's a little bit more interesting, but the faces are still kind of distorted. This scene might be a little bit more difficult, but playing around with more simple scenes can produce better results. For example, I downloaded this Burmese mountain dog. On the left, the original, on the right, the output. And the reference image was just Gromit himself without Wallace in the scene. And it did a pretty good job, but the teeth are still a little bit odd here. But I mean, this is amazing, right? Because you can pretty much take any video clip that you have and apply a new style to it.
Future of Stylization
So you can imagine filming live action scenes kind of like what they do in Avatar and they're going around and acting, and then just typing in a text command, make it look like Avatar, the results are going to be there instantaneously. Of course, this is still in beta. It's still being developed. It's not perfect, but could you imagine? Give it another 10 years where it's going to be. All right, so I put together a montage, but before I show you the montage, I want to show you an anime-style result. So I downloaded this magician card trick from Envato Elements.
And here's the reference image from an anime cartoon. And this is the result. I mean, this is just fantastic, right? So here's the side-by-side, and you can see it uses the same motion, but applies the style from the reference image. It's a little rough around the eyes, but as a first pass, I think it's phenomenal.
Ai Montage - Van Gogh Style
All right, so now to the montage. I took a bunch of action shots and applied the Vincent van Gogh style to create a montage called " What if a Rocky Film was Directed by Vincent van Gogh"? And I downloaded kind of a rocky soundtrack from Envato Elements.
I love the result of the swimmer because it took elements of the painting Starry Night and applied it to the water and I just think it looks beautiful. What I realized after I finished this is that you can start using Gen 1 to stylize really cool music videos. So you can take all of your shots and say you want it done in this painterly style or anime style and you don't have to do any animation from scratch. It's all done for you. So as you saw I was using Envato Elements clips to generate these new AI clips which is why I'm very excited that Envato is sponsoring today's video. You can actually sign up for a 7 day free trial using my link below to download as much as you want to test out. So while the Envato Elements stock video library is massive, one of my favorite parts about Elements is that you can go to video templates and pretty much find anything that you need. If you want a retro lower third, you can see over here these hand drawn lower thirds. There's also these retro 80s titles here. Or let's say you need film overlays, After Effects vintage film looks. There's dust and smoke overlays as well. The possibilities here are endless. They also have thousands and thousands of music tracks and any sound effects you can think of as well. For example, swells. Phone pickup. Everything you need to build the sound design and after all, half of your video is audio. But I think overall my favorite part about Elements is that the licensing is simple. You can use any of their assets in any type of project. You don't have to worry about signing up for a commercial plan to use it for client projects. There's one plan and it covers all. So I've dropped down my affiliate link below where you can sign up to get 7 days free. Alright let's get back into Gen 1.
Testing: Hybrid Scenes
So another use case is what I like to call hybrid mixing scenes. So for example, here, the spreading of jam on a piece of toast. Ian from Learn How to Edit Stuff uploaded this clip to Gen 1 and he typed in waves crashing into a sandy beach and it turned this clip into this. So it's mixing of two different scenes, in this case, peanut butter and jelly sandwich and a beach. And this got me thinking about different ways that you can use this approach for example, the song Cake by the Ocean. I was like, oh, this would be so cool to create a lyric video and actually have shots of cake by the ocean. So the first one I have here is this cake shot. And I typed in Cake by the Ocean and it spit out this. You can see that it added kind of coral stylization on top of the cake. Another example here is somebody taking a piece of cake out from this shot and it turned it into this.
And the text prompt I use for this one is Cake in front of an ocean with a sunset. And it did a pretty good job. Although I think it's funny that it made it look like the hand is wearing latex gloves because here there's no gloves. Quite interesting. Here's another one I uploaded of this giant waterfall and I typed in giant taking a bath and it spit out this. And you can see it looks super cool. The style, I didn't even tell it to produce a particular style. It just put the giant in the waterfall taking a bath. Obviously the anatomy can be improved. So this is one way that you can use it to create hybrid scenes where you would take something that's normally not there and place it into an existing scene. You've probably seen this done with Photoshop a bunch, but now it can be done using Gen 1.
Testing: Isolation/Masking
So another way that you can use Gen 1 is by isolating the foreground from the background. One way of doing that is using the green screen method. Here I uploaded the skateboarder and I basically typed in a command to make it have a green screen behind him. And this was the result. You can see it did a really good job at rotoscoping him out from the background. So that way you could take this and pull it into any program and replace the background. But the whole point of Gen 1 is to be able to use it to place something behind him. So what you can do instead of doing a green screen, you can isolate the foreground from the background. For this same clip, I typed in a command to isolate the foreground and place a rainbow magical landscape behind him. Here's the result. I mean, this is just beautiful. I love that it created that rainbow road kind of like in Mario Kart. Here's another example of a monkey. I typed in a prompt, " Add polka dots to the monkey".
It's kind of funny, right? But you can apply styles to moving objects. No animation, no rotoscoping is necessary. So here's another example where I asked Gen 1 to apply a Van Gogh style to my video clip. And the first time it applied it to the entire video clip. But the second time I asked it to just apply it to the background, and this was the result. It did a good job at identifying my borders, but it also did some strange things like it made my eye color really dark. And in the beginning, it made me miss some teeth. So it still does this kind of strange rendering on top of the foreground, even though I told it not to. And I think this is just because it's still in the research development stage. But overall, I'm really impressed with the results of the background. I downloaded a bunch of skateboarder clips and I asked Gen 1 to place a majestic rainbow landscape behind all of them.
Ai Montage -- Skateboard Magic Landscape
And I added some nice music and here's the result.
Beautiful masterpiece. And while there's many different ways of course you can play around with Gen 1, the last method I used while I was testing was applying specific prompts to abstract objects.
Testing: Abstract Renders
So for example I have this abstract 3D render here and I typed in a prompt to turn it into futuristic looking cars floating in a sky. To give it more context here if you go to the Gen 1 page and you scroll down you can see they have a 3D render that's untextured and Gen 1 was able to add an environment around it. So let me show you the results here. So they look like tiny little square cars just floating around. It didn't add much to the background because the background was just blue so it wasn't able to generate a sky or a cityscape behind it yet.
Final Thoughts
So this is really a wild and exciting time to be a part of AI video generation. I could see myself using this to add styles to particular clips or scenes in my video edits. Of course I can't just tell Gen 1 to edit together a series of skateboarder clips to a nice abstract music track. I still had to take all of those skateboard shots and line them up, edit it to the beat of the music with my own stylistic choices. That could be the future, but right now this is just a tool to help us find that imagination faster. Could you imagine a future where VFX artists are working hand-in-hand with directors just typing in commands via text saying, " Hey we need to change the scene to look like this". Like that is kind of the future. Of course there's going to be some more customization there, but imagine editing using codes and using commands. This is kind of what Gen 1 is. And I think James Cameron is kind of there where he's able to record the live action and mix it with animation. But that approach takes years and with Gen 1 you can apply an animation style to live action almost instantaneously. It still has a long way to go, but the future is here my friends. It's very exciting. So sign up for the waitlist and start testing it out for yourself. And if you want to try the other AI tools already on Runway, I put my affiliate link just down below and you can use my code " GAL" to get 10% off any plan. All right, thank you guys so much for watching and if you want more Gen 1 videos on this channel, be sure to give it a thumbs up. And as always, keep creating better video with Gal. See you next time. Bye!