| title |
|---|
ML x Video |
Here we will show how [[Machine Learning]] (ML) can be applied on the moving image (video, film). It's about generating and manipulating images or whole video sequences. There are possible use cases in the field of media art, cinema, storytelling, video installations, music videos, GIF memes, and more!
The following capabilities will most probably be integrated in popular image and video editing tools very soon.. Will we still call them "AI" then? 🤔
Machine Learning can infer depth information from a 2D image. With this information you can add various 3D effects to images, like this parallax effect made by Cyril Diagne
🤔 Some of these examples seem "magic" because they create more out of less: adding color information, increasing resolution, adding depth... But keep in mind that Machine Learning is based on statistics, the algorithms have "learned" from large amount of example footage what the probabilities for a specific depth/color/shape are. That makes them susceptible for [[bias]]. Keep that in mind when you work with these apparently objective tools!
A deepfake is when you "puppeteer" a target video (most often of a human being) with a source video. Here are a few examples:
<iframe width="600" height="383" src="https://www.youtube.com/embed/dCTM2lvm0QE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>Sassy Justice is a Youtube series made by the makers of South Park. Watch the full series here!
Here's a real-time example, "First order Kiss" by Shardcore. He writes: "I tried lip-sync’ing to Prince via Avatarify and recording the results. Frame-rate is a bit crappy, but not bad for ‘real-time-processing’ The future of karaoke, perhaps?"
<iframe src="https://player.vimeo.com/video/420596414?h=0f08061b90" width="300" height="300" frameborder="0" allow="autoplay; fullscreen; picture-in-picture" allowfullscreen></iframe>Here's a piece of video art by British artist Libby Heaney in which she plays with gender roles by becoming Elvis herself..
<iframe width="600" height="383" src="https://www.youtube.com/embed/CXMgfZEP2cA?list=TLGGpPagydLuJDQxMDAxMjAyMg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>Horse2Zebra (made with CycleGAN)
🤔 As you can see these new kind of "filters" are doing more than just manipulating the "style" of an image, they seem to somehow manipulate the "semantics" of an image as well!
The [[latent space]] is a high dimensional mathematical space of possible representations (depending on your training data). By "walking" the latent space we are interpolating between the points in the latent space and thus create animations of organic change.
<iframe border=0 frameborder=0 height=650 width=550 display="inline" src="https://twitframe.com/show?url=https://twitter.com/RypeArts/status/1364054338943873025 "> </iframe> <iframe src="https://player.vimeo.com/video/520039359?h=06911ac0de" width="640" height="480" frameborder="0" allow="autoplay; fullscreen; picture-in-picture" allowfullscreen></iframe>Latent Riot Space Walk from Moving Target Collective on Vimeo.
ML can also be used to assist you in the editing process by analysing your footage with image recognition and other [[computer vision]] techniques. You can analyze, organize and filter your footage by the objects in the shot, angle, lightning, scenery, etc.
<iframe width="600" height="383" src="https://www.youtube.com/embed/fsQhOCkczHQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> <iframe border=0 frameborder=0 height=650 width=550 display="inline" src="https://twitframe.com/show?url=https://twitter.com/RiversHaveWings/status/1447708268789338119 "> </iframe> <iframe border=0 frameborder=0 height=650 width=550 display="inline" src="https://twitframe.com/show?url=https://twitter.com/Buntworthy/status/1440244134724308995 "> </iframe> <iframe border=0 frameborder=0 height=750 width=550 display="inline" src="https://twitframe.com/show?url=https://twitter.com/unltd_dream_co/status/1469085102500163584">
</iframe> <iframe border=0 frameborder=0 height=450 width=550 display="inline" src="https://twitframe.com/show?url=https://twitter.com/chigozienri/status/1417957701087281162 "> </iframe> <iframe width="600" height="383" src="https://www.youtube.com/embed/5dfCmSY8Ws0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> <iframe border=0 frameborder=0 height=550 width=550 display="inline" src="https://twitframe.com/show?url=https://twitter.com/xsteenbrugge/status/1466519092916019204 "> </iframe> <iframe border=0 frameborder=0 height=750 width=550 display="inline" src="https://twitframe.com/show?url=https://twitter.com/ak92501/status/1463691106416214019 "> </iframe>
Image from this 2018 article: https://www.technologyreview.com/2018/12/03/138834/ai-software-can-dream-up-an-entire-digital-world-from-a-simple-sketch/
- RunwayML
- Avatarify
- Kaspar.ai
Check out Alexa's are.na Board here where she regularly dumps interesting things she finds on the internet