fomm_bibi.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "fomm-bibi.ipynb",
      "private_outputs": true,
      "provenance": [],
      "machine_shape": "hm",
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/eyaler/avatars4all/blob/master/fomm_bibi.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cdO_RxQZLahB"
      },
      "source": [
        "# Demo for paper \"First Order Motion Model for Image Animation\"\n",
        "\n",
        "## **Talking head** (VoxCeleb) model\n",
        "\n",
        "### Original project: https://aliaksandrsiarohin.github.io/first-order-model-website\n",
        "\n",
        "#### Made just a little bit more accessible by Eyal Gruss ([@eyaler](https://twitter.com/eyaler) / [eyalgruss.com](https://eyalgruss.com) / [eyalgruss@gmail.com](mailto:eyalgruss@gmail.com))\n",
        "\n",
        "##### Short link here: https://j.mp/vid2head\n",
        "\n",
        "##### Click below for more refrences:"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Original notebook: https://colab.research.google.com/github/AliaksandrSiarohin/first-order-model/blob/master/demo.ipynb\n",
        "\n",
        "##### Faceswap notebook: https://colab.research.google.com/github/AliaksandrSiarohin/motion-cosegmentation/blob/master/part_swap.ipynb\n",
        "\n",
        "##### Notebook with video enhancement: https://colab.research.google.com/github/tg-bomze/Face-Image-Motion-Model/blob/master/Face_Image_Motion_Model_(Photo_2_Video)_Eng.ipynb\n",
        "\n",
        "##### Avatarify - a live vesrsion (requires local installation): https://github.com/alievk/avatarify\n",
        "\n",
        "##### Wav2Lip - a different model for lip syncing from audio: http://bhaasha.iiit.ac.in/lipsync\n",
        "\n",
        "\n",
        "#### **Stuff I made**:\n",
        "##### Avatars4all repository: https://github.com/eyaler/avatars4all\n",
        "##### Notebook for live webcam in the browser: https://colab.research.google.com/github/eyaler/avatars4all/blob/master/fomm_live.ipynb\n",
        "##### Notebook for talking head model: https://colab.research.google.com/github/eyaler/avatars4all/blob/master/fomm_bibi.ipynb\n",
        "##### Notebook for full body models (FOMM): https://colab.research.google.com/github/eyaler/avatars4all/blob/master/fomm_fufu.ipynb\n",
        "##### Notebook for full body models (impersonator): https://colab.research.google.com/github/eyaler/avatars4all/blob/master/ganozli.ipynb\n",
        "##### Notebook for full body models (impersonator++): https://colab.research.google.com/github/eyaler/avatars4all/blob/master/ganivut.ipynb\n",
        "##### Notebook for Wav2Lip audio based lip syncing: https://colab.research.google.com/github/eyaler/avatars4all/blob/master/melaflefon.ipynb\n",
        "##### List of more generative tools (outdated): https://j.mp/generativetools"
      ],
      "metadata": {
        "id": "gJ1cjc3CbfMQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Run me!"
      ],
      "metadata": {
        "id": "VmrY7hySV9ue"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ANINpF9JKuk8",
        "cellView": "form"
      },
      "source": [
        "#@title (1) Setup\n",
        "\n",
        "%cd /content\n",
        "!git clone --depth 1 https://github.com/eyaler/first-order-model\n",
        "!wget --no-check-certificate -nc https://openavatarify.s3.amazonaws.com/weights/vox-adv-cpk.pth.tar\n",
        "!wget --no-check-certificate -nc https://eyalgruss.com/fomm/vox-adv-cpk.pth.tar\n",
        "\n",
        "!mkdir -p /root/.cache/torch/hub/checkpoints\n",
        "%cd /root/.cache/torch/hub/checkpoints\n",
        "!wget --no-check-certificate -nc https://eyalgruss.com/fomm/s3fd-619a316812.pth\n",
        "!wget --no-check-certificate -nc https://eyalgruss.com/fomm/2DFAN4-11f355bf06.pth.tar\n",
        "%cd /content\n",
        "\n",
        "!pip install -U git+https://github.com/ytdl-org/youtube-dl\n",
        "!pip install git+https://github.com/1adrianb/face-alignment@v1.0.1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2iGy_8v-4BrO",
        "cellView": "form"
      },
      "source": [
        "#@title (2) Get the Driver video and Avatar image from the web\n",
        "#@markdown 1. You can change the URLs to your **own** stuff from most video platforms!\n",
        "#@markdown 2. Alternatively, you can upload **local** files in the next cells (2a, 2b)\n",
        "\n",
        "video_url = 'https://www.youtube.com/watch?v=OziXYniB5x4' #@param {type:\"string\"}\n",
        "limit_video_height = \"1080\" #@param [360, 480, 720, 1080, 1440, 2160, 4320] {allow-input: true}\n",
        "image_url = 'https://www.srugim.co.il/wp-content/uploads/2010/10/%D7%90%D7%97%D7%9E%D7%93-%D7%98%D7%99%D7%91%D7%99.jpg' #@param {type:\"string\"}\n",
        "\n",
        "if video_url:\n",
        "  !rm -f /content/video.mp4\n",
        "  !youtube-dl --no-check-extensions --no-playlist -f \"bestvideo[ext=mp4][vcodec!*=av01][height<=$limit_video_height]+bestaudio[ext=m4a]/mp4[height<=$limit_video_height][vcodec!*=av01]/mp4[vcodec!*=av01]/mp4\" \"$video_url\" --merge-output-format mp4 -o /content/video\n",
        "  !mv /content/video.mp4 /content/video\n",
        "\n",
        "if image_url:\n",
        "  !wget \"$image_url\" -O /content/image"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8-pflQXqXq7_",
        "cellView": "form"
      },
      "source": [
        "#@title (2a) Optionally upload local Driver video { run: \"auto\" }\n",
        "#@markdown Instructions: mark the checkbox + press play if it doesn't start by itself + click the upload button that will appear below\n",
        "manually_upload_video = False #@param {type:\"boolean\"}\n",
        "if manually_upload_video:\n",
        "  from google.colab import files\n",
        "  import shutil\n",
        "\n",
        "  %cd /content/sample_data\n",
        "  try:\n",
        "    uploaded = files.upload()\n",
        "  except Exception as e:\n",
        "    %cd /content\n",
        "    raise e\n",
        "\n",
        "  for fn in uploaded:\n",
        "    shutil.move('/content/sample_data/'+fn, '/content/video')\n",
        "    break\n",
        "  %cd /content"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "31DE8-cwX0eI",
        "cellView": "form"
      },
      "source": [
        "#@title (2b) Optionally upload local Avatar image { run: \"auto\" }\n",
        "#@markdown Instructions: mark the checkbox + press play if it doesn't start by itself + click the upload button that will appear below\n",
        "manually_upload_image = False #@param {type:\"boolean\"}\n",
        "if manually_upload_image:\n",
        "  from google.colab import files\n",
        "  import shutil\n",
        "\n",
        "  %cd /content/sample_data\n",
        "  try:\n",
        "    uploaded = files.upload()\n",
        "  except Exception as e:\n",
        "    %cd /content\n",
        "    raise e\n",
        "\n",
        "  for fn in uploaded:\n",
        "    shutil.move('/content/sample_data/'+fn, '/content/image')\n",
        "    break\n",
        "  %cd /content"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bl8iCSyFKXBy",
        "cellView": "form"
      },
      "source": [
        "#@title (3) Optionally (but recommended) shorten Driver video\n",
        "start_seconds = 0 #@param {type:\"number\"}\n",
        "duration_seconds =  60#@param {type:\"number\"}\n",
        "start_seconds = max(start_seconds,0)\n",
        "duration_seconds = max(duration_seconds,0)\n",
        "#@markdown (use duration_seconds = 0 for unrestricted duration)\n",
        "\n",
        "if start_seconds or duration_seconds:\n",
        "  !mv /content/video /content/full_video\n",
        "  !ffmpeg -ss $start_seconds -t $duration_seconds -i /content/full_video -f mp4 /content/video -y\n",
        "else:\n",
        "  print('Using full video.')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Oxi6-riLOgnm",
        "cellView": "form"
      },
      "source": [
        "#@title (4) Prepare assets\n",
        "#@markdown If you ran out of RAM this means that the video is too large. You can shorten it above (3), or try to download with a smaller height limit in (2).\n",
        "\n",
        "center_video_to_head = True #@param {type:\"boolean\"}\n",
        "crop_video_to_head = True #@param {type:\"boolean\"}\n",
        "video_crop_expansion_factor = 2.5 #@param {type:\"number\"}\n",
        "center_image_to_head = True #@param {type:\"boolean\"}\n",
        "crop_image_to_head = False #@param {type:\"boolean\"}\n",
        "image_crop_expansion_factor = 2.5 #@param {type:\"number\"}\n",
        "video_crop_expansion_factor = max(video_crop_expansion_factor, 1)\n",
        "image_crop_expansion_factor = max(image_crop_expansion_factor, 1)\n",
        "\n",
        "import imageio\n",
        "import numpy as np\n",
        "from skimage import img_as_ubyte\n",
        "from skimage.transform import resize\n",
        "from IPython.display import Video, clear_output\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "\n",
        "import face_alignment\n",
        "fa = face_alignment.FaceAlignment(face_alignment.LandmarksType._2D, flip_input=True,\n",
        "                                      device='cuda')\n",
        "\n",
        "def create_bounding_box(target_landmarks, expansion_factor=1):\n",
        "    target_landmarks = np.array(target_landmarks)\n",
        "    x_y_min = target_landmarks.reshape(-1, 68, 2).min(axis=1)\n",
        "    x_y_max = target_landmarks.reshape(-1, 68, 2).max(axis=1)\n",
        "    expansion_factor = (expansion_factor-1)/2\n",
        "    bb_expansion_x = (x_y_max[:, 0] - x_y_min[:, 0]) * expansion_factor\n",
        "    bb_expansion_y = (x_y_max[:, 1] - x_y_min[:, 1]) * expansion_factor\n",
        "    x_y_min[:, 0] -= bb_expansion_x\n",
        "    x_y_max[:, 0] += bb_expansion_x\n",
        "    x_y_min[:, 1] -= bb_expansion_y\n",
        "    x_y_max[:, 1] += bb_expansion_y\n",
        "    return np.hstack((x_y_min, x_y_max-x_y_min))\n",
        "\n",
        "def fix_dims(im):\n",
        "    if im.ndim == 2:\n",
        "        im = np.tile(im[..., None], [1, 1, 3])\n",
        "    return im[...,:3]\n",
        "\n",
        "def get_crop(im, center_face=True, crop_face=True, expansion_factor=1, landmarks=None):\n",
        "    im = fix_dims(im)\n",
        "    if (center_face or crop_face) and not landmarks:\n",
        "        landmarks = fa.get_landmarks_from_image(im)\n",
        "    if (center_face or crop_face) and landmarks:\n",
        "        rects = create_bounding_box(landmarks, expansion_factor=expansion_factor)\n",
        "        x0,y0,w,h = sorted(rects, key=lambda x: x[2]*x[3])[-1]\n",
        "        if crop_face:\n",
        "            s = max(h, w)\n",
        "            x0 += (w-s)//2\n",
        "            x1 = x0 + s\n",
        "            y0 += (h-s)//2\n",
        "            y1 = y0 + s\n",
        "        else:\n",
        "            img_h,img_w = im.shape[:2]\n",
        "            img_s = min(img_h,img_w)\n",
        "            x0 = min(max(0, x0+(w-img_s)//2), img_w-img_s)\n",
        "            x1 = x0 + img_s\n",
        "            y0 = min(max(0, y0+(h-img_s)//2), img_h-img_s)\n",
        "            y1 = y0 + img_s\n",
        "    else:\n",
        "        h,w = im.shape[:2]\n",
        "        s = min(h,w)\n",
        "        x0 = (w-s)//2\n",
        "        x1 = x0 + s\n",
        "        y0 = (h-s)//2\n",
        "        y1 = y0 + s\n",
        "    return int(x0),int(x1),int(y0),int(y1)\n",
        "\n",
        "def pad_crop_resize(im, x0=None, x1=None, y0=None, y1=None, new_h=256, new_w=256):\n",
        "    im = fix_dims(im)\n",
        "    h,w = im.shape[:2]\n",
        "    if x0 is None:\n",
        "      x0 = 0\n",
        "    if x1 is None:\n",
        "      x1 = w\n",
        "    if y0 is None:\n",
        "      y0 = 0\n",
        "    if y1 is None:\n",
        "      y1 = h\n",
        "    if x0<0 or x1>w or y0<0 or y1>h:\n",
        "        im = np.pad(im, pad_width=[(max(-y0,0),max(y1-h,0)),(max(-x0,0),max(x1-w,0)),(0,0)], mode='edge')\n",
        "    return resize(im[max(y0,0):y1-min(y0,0),max(x0,0):x1-min(x0,0)], (new_h, new_w))\n",
        "\n",
        "source_image = imageio.imread('/content/image')\n",
        "source_image = pad_crop_resize(source_image, *get_crop(source_image, center_face=center_image_to_head, crop_face=crop_image_to_head, expansion_factor=image_crop_expansion_factor))\n",
        "\n",
        "with imageio.get_reader('/content/video', format='mp4') as reader:\n",
        "  fps = reader.get_meta_data()['fps']\n",
        "\n",
        "  driving_video = []\n",
        "  landmarks = None\n",
        "  try:\n",
        "      for i,im in enumerate(reader):\n",
        "          if not crop_video_to_head:\n",
        "              break\n",
        "          landmarks = fa.get_landmarks_from_image(im)\n",
        "          if landmarks:\n",
        "              break\n",
        "      x0,x1,y0,y1 = get_crop(im, center_face=center_video_to_head, crop_face=crop_video_to_head, expansion_factor=video_crop_expansion_factor, landmarks=landmarks)\n",
        "      reader.set_image_index(0)\n",
        "      for im in reader:\n",
        "          driving_video.append(pad_crop_resize(im,x0,x1,y0,y1))\n",
        "  except RuntimeError:\n",
        "      pass\n",
        "\n",
        "def vid_display(path, source, driving, generated=None):\n",
        "    imageio.mimwrite(path, [img_as_ubyte(np.hstack([source, driving[i]] + ([generated[i]] if generated else []))) for i in range(len(driving))], fps=fps)\n",
        "    display(Video(path, embed=True, html_attributes='autoplay controls loop'))\n",
        "    print('If video is not playing you can find it here:', path)\n",
        "\n",
        "clear_output()\n",
        "if landmarks:\n",
        "  print('first found head in frame %d'%i)\n",
        "vid_display('/content/input_concat.mp4', source_image, driving_video)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YFClprBP9WUn",
        "cellView": "form"
      },
      "source": [
        "#@title (5) Find best alignment\n",
        "\n",
        "%cd /content/first-order-model\n",
        "from demo import load_checkpoints\n",
        "generator, kp_detector = load_checkpoints(config_path='/content/first-order-model/config/vox-adv-256.yaml',\n",
        "                            checkpoint_path='/content/vox-adv-cpk.pth.tar')\n",
        "\n",
        "from scipy.spatial import ConvexHull\n",
        "def normalize_kp(kps):\n",
        "    max_area = 0\n",
        "    max_kp = None\n",
        "    for kp in kps:\n",
        "        kp = kp - kp.mean(axis=0, keepdims=True)\n",
        "        area = ConvexHull(kp[:, :2]).volume\n",
        "        area = np.sqrt(area)\n",
        "        kp[:, :2] = kp[:, :2] / area\n",
        "        if area>max_area:\n",
        "          max_area = area\n",
        "          max_kp = kp\n",
        "    return max_kp\n",
        "\n",
        "from tqdm import tqdm\n",
        "\n",
        "kp_source = fa.get_landmarks_from_image(255 * source_image)\n",
        "if kp_source:\n",
        "  norm_kp_source = normalize_kp(kp_source)\n",
        "\n",
        "norm  = float('inf')\n",
        "best = 0\n",
        "best_kp_driving = None\n",
        "for i, image in tqdm(enumerate(driving_video)):\n",
        "  kp_driving = fa.get_landmarks_from_image(255 * image)\n",
        "  if kp_driving:\n",
        "    norm_kp_driving = normalize_kp(kp_driving)\n",
        "    if kp_source:\n",
        "      new_norm = (np.abs(norm_kp_source - norm_kp_driving) ** 2).sum()\n",
        "      if new_norm < norm:\n",
        "        norm = new_norm\n",
        "        best = i\n",
        "        best_kp_driving = kp_driving\n",
        "    else:\n",
        "      best_kp_driving = kp_driving\n",
        "      break\n",
        "\n",
        "from logger import Visualizer\n",
        "vis = Visualizer(kp_size=3, colormap='gist_rainbow')\n",
        "source_with_kp = vis.draw_image_with_kp(source_image, kp_source[0]*2/np.array(source_image.shape[:2][::-1])[np.newaxis]-1) if kp_source else source_image\n",
        "driving_with_kp = vis.draw_image_with_kp(driving_video[best], best_kp_driving[0]*2/np.array(driving_video[best].shape[:2][::-1])[np.newaxis]-1) if best_kp_driving else driving_video[best]\n",
        "\n",
        "clear_output()\n",
        "print('\\nbest frame=%d'%best)\n",
        "from google.colab.patches import cv2_imshow\n",
        "cv2_imshow(np.hstack([source_with_kp, driving_with_kp])[...,::-1]*255)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SB12II11kF4c",
        "cellView": "form"
      },
      "source": [
        "#@title (6) Animate\n",
        "\n",
        "exaggerate_factor = 1 #@param {type:\"slider\", min:0.1, max:5, step:0.1}\n",
        "adapt_movement_scale = True #@param {type:\"boolean\"}\n",
        "use_relative_movement = True #@param {type:\"boolean\"}\n",
        "use_relative_jacobian = True #@param {type:\"boolean\"}\n",
        "\n",
        "import torch\n",
        "\n",
        "def full_normalize_kp(kp_source, kp_driving, kp_driving_initial, adapt_movement_scale=False,\n",
        "                 use_relative_movement=False, use_relative_jacobian=False, exaggerate_factor=1):\n",
        "    if adapt_movement_scale:\n",
        "        source_area = ConvexHull(kp_source['value'][0].data.cpu().numpy()).volume\n",
        "        driving_area = ConvexHull(kp_driving_initial['value'][0].data.cpu().numpy()).volume\n",
        "        adapt_movement_scale = np.sqrt(source_area) / np.sqrt(driving_area)\n",
        "    else:\n",
        "        adapt_movement_scale = 1\n",
        "\n",
        "    kp_new = {k: v for k, v in kp_driving.items()}\n",
        "\n",
        "    if use_relative_movement:\n",
        "        kp_value_diff = (kp_driving['value'] - kp_driving_initial['value'])\n",
        "        kp_value_diff *= adapt_movement_scale * exaggerate_factor\n",
        "        kp_new['value'] = kp_value_diff + kp_source['value']\n",
        "\n",
        "        if use_relative_jacobian:\n",
        "            jacobian_diff = torch.matmul(kp_driving['jacobian'], torch.inverse(kp_driving_initial['jacobian']))\n",
        "            kp_new['jacobian'] = torch.matmul(jacobian_diff, kp_source['jacobian'])\n",
        "\n",
        "    return kp_new\n",
        "\n",
        "def make_animation(source_image, driving_video, generator, kp_detector, adapt_movement_scale=False,\n",
        "                 use_relative_movement=False, use_relative_jacobian=False, cpu=False, exaggerate_factor=1):\n",
        "    with torch.no_grad():\n",
        "        predictions = []\n",
        "        source = torch.tensor(source_image[np.newaxis].astype(np.float32)).permute(0, 3, 1, 2)\n",
        "        if not cpu:\n",
        "            source = source.cuda()\n",
        "        driving = torch.tensor(np.array(driving_video)[np.newaxis].astype(np.float32)).permute(0, 4, 1, 2, 3)\n",
        "        kp_source = kp_detector(source)\n",
        "        kp_driving_initial = kp_detector(driving[:, :, 0])\n",
        "\n",
        "        for frame_idx in tqdm(range(driving.shape[2])):\n",
        "            driving_frame = driving[:, :, frame_idx]\n",
        "            if not cpu:\n",
        "                driving_frame = driving_frame.cuda()\n",
        "            kp_driving = kp_detector(driving_frame)\n",
        "            kp_norm = full_normalize_kp(kp_source=kp_source, kp_driving=kp_driving,\n",
        "                                   kp_driving_initial=kp_driving_initial, adapt_movement_scale=adapt_movement_scale, use_relative_movement=use_relative_movement,\n",
        "                                   use_relative_jacobian=use_relative_jacobian, exaggerate_factor=exaggerate_factor)\n",
        "            out = generator(source, kp_source=kp_source, kp_driving=kp_norm)\n",
        "\n",
        "            predictions.append(np.transpose(out['prediction'].data.cpu().numpy(), [0, 2, 3, 1])[0])\n",
        "    return predictions\n",
        "\n",
        "predictions_forward = make_animation(source_image, driving_video[best:], generator, kp_detector, adapt_movement_scale=adapt_movement_scale, use_relative_movement=use_relative_movement,\n",
        "                                   use_relative_jacobian=use_relative_jacobian, exaggerate_factor=exaggerate_factor)\n",
        "predictions_backward = make_animation(source_image, driving_video[:(best+1)][::-1], generator, kp_detector, adapt_movement_scale=adapt_movement_scale, use_relative_movement=use_relative_movement,\n",
        "                                   use_relative_jacobian=use_relative_jacobian, exaggerate_factor=exaggerate_factor)\n",
        "\n",
        "imageio.mimwrite('/content/generated.mp4', [img_as_ubyte(frame) for frame in predictions_backward[::-1] + predictions_forward[1:]], fps=fps)\n",
        "!ffmpeg -i /content/generated.mp4 -i /content/video -c:v libx264 -c:a aac -map 0:v -map 1:a? -pix_fmt yuv420p /content/final.mp4 -profile:v baseline -movflags +faststart -y\n",
        "#video can be downloaded from /content/final.mp4\n",
        "\n",
        "clear_output()\n",
        "vid_display('/content/output_concat.mp4', source_image, driving_video, predictions_backward[::-1] + predictions_forward[1:])\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lH4oJcBOKZcf",
        "cellView": "form"
      },
      "source": [
        "#@title (7) Download\n",
        "#@markdown 1. If it fails try running this cell again.\n",
        "#@markdown 2. Alternatively, you can manually download \"final.mp4\" from the folder on the left (click \"Refresh\" if missing).\n",
        "\n",
        "print() #see https://github.com/googlecolab/colabtools/issues/468\n",
        "from google.colab import files\n",
        "files.download('/content/final.mp4') #fails for Firefox private window"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0VLWbxDiq_jd",
        "cellView": "form"
      },
      "source": [
        "#@title (8) Optionally apply Wav2Lip post processing and download\n",
        "wav2lip_post_processing = True #@param {type: \"boolean\"}\n",
        "smooth_face_detection = True #@param {type: \"boolean\"}\n",
        "if wav2lip_post_processing:\n",
        "  !pip install librosa==0.9.2\n",
        "  !pip install -U gdown\n",
        "  !rm -rf /content/final_wav2lip.mp4\n",
        "  %cd /content\n",
        "  !git clone --depth 1 https://github.com/eyaler/Wav2Lip.git\n",
        "  import os\n",
        "  if not os.path.exists('/content/Wav2Lip/checkpoints/wav2lip_gan.pth'):\n",
        "    !gdown https://drive.google.com/uc?id=1dwHujX7RVNCvdR1RR93z0FS2T2yzqup9 -O /content/Wav2Lip/checkpoints/wav2lip_gan.pth\n",
        "  !wget --no-check-certificate -nc https://eyalgruss.com/fomm/wav2lip_gan.pth -O /content/Wav2Lip/checkpoints/wav2lip_gan.pth\n",
        "  #!wget --no-check-certificate -nc https://eyalgruss.com/fomm/wav2lip.pth -O /content/Wav2Lip/checkpoints/wav2lip.pth\n",
        "  !cp /root/.cache/torch/hub/checkpoints/s3fd-619a316812.pth /content/Wav2Lip/face_detection/detection/sfd/s3fd.pth\n",
        "  %cd /content/Wav2Lip\n",
        "  !rm -rf /content/Wav2Lip/temp/*\n",
        "  nosmooth = '' if smooth_face_detection else '--nosmooth'\n",
        "  !python inference.py --checkpoint_path checkpoints/wav2lip_gan.pth --face /content/final.mp4 --audio /content/final.mp4 --pads 0 20 0 0 $nosmooth --outfile /content/final_wav2lip.mp4\n",
        "  if os.path.exists('/content/Wav2Lip/temp/faulty_frame.jpg'):\n",
        "    import cv2\n",
        "    print('\\nFace not detected - will use whole frame')\n",
        "    video_stream = cv2.VideoCapture(infile)\n",
        "    still_reading, frame = video_stream.read()\n",
        "    x1 = y1 = 0\n",
        "    y2,x2 = frame.shape[:2]\n",
        "    if x2>h:\n",
        "      x1 = (x2-h)//2\n",
        "      x2 = x1+y2\n",
        "    !python inference.py --checkpoint_path checkpoints/wav2lip_gan.pth /content/final.mp4 --audio /content/final.mp4 --box $y1 $y2 $x1 $x2 --pads 0 20 0 0 --outfile /content/final_wav2lip.mp4\n",
        "  from google.colab import files\n",
        "  files.download('/content/final_wav2lip.mp4')"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}