How to overlay two videos with blend filter in ffmpeg

Question

I need to do a lot of videos with the next specifications:

• A background video (bg.mp4)
• Overlay a sequence of png images img1.png to img300.png (img%d.png) with a rate of 30 fps
• Overlay a video with dust effects using a blend-lighten filter (dust.mp4)
• Scale all the inputs to 1600x900 and if the input have not the aspect ratio, then crop them.
• Specify the duration of the output-video to 10 sec (is the duration of image sequence at 30fps).

I've being doing a lot of test with different commands but always shows error.

Answer 1

Well, I think I got it in the next command:

ffmpeg -ss 00:00:18.300 -i music.mp3 -loop 1 -i bg.mp4 -i ac%d.png -i dust.mp4 -filter_complex "[1:0]scale=1600:ih*1200/iw, crop=1600:900[a];[a][2:0] overlay=0:0[b]; [3:0] scale=1600:ih*1600/iw, crop=1600:900,setsar=1[c]; [b][c] blend=all_mode='overlay':all_opacity=0.2" -shortest -y output.mp4

I'm going to explain in order to share what I've found:

• Declaring the inputs:

ffmpeg -ss 00:00:18.300 -i music.mp3 -loop 1 -i bg.mp4 -i ac%d.png -i dust.mp4

• Adding the filter complex. First part: [1,0] is the second element of the inputs (bg.mp4) and scaling to get the max values, and then cropping with the size I need, the result of this opperation, is in the [a] element.

[1:0]scale=1600:ih*1600/iw, crop=1600:900, setsar=1[a];

• Second Part: Put the PNGs sequence over the resized video (bg.mp4, now [a]) and saving the resunt in the [b] element.

[a][2:0] overlay=0:0[b];

• Scaling and cropping the fourth input (overlay.mp4) and saving in the [c] element.

[3:0]scale=1600:ih*1600/iw, crop=1600:900,setsar=1[c];

• Mixing the first result with the overlay video with an "overlay" blending mode, and with an opacity of 0.1 because the video has gray tones and makes the result so dark.

[b][c] blend=all_mode='overlay':all_opacity=0.1

That's all. If anyone can explay how this scaling filter works, I would thank a lot!

Answer 2

I needed to process a stack of images and was unable to get ffmpeg to work for me reliably, so I built a Python tool to help mediate the process:

#!/usr/bin/env python3
import functools
import numpy as np
import os
from PIL import Image, ImageChops, ImageFont, ImageDraw
import re
import sys
import multiprocessing
import time

def get_trim_box(image_name):
  im = Image.open(image_name)
  bg = Image.new(im.mode, im.size, im.getpixel((0,0)))
  diff = ImageChops.difference(im, bg)
  diff = ImageChops.add(diff, diff, 2.0, -100)
  #The bounding box is returned as a 4-tuple defining the left, upper, right, and lower pixel coordinate. If the image is completely empty, this method returns None.
  return diff.getbbox()



def rect_union(rect1, rect2):
  left1, upper1, right1, lower1 = rect1
  left2, upper2, right2, lower2 = rect2
  return (
    min(left1,left2),
    min(upper1,upper2),
    max(right1,right2),
    max(lower1,lower2)
  )



def blend_images(img1, img2, steps):
  return [Image.blend(img1, img2, alpha) for alpha in np.linspace(0,1,steps)]



def make_blend_group(options):
  print("Working on {0}+{1}".format(options["img1"], options["img2"]))
  font = ImageFont.truetype(options["font"], size=options["fontsize"])
  img1 = Image.open(options["img1"], mode='r').convert('RGB')
  img2 = Image.open(options["img2"], mode='r').convert('RGB')
  img1.crop(options["trimbox"])
  img2.crop(options["trimbox"])
  blends = blend_images(img1, img2, options["blend_steps"])
  for i,img in enumerate(blends):
    draw = ImageDraw.Draw(img)
    draw.text(options["textloc"], options["text"], fill=options["fill"], font=font)
    img.save(os.path.join(options["out_dir"],"out_{0:04}_{1:04}.png".format(options["blendnum"],i)))



if len(sys.argv)<3:
  print("Syntax: {0} <Output Directory> <Images...>".format(sys.argv[0]))
  sys.exit(-1)

out_dir     = sys.argv[1]
image_names = sys.argv[2:]

pool = multiprocessing.Pool()

image_names = sorted(image_names)
image_names.append(image_names[0]) #So we can loop the animation

#Assumes image names are alphabetic with a UNIX timestamp mixed in.
image_times = [re.sub('[^0-9]','', x) for x in image_names]
image_times = [time.strftime('%Y-%m-%d (%a) %H:%M', time.localtime(int(x))) for x in image_times]

#Crop off the edges, assuming upper left pixel is representative of background color
print("Finding trim boxes...")
trimboxes = pool.map(get_trim_box, image_names)
trimboxes = [x for x in trimboxes if x is not None]
trimbox   = functools.reduce(rect_union, trimboxes, trimboxes[0])

# #Put dates on images
testimage = Image.open(image_names[0])
font      = ImageFont.truetype('DejaVuSans.ttf', size=90)
draw      = ImageDraw.Draw(testimage)
tw, th    = draw.textsize("2019-04-04 (Thu) 00:30", font) 
tx, ty    = (50, trimbox[3]-1.1*th)   # starting position of the message

options = {
  "blend_steps": 10,
  "trimbox":     trimbox,
  "fill":        (255,255,255),
  "textloc":     (tx,ty),
  "out_dir":     out_dir,
  "font":        'DejaVuSans.ttf',
  "fontsize":    90
}

#Generate pairs of images to blend
pairs = zip(image_names, image_names[1:])
#Tuple of (Image,Image,BlendGroup,Options)
pairs = [{**options, "img1": x[0], "img2": x[1], "blendnum": i, "text": image_times[i]} for i,x in enumerate(pairs)]

#Run in parallel
pool.map(make_blend_group, pairs)

This produces a series of images which can be made into a video like this:

ffmpeg -pattern_type glob -i "/z/out_*.png" -pix_fmt yuv420p -vf "pad=ceil(iw/2)*2:ceil(ih/2)*2" -r 30 /z/out.mp4