Project

This project is based on the paper Fast-Forward Video Based on Semantic Extraction on the IEEE International Conference on Image Processing (ICIP 2016). It implements a semantic fast-forward method for First-Person videos.

For more information and visual results, please access the project page.

Contact

Authors

• Washington Luis de Souza Ramos - MSc student - UFMG - [email protected]
• Michel Melo da Silva - PhD student - UFMG - [email protected]
• Mario Fernando Montenegro Campos - Advisor - UFMG - [email protected]
• Erickson Rangel do Nascimento - Advisor - UFMG - [email protected]

Institution

Federal University of Minas Gerais (UFMG)
Computer Science Department
Belo Horizonte - Minas Gerais -Brazil

Laboratory

VeRLab: Laboratory of Computer Vison and Robotics
http://www.verlab.dcc.ufmg.br

Code

Dependencies

• MATALB 2015a or higher

Usage

• The project processing is decribed by the following flowchart:

1. Optical Flow Estimator:

   The first step processing is to estimate the Optical Flow of the Input Video.

   1. First, you should download the Poleg et al. 2014 Flow Estimator code from the link.
   2. Navigate to the download folder and unzip the code.
   3. Into the Vid2OpticalFlowCSV\Example folder, run the command:

Vid2OpticalFlowCSV.exe -v < video_filename > -c < config.xml > -o < output_filename.csv >

Options	Description	Type	Example
< Video_filename >	Path and filename of the video.	String	~/Data/MyVideos/myVideo.mp4
< config.xml >	Path to the configuration XML file.	String	../default-config.xml
< output_filename.csv >	Path to save the output CSV file.	String	myVideo.csv

Save the file using the same same name of the video file with extension ".csv".

2. Semantic Extractor:

   The second step is to extract the semantic information over all frames of the Input video and save it in a CSV file. On the MATLAB console, go to the project folder and run the command:

>> ExtractAndSave(< Video_filename >)

Parameters	Description	Type	Example
< Video_filename >	Path and filename of the video.	String	~/Data/MyVideos/Video.mp4

3. Calculate Speed-up rates

   To calculate the speed-up rates for each type of segment, on the MATLAB console, go to the project folder and run the following commands:

>> [~, < Num_non_semantic_Frames >, < Num_semantic_frames >] = GetSemanticRanges(< Semantic_Data_MAT_filename >);

Parameters	Description	Type	Example
< Semantic_Data_MAT_filename >	File created by the step 2	String	'../example_face_extracted.mat'

Output	Description	Type	Example
< Num_non_semantic_Frames >	Number of frames in the Non-Semantic segments. (Will be used below.)	Integer	Tns
< Num_semantic_Frames >	Number of frames in the Semantic segments. (Will be used below.)	Integer	Ts

4. Calculate Speed-up rates

   To calculate the speed-up rates for each type of segment, on the MATLAB console, go to the project folder and run the command:

>> SpeedupOptimization( < Num_non_semantic_Frames >, < Num_semantic_frames >, < Desired_speedup >, < Max_speedup>, < lambda_1 >, < lambda_2 >, < show_plot > )

Parameters	Description	Type	Example
< Num_non_semantic_Frames >	Number of frames in the Non-Semantic segments.	Integer	7643
< Num_semantic_Frames >	Number of frames in the Semantic segments.	Integer	12935
< Desired_speedup >	Desired speed-up rate to the whole video.	Integer	10
< Max_speedup >	Maximum allowed jump.	Integer	10
< Lambda_1 >	Value of Lambda 1 in the optimization function.	Integer	40
< Lambda_2 >	Value of Lambda 1 in the optimization function.	Integer	8
< show_plot >	Flag to show the search space create by the optimization function.	Boolean	false

5. Create Experiment:

   To run the code, you should create an experiment entry. On a text editor, add a case to the GetVideoDetails function in the file SemanticSequenceLibrary.m:

function [videoFile, startInd, endInd, filename, fps] = GetVideoDetails(video_dir,exp_name)

 ...

 case < Experiment_name >
     filename = < video_filename >;
     startInd = < start_index_frame > ;
     endInd   = < final_index_frame >;
     fps      = < video_frames_per_second >;
					
  ... 
  

Fields	Description	Type	Example
< Experiment_name >	Name to identify the experiment.	String	MyVideo
< video_filename >	Filename to the video.	String	myVideo.mp4
< start_index_frame >	Frame index to start the processing.	Integer	1
< final_index_frame >	Frame intex to stop the processing.	Integer	16987
< video_frames_per_second >	Frames per second of the video.	Integer	30

5. Semantic Fast-Forward:

   After the previous steps, you are ready to accelerate the Input Video. On MATLAB console, go to the project folder and run the command:

>> SpeedupVideo(< Video_dir >, < Experiment_name >, < Semantic_extractor_name >, < Semantic_speedup >, < Non_semantic_speedup >, < Shakiness_weights >, < Velocity_weights >, < Appearance_weights >, < Semantic_weights > )

Parameters	Description	Type	Example
< Video_dir >	Path to the folder where the video file is.	String	~/Data/MyVideos
< Experiment_name >	Name set in the SemanticSequenceLibrary.m file.	String	My_Video
< Semantic_extractor_name >	Semantic extractor algorithm used in the Semantic Extractor step.	String	'face' or 'pedestrian'
< Semantic_speedup >	Desired speed-up assign to the semantic segments of the input video.	Integer	3
< Non_semantic_speedup >	Desired speed-up assign to the semantic segments of the input video.	Integer	14
< Shakiness_weights >	Tuple of weights related to the shakiness term in the edge weight formulation.	[Integer, Integer]	[10,250]
< Velocity_weights >	Tuple of weights related to the shakiness term in the edge weight formulation.	[Integer, Integer]	[50,3000]
< Appearance_weights >	Tuple of weights related to the shakiness term in the edge weight formulation.	[Integer, Integer]	[0,100]
< Semantic_weights >	Tuple of weights related to the shakiness term in the edge weight formulation.	[Integer, Integer]	[500,8]

Acknowledgements

1. EgoSampling: Y. Poleg, T. Halperin, C. Arora, S. Peleg, Egosampling: Fast-forward and stereo for egocentric videos, in: IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 4768-4776. doi:10.1109/CVPR.2015.7299109.
2. LK_Blocked_Optical_Flow: Y. Poleg, C. Arora, S. Peleg, Temporal segmentation of egocentric videos, in: IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2537-2544. doi:10.1109/CVPR.2014.325.
3. NPD_Face_Detector: S. Liao, A. K. Jain, S. Z. Li, A fast and accurate unconstrained face detector, IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (2) (2016), pp. 211-223. doi:10.1109/TPAMI.2015.2448075.

Citation

If you are using it for academic purposes, please cite:

W. L. S. Ramos, M. M. Silva, , M. F. M. Campos, E. R. Nascimento, Fast-forward video based on semantic extraction, in: -IEEE International Conference on Image Processing, Phoenix, AZ, 2016, pp. 3334-3338. doi:10.1109/ICIP.2016.7532977.

Bibtex entry

@InProceedings{Ramos2016,
author = {W. L. S. Ramos and M. M. Silva and M. F. M. Campos and E. R. Nascimento},
booktitle = {IEEE International Conference on Image Processing},
title = {Fast-forward video based on semantic extraction},
year = {2016},
month = {Sept},
Address = {Phoenix, AZ},
pages = {3334-3338},
doi = {10.1109/ICIP.2016.7532977}
}

Enjoy it.