Tutorial – How to synchronize FPS to BPM between footage and audio, using Grasshopper and FFmpeg

Опубликовано: 13 Октябрь 2024
на канале: Sanning Arkitekter

Timestamps:
0:00 [Disclaimer] Tutorial of dubious quality – recommended to search for alternative source of learning
0:18 [Intro] Overview of workflow at hand
1:14 [Song BPM] Introduction to free web service (using data from Spotify) detailing the BPM for published songs
1:28 [FFmpeg] Using Command Prompt and FFmpeg to merge two video clips into one
3:16 [Grasshopper] Using a custom Grasshopper definition to calculate synchronized frame rate to pace of music (with tolerance)
4:26 [Grasshopper] Considerations regarding the frame rate so as to avoid problems during playback, and choosing a fill-in frame rate
5:51 [FFmpeg] Using PowerShell and FFmpeg to extract frames from merged video
6:33 [FFmpeg] Using PowerShell and FFmpeg to restitch the extracted frames to a timelapse video
7:17 [DaVinci Resolve] Creating a DaVinci Resolve project, adjusting project frame rate, and adding the timelapse
8:01 [DaVinci Resolve] Adding the audio layer, and adjusting the temporal position of it so as to match the timelapse video image changes
9:00 [DaVinci Resolve] The current playback head position shows the frame that is immediately to the right of the playback head
9:30 [DaVinci Resolve] If the relationship between BPM and FPS is not possible to express as an integer, the synchronization will fluctuate
9:52 [DaVinci Resolve] Using a higher framerate for the intermediary footage might diminish the problem of unsynchronized audio
10:11 [DaVinci Resolve] Adding fade-out and fade-in to the audio and video clips
10:25 [DaVinci Resolve] Exporting the final movie clip
10:49 [VLC] Final movie clip
12:19 [Outro] Evaluating the final movie clip

.........................................

Links:

Tutorial assets:
https://drive.google.com/drive/folder...

FFmpeg project page:
https://www.ffmpeg.org/

FFmpeg download mirror:
https://www.gyan.dev/ffmpeg/builds/

SongBPM (song data provided by Spotify):
https://songbpm.com/

Sublime Text:
https://www.sublimetext.com/

VLC:
https://www.videolan.org/

Rhinoceros (Grasshopper included):
https://www.rhino3d.com/

DaVinci Resolve:
https://www.blackmagicdesign.com/prod...

.........................................

It is assumed that you already have all applicable software installed.
For an introduction on how to download and use FFmpeg, you may watch this tutorial:
• 5 minutes on: File management – Merge...

.........................................

Make sure that any command which references the FFmpeg installation is adjusted to your own installation directory.

.........................................

Example commands:

Command Prompt:
(Replace ¶ Pilcrow sign with right-pointing bracket/greater than sign (YouTube forbids this sign in descriptions due to it interfering with HTML/CSS)

D:
cd D:\Tutorials_ProducedOutsideOfWorkHours\240421_HowToSyncFrameRateWithBPM\Footage
dir /b ¶ FileList.txt

(File list should look like this:)
file _DSC1168.MOV
file _DSC1169.MOV
file _DSC1170.MOV
file _DSC1171.MOV
file _DSC1172.MOV

D:\FFmpeg\ffmpeg-2023-05-18-git-01d9a84ef5-full_build\bin\ffmpeg.exe -safe 0 -f concat -i FileList.txt -vcodec copy -acodec aac -strict -2 -b:a 384k Merged.mp4

PowerShell:

D:\FFmpeg\ffmpeg-2023-05-18-git-01d9a84ef5-full_build\bin\ffmpeg.exe -i Merged.mp4 -vf fps=0.188874 frame%04d.png

.........................................

This tutorial demonstrates to KTH Architecture students how to synchronize the frame rate (FPS) of a video clip to the pacing (BPM) of an audio clip. To do this, a Grasshopper definition is used, and FFmpeg is used to extract frames and stitch them together. DaVinci Resolve is used to merge the video and the audio layers to one final clip.

The operating system used for this tutorial is Windows 10. Running another operating system (e.g. Linux, MacOS) might amount to a different user experience or software incompatibility.

If you want to check in which computer room(s) any relevant software might be installed in – CADLAB, DFL, and ARCPLAN are in the KTH School of Architecture building – please check the following link for current status:
https://www.kth.se/en/student/it/camp...

Disclaimer:
This tutorial aims to increase student awareness of the versatility of the digital tools available for use within the context of the architectural education offered by KTH. As such, it might not be generally applicable, but on the other hand, if at least one architecture student is helped by it in fulfilling deliverables requirements or being taught something of genuine interest, the purpose of this tutorial is satisfactorily achieved. There might be inaccuracies in this tutorial – if you identify any significant one, please tell us in the comments.