I'm trying to implement a motion vector warping like Autodesk Flame (https://www.youtube.com/watch?v=L7lRjsngKUQ).
I can get a motion vectors by deep learning based method. Now, to create the texture for the 2nd frame from the first frame, I warp the texture using backward motion vectors. To create the 3rd frame, I accumulate the motion vectors and use it to warp a texture.
There is no major problem when the number of frames is small (< 20), but as the number of frames increases, the error accumulates.
Is there a good way to find the deformation from the first frame to any frame without accumulating errors?
I tried to calculate the motion vectors directly from the first frame, but I could not get the occlusion handling right when the motion was large.