H264 video looks like it is flowing

I started having issue after I changed a few things around in the code and now I am totally unable to find what has changed from before. I used to get consistent encoding time and quality but now video looks smeared when recorded with 640x480 sensor and it takes more than 40ms to encode, meaning I am unable to keep stable 25fps.

Here is the effect that I am getting ( also there is a line between video encoded on different core, was also wondering if it can be mitigated). PS. Video is from an infrared array sensor ant the camera is pointed at the table. I am attaching one video of it working and another video shows the way it malfunctions