This is unexpectedly good..
Instead of using colmap. Try using pycolmap panorama_
sfm.py. It automatically process the equirectangular images into colmap dataset instead of splitting the equirectangular via ffmpeg.
The cons is no filtering sharp images in panorama_
sfm.py.