This is unexpectedly good..
Instead of using colmap. Try using pycolmap panorama_sfm.py. It automatically process the equirectangular images into colmap dataset instead of splitting the equirectangular via ffmpeg.
The cons is no filtering sharp images in panorama_sfm.py.