Abstract: Traditional computer vision methods usually focus on the in-plane dynamic response of structures. Therefore, this paper proposes an image phase-based stereo matching temporal analysis method to achieve targetless robust monitoring of three-dimensional structural deformation. This method uses 2D-Gabor filters and Gaussian pyramid gradient algorithms for image preprocessing, applies a phase-based dense optical flow tracking algorithm and an improved semi-global block matching (SGBM) algorithm to realize full-field measurement of structural displacement in the region of interest, and further proposes an intuitive displacement-strain conversion method to measure three-dimensional strain of structures. Through virtual reality experiments based on physics-based graphics models (PBGM), it is verified that the error of this method compared with 3D-DIC and finite element analysis deformation is less than 2%; in vibration tests of outdoor bridge structures in the laboratory, the deformation error compared with traditional testing methods can be controlled within 8%, meeting engineering application accuracy. Without compromising accuracy, this method achieves targetless robust monitoring of three-dimensional structural deformation, and better solves the problems of large environmental impact and high cost in traditional structural deformation monitoring.