In performance, carrier frequency offset generated by the system synchronization can cause not only nonlinear frequency offset self-interference in the ultra-wideband millimeter-wave multi-input multioutput (MIMO) orthogonal frequency division multiplexing (OFDM) system, but also influence of subsequent channel estimation performance. For this reason, this paper proposes a method for joint estimation of nonlinear frequency offset self-interference and channel, by exploiting the hyper-dimensional hidden space of data, reducing the complex multi-objective optimization problem into multiple parallel solvable sub-optimization problems, and using the sparse characteristics of ultra-wideband millimeter wave channels to realize iterative estimation of carrier frequency offset and channel based on compressive sensing. Compared with the existing methods, the simulation results show that the proposed method can effectively reduce the computational complexity by nearly one order of magnitude, improve the accuracy of channel estimation, and reduce the system bit error rate.