Abstract: The purpose of this paper is to study the optimization method for problems of rigid body dynamics systems based on the Gaussian principle of the least constraint in generalized coordinates. Compared to the current modeling methods with the Gaussian principle of the least constraint in the form of the particles, it is shown that modeling the Gaussian principle of the least constraint in generalized coordinates has no requirements upon the coordinates, and the modeling process becomes easy and versatile. Two different optimization models are established in this paper under constrained and unconstrained conditions in the example of a double pendulum. A dynamical numerical simulation and a comparative analysis are carried out with a model treated with the Lagrange differential equation using a software MATLAB and the effectiveness of the method proposed in this paper is verified.