Abstract: In vibration isolation field, nonlinear vibration isolation system catch more attention than linear system because of the better vibration isolation performance. In this paper, a novel nonlinear vibration isolation system with geometric nonlinear friction damping is proposed by add two friction damper that perpendicular to the movement direction of the isolated object. The absolute and relative displacement transmissibility of such kind of vibration isolation system are studied in this paper. Different from the friction damper which usually assuming that the friction force is constant, the friction force studied in this paper is proportional to the displacement of the isolated mass by configuring two linear friction dampers perpendicular to the moving direction of the mass. The mathematical model of the friction damping and the forced vibration of the system are established. The dynamic equation is solved by using Harmonic Balance Method (HBM) subsequently by making some simplification. The result solved by HBM is verified numerically. The performance of the nonlinear vibration isolation system is compared with that of a linear one by the performance index defined by absolute and relative transmissibility. The geometric nonlinear friction can offer small or large friction damping depends on the relative displacement, therefore, the nonlinear friction force can improve the transmissibility for both absolute and relative displacement at resonance and the higher frequencies region if the damping values are chosen carefully which surpass a traditional Kevin vibration isolator model. Meanwhile, the nonlinear vibration isolation system can enlarge the application region for different excitation amplitude and avoid the system failure though the responses of the isolated mass is amplified at low frequency. The vibration isolation system with the configuration of the friction damper proposed is very suitable for both resonance and higher frequencies vibration control. The conclusions given are of importance when design and choosing the friction damping parameters.