Abstract: In order to study the working mode selection mechanism of gyrotrons, this paper investigates mode selective properties of three kinds of gyrotron cavities, i.e. cylindrical open cavity, coaxial cavity with smooth insert and coaxial cavity with corrugated insert, and further discusses the mode competition phenomenon in these cavities. Through the calculation of dispersive curves of coaxial waveguide, it is determined that coaxial cavities can rarefy the mode spectrum and reduce the competing action of some high-order modes, compared with the cylindrical open cavity, enabling stable single mode operation of gyrotron. Finally, the numerical calculation of dispersion characteristics of a coaxial gyrotron in the KIT(Karlsruhe Institute of Technology) is presented, and the results indicate that