A nonlinear evolution equation for the radiation field in magnetically insulated transmission line oscillator (MILO) is derived and the threshold conditions of the nonlinear unstable solution are studied. The results show: (1) The instability will arise when the ratio of the nonlinear growth rate to the linear growth rate is lower than 1.2 even if the detuning is arbitrarily slight. (2) If the value of g and γ (the ratio of the nonlinear growth rate to the linear growth rate for the phase) is far from g+γ=1, the instability arises easily because of the critical value of detuning becoming smaller and the critical value of linear growth rate becoming larger. (3) For the larger linear growth rate, the instability solution is relatively difficult.