Abstract: In order to unfold the measured recoil electron spectrum generated by gamma ray in a deuterated liquid scintillator, standard gamma sources were used to measure the -ray response of BC537 detector and the -ray response function for the BC537 detector was calculated by Geant4 codes which was based on the Monte-Carlo method. The effects of the aluminum container and the beam radius on the detection efficiency were investigated as a function of the beam energy by Geant4 simulation. The simulated response function shows a good agreement with the experimental data at different energies. With the response function and the recoil electron spectrum of three reference -ray sources, an iterative unfolding algorithm was used to unfold the -ray spectrum, and the results further verify the practicability of the simulated response function.