We consider the inverse problem of reconstructing the optical parameters for stationary radiative transfer equation (RTE) from velocity-averaged measurement. The RTE often contains multiple scales char- acterized by the magnitude of a dimensionless parameterthe Knudsen number (Kn). In the diffusive scaling (Kn ≪ 1), the stationary RTE is well approximated by an elliptic equation in the forward setting. However, the inverse problem for the elliptic equation is acknowledged to be severely ill-posed as compared to the well- posedness of inverse transport equation, which raises the question of how uniqueness being lost as Kn → 0. We tackle this problem by examining the stability of inverse problem with varying Kn. We show that, the discrepancy in two measurements is amplified in the reconstructed parameters at the order of Knp (p = 1 or 2), and as a result lead to ill-posedness in the zero limit of Kn. Our results apply to both continuous and discrete settings. Some numerical tests are performed in the end to validate these theoretical findings.