A method for determining failure of an electromechanically actuated gas shut off valve includes sensing and recording a gas fuel rail pressure and a boost pressure from an air intake manifold at a first time after the dual fuel engine has been started. The method includes opening the gas shut off valve at a second time, holding the gas shut off valve in its open state, and then closing the gas shut off valve after a predetermined interval at a third time. The method includes comparing an actual gas rail pressure decay rate to a threshold gas rail pressure decay rate for the predetermined interval, and determining failure of the gas shut off valve when the actual gas rail pressure decay rate is less than the threshold gas rail pressure decay rate. Upon determining failure of the gas shut off valve, the method also includes initiating a mitigating action.