The ADP7182 works fine based on your setup above. I am getting a 3.29V at the EN pin when the FET is fully turned off which is enough to ENABLE the ADP7182.
I think the gate bias voltage applied to the FET causes the 0.45V undesirable voltage. Make sure that the FET is fully turned off (you can try VGS= 0V or lower, down to -8V). Also, avoid floating the gate pin to prevent erratic conditions.