'and what exactly is he asking me to leave?'

i think he's asking you to leave yourself (your ego) and the idea that you are separate--which it sounds as though you have already done:  'we are all part of each other and there is no separation.'  

if you feel that you need to 'leave' even MORE of some residual part of yourself by becoming something that never even resembled you, maybe it would be worth trying to understand why that feeling is so strongly held in you?  

as you say, there is no romanticism in the leaving.  we don't leave to become 'better,' we leave to become more ourselves (my opinion, anyway).  we leave because we are somehow convinced that there is something more satisfying out there than what we've found so far.  and we're 'right'.  

i think it's when i realize that i am not my own, but that i am deeply embedded in and linked to the 'whole', that i find my true place.  it's when i leave the idea that i am 'right' and accept the belief that i don't actually know for sure, but i'm still going with it, that i am on the path for home.