I'm glad you came by and commented because it's an issue that always needs clarification.

I had a fallen pastor's wife email me a few months ago. Her question was, "He's not repenting, am I supposed to forgive him?" In fact, not only was he not repenting, but he was telling her that everything was her fault, she was to blame for his adultery, and that her unwillingness to forgive was a sin. The entire time, he was ignoring his own sin and not asking forgiveness.

Restoring someone and walking by them also means turning them toward the right path - toward repentance. The pastor's wife also needs love and compassion. You are hurting and in need of care. If his attitude truly is one of "freedom" since he doesn't have to worry about his former marriage, then he's probably in the same situation many fallen pastors are right after their fall - denial. He does need help, but not the kind he's getting. 

Pray that God will work on his heart. And your heart as well. Love your children as you know how to. Pray for godly, discerning people to rise up and restore your husband as he should be. And I will be praying for you both.