After my dad passed my mother tried to find another father-figure so to speak. The first guy wasn't too bad. He was always helping and the main reason I got into Atari and computers in the first place. The guy she found after that, however, was a complete opposite. He almost acted like he wanted nothing to do with me and rarely said anything to me or did anything with me. If he did it was mostly to "correct" me. He never laid a hand on me but he distanced himself from me and, at times, I felt like he was trying to separate Mom and me. The only relief I got was the weekends I would visit either of my grandparents.
I ran into him once at a firework stand not long before Mom passed away in '98. It was strange when I ran into him because I didn't know how to react. All the fears I had of him simply because he never gave me the time of day started to creep back. He acted like I was his best friend in the world and we visited for a almost an hour. It was strange and I still have never figured it out. So I can somewhat relate to how you are feeling.
I would honestly take the time to visit the guy while you can. Even if you say you hated him in your early years he was there so that might account to why you feel how you feel now. There is nothing to be sorry for...allow your family the time to get the closure you are going miss if you don't visit him. Give him a chance, as well, to go in peace if that is his wish.
Sharing something like this on Facebook can generate a lot of mixed feelings and responses. At least here you know you have friends who will be there as much as they can for you. You have my sympathy and prayers are being sent to your family in your time of need.