Finally the env issue is resolved
It was a frustrating experience, don't know what is wrong with conda and pip, they are like the wife and mistress who can't live together. But after a lot of tinkering I was just able to balance the environment with the perfect blend of dependencies. I can freely test out holoviz now.