Voters Didn't Give Sabarimala Issue As Much Importance As BJP: Tharoor