I interpreted the OP's use of the word "baggage" not to mean a burden or impediment for individual speakers, but instead a cultural and political reason why some Indians might hesitate to consider English their national language.