In my experience as a foreigner in Scandinavia, Swedes are way way faster to switch to English, many do it almost as soon as I make a small mistake, whereas Norwegians are much more tolerant and will keep up unless I switch myself.