Everyone should check out the interpret-community Python package developed by Microsoft that encapsulates a lot of these what if scenarios. There's an equivalent "no-code" dashboard. More importantly, it relies heavily on SHAP where possible, which I think is the cat's pajamas.
It is now in my default toolbox and almost all my models go through it.
I use it also to “teach“ data science.
Plus I met Rich Caruana who was presenting this at KDD, besides his awesome work (check out his paper on pneumonia), he’s a warm, simple and really receptive person.
If you want to try out the what-if tool in a jupyter notebook (for free), create an account here on www.hopsworks.ai. It's our new SaaS ML platform, with a feature store. If you run the deep learning tour, it will populate a project with this notebook (it's open-source):
