Guake and Yakuake have this always-on-top terminal overlay mode which is really nice for this kind of workflow. Works exceptionally well with multiple workspaces I prefer one single big monitor with everything maximized and multiple workspaces.

iterm2 on mac also has a similar mode, but it is a bit finicky to set it up (it is really hidden in the settings).