Another way to look at a land tax (instead of as a wealth tax) is that it's sort of "user fee". You're paying society "rent" (a land tax was called ground rent by Adam Smith) for depriving the commons of that land.

Typically, land value tax is based on the rental value of land rather than the market value, so it's even more in line with this model.

We want to move to a society where land is not treated as wealth, but rather as a resource to use (because that leads to less land speculation, and more productive use of land), so this model is fine.