I think that what you're actually looking for is The Rule of Least Power, just applied to types. You should choose the lightest weight, conceptually simplest approach you can that keeps the code intelligible to others.

Smart Pointers should be used only when regular borrowing does not work.