Rising inflation has meant fewer restaurant meals, getaways and even doctor visits for many Americans, as nearly 4 in 10 say their family finances have gotten worse in the last year.