Congress is considering whether to charge home buyers and home owners higher Fannie Mae and Freddie Mac loan fees to help cover the cost of extending cuts in the payroll tax through the end of the year.

The payroll tax cut is scheduled to end Feb. 29, when a two-month extension passed late last year by Congress expires.

The tax cut has been saving the average worker about $20 a week in taxes and continued unemployment benefits for 2 million Americans.

To cover the $33 billion cost of extending the payroll tax cuts, Congress came up with a list of spending reductions and fee hikes, including an increase in the fees lenders pay to have Fannie Mae or Freddie Mac guarantee a mortgage.

Lenders pass along those higher costs to consumers, making it more expensive to buy a house, or refinance a mortgage.

Savings not noticed?

Championed by the Obama administration, the payroll tax holiday didn’t register with most voters, according to a poll done for The Hill, a nonpartisan newspaper covering Congress. Only 35% of Democrats and 21% of Republicans polled said they benefitted from the payroll tax cut.

“The poll lends some credence to GOP arguments that many taxpayers didn’t notice they were getting more money in their paychecks,” The Hill reporter Vicki Needham wrote.