On the agenda Tuesday was whether to rescind or basically repeal the March vote.
More than 900 people were against the idea to take back the vote, and 493 wanted to cancel the vote.
"I trust the town officials and town board for all the work they did, I think we have an effective plan here," said Noah Fishman, Waterbury Resident.
"We've got a 5.9 percent increase in the state budget which they just passed last week, and some of the folks that are a little bit older are on limited income, just can't continue to go on and pay more and more," said Everett Coffey, Waterbury Resident.
The $5 million project would renovate the current library to house the town offices, along with the historical society and town library.
Waterbury is counting on a $1 million dollar Disaster Recovery grant which expires in June.
Copyright 2016 Nexstar Broadcasting, Inc. All rights reserved. This material may not be published, broadcast, rewritten, or redistributed.