I have about $10,000 into this deal already - And am closing already closing on the last day available according to the contract we signed -So, I have to be careful.
I know from experience (I was the seller in a deal that didn't close due to last-minute buyer's remorse, not even a good reason stated) that in most states you will get all of your earnest money and deposit back. Consult a lawyer for $100-200 or so to be certain. A lot less expensive than groveling in front of some homeowners' board for years.
Look at the motivations here. They have only sold one parcel, and with real estate the way it is they need to keep every sale they have. They are not going to be sticky about meeting deadlines. Write the seller a letter and say you can't close because of this issue. Put the burden back on him. You aren't refusing to close on time, but the meeting you had to attend to get your approval was postponed -- that is more his fault than yours. This alone is sufficient to postpone the closing until you can be certain you will be allowed to build your structure.
Once you have bought, the homeowners' board are not your friends. They have every motivation to force you into the most expensive possible building. Why? Because it increases their nearby property value at no cost to them. The board will never state this, and the idea may not even form completely in their minds, but at some gut-instinct level they know this.
The wisest man I have ever known told me that in order to understand any human situation: "Look for where the profit lies."