In 1714 Luis Moses Gomez, who as a child, had fled with his family from the Spanish inquisition, purchased 6,000 acres of land along the Hudson Highlands where several Indian trails converged. On the western border he built a fieldstone blockhouse trading post into the side of a hill along a stream that became known as "Jews Creek."

The great walls of the trading post - which are about three feet thick - survive, as do two original fireplaces and evidence of the limestone floors that covered the ground of the main room. The main resources of timber and lime drove the industry he and his son Daniel conducted on the property. The trading post served to provide basic commodities to other local residents, settlers traveling north and the few remaining Native Americans still trading along the Hudson River. On a northern point of the river front property, stood the Duyfil's Danskammer, a Native American ceremonial campsite described in the ship's diaries of Henry Hudson in 1609. For over thirty years, Luis Gomez and his sons conducted a thriving enterprise from the trading post. A leader of Colonial Jewish America, Luis Moses Gomez became parnas (president) of Shearith Israel, the Spanish and Portuguese Congregation. Under his leadership, he led the drive to build the Mill Street Synagogue, the first Synagogue in New York. Among the more prominent descendents and extended family were the poet Emma Lazarus and Supreme Court Justice Benjamin Cardozo, and New York Governor Hamilton Fish.

Before the Revolutionary War, Gomez Mill House was purchased by Wolfert Acker, a Dutch-American who added a second story and attic with bricks made from clay found nearby. During the Revolutionary War, Acker served as a lieutenant in the New Marlborough Company of Minute Men and chairman of the Committee of Safety while General Washington's army was camped close by in Newburgh. The house became a center for meetings of the new American patriots. After the war, Acker established a landing on the Hudson with a ferry to cross the river to Beacon, and a packet line to carry freight.

In the 19th century, gentleman farmer and conservationist, William Henry Armstrong and his family made Gomez Mill House their home for five decades. They added the kitchen wing and porch and rolling stone walls to the property. At the Danskammer, painter, statesman and brother D. Maitland Armstrong lived with other members of the prominent Armstrong family.

The most famous owner in the 20th century was Dard Hunter, renowned craftsman and paper historian who, just prior World War I, built a paper mill on Jew's Creek in the shape of an English country cottage complete with a thatched roof. Students from all over the world came to learn from him as he made paper by hand, cut and cast type and hand-printed his own books. The Gomez Foundation for Mill House restored Hunter's Mill and the mill dam and bridge in 1997, which in 2010 underwent a second major restoration.