When the Town was settled on the site of the famous old Indian Reservation in 1801. It has had many names over the years. One time called Caryville in honor of Alfred Cary who found the Cary Collegiate Seminary, then Plain Brook and finally Oakfield. In 1825 a very important type of stone was found in Oakfield called gypsum. In 1842 Stephen Olmstead built a water powered grist mill and there also milled gypsum. This began the explosion of gypsum mining in the area. The first school opened in 1817 and the first church was a Free Will Baptist Church in 1830. Colonel Alfred Cary opened the first store in 1833.

In 1840 Col. Cary built the Cary Collegiate Seminary which opened in 1844. Oakfield was set off from Elba on April 11, 1842. It received its name from the fact that the lands of the town were chiefly “Oak Openings.” It lies on the north border of the County, west of the center. The surface is level or gently undulating. Oak Orchard Creek, the principal stream, flows westerly through the north part of the town. The village was incorporated in 1858 and the first officers were elected at a meeting at the Olcott House later known as the Arnold House.

During the height of the mining years the town boasted several hotels, hardware stores, banks, blacksmiths, shoemakers, lumberyards, pharmacists and many other businesses around the area. Around the village were large expansive farms that boasted large quantities of vegetables and dairy. The area also home to a large railroad yard that is long since gone. Just outside of the village is a landmark of note. It is the very old remains of a Native village that dates back to about the 1400’s or earlier. The English translation for the site was the “Double Walled Fort”. It was recorded in the 1788 that there was also a “Bone Fort” located nearby. It is unclear what type of bones these were. Today the town and village still remain principally agriculture. Mostly diary and vegetables can still be found in the surrounding area.