Grayville's library owes its beginnings to the Grayville Shakespeare Club's efforts starting in 1901.  It took until 1909 for the club to form the first library on the second floor of the City Hall building located on the southeast corner of North and Main Streets.  By 1911 the library outgrew the space, and the Shakespeare Club contacted the Carnegie Foundation and procured $7,000 to build the Grayville Carnegie Library on Mill Street, which opened in 1913.

By 2001 the library had again outgrown its space, and the building had structural issues.  Herschel and Becky Groff spearheaded the drive for a new library building, and the Grayville Library Boosters were formed in 2002.  After much hard work and fundraising by the Boosters,  the present library building at 118  S. Middle Street was opened to the public in 2011.   Its new name was the Groff Memorial Public Library & Museum in honor of the work Herschel and Becky put into obtaining a new library for the Grayville community.