I'm not entirely sure where downtown Grayson is but Grayson Coffee House seems to be centrally located on Grayson Parkway. It's in a very pleasant old house with several rooms in which patrons can relax with their coffee. Based on the look of some of the exposed wood inside, I would guess that the core of the house is about a century old.
Here is my iced coffee, which was very tasty. They use Jittery Joe's coffee, which came through nicely along with the cream. They have a big selection of baked goods. Their breakfast selection seems fairly complete, including oatmeal. They also run a car service in the mornings---just park in back and they will come out the back door to help you.