Reader Jonathan mentioned that he

Reader Jonathan mentioned that he was reading a book about haunted locations and came across a description of Glamis Castle in Scotland which mentions poltergeist activity (Peeves, anyone?), the “Grey Lady” ghost haunting it (you might remember she is the ghost for Ravenclaw), and a secret chamber in which a horrible monster resides that is the rightful heir to the house (Hmmm. Chamber of Secrets? Heir of Slytherin?). Sounds like it may have had a bit of influence on Hogwarts. Here’s some snippets I found across the Web describing the hauntings:

The family chapel is haunted by a Grey Lady who is said to be the spirit of Lady Janet Douglas, burned at the stake as a witch on Castle Hill, Edinburgh in 1537, on charges of plotting to poison the King. It is likely that the charges were fabricated for political motives. The apparition has been seen relatively recently in the chapel by a number of witnesses. She is also said to appear above the clock tower.
http://www.mysteriousbritain.co.uk/hauntings/glamis.html

But the strangest and the more famous thing about the Glamis Castle lies in a awful secret that only the leader of the familly Strathmore know. People are saying the secret goes from father to son. Some people think it is about the “Glamis monster”, an unnatural child born with diminutive limbs and no neck that got hidden in a secret room for all his life.
http://www.castleofspirits.com/glamis.html

In 1908 a writer in Notes and Queries – by no means given to publishing mere sensationalism or idle gossip – referred to the current story “that in the Castle Glamis there is a secret chamber, in which is confined a monster who is the rightful heir to the titles and property, but is so unpresentable that it is necessary to keep him perpetually out of sight”. And in 1925 another writer referred to the castle and “its inviolable secret, which is no nearer solution today than it was hundreds of years ago”.
http://members.nbci.com/janicekmc/Halloween.htm