Harry Spencer Chichester, 2nd Baron Templemore, was the eldest son of Arthur Chichester, 1st Baron Templemore, and Lady Augusta Paget, daughter of the 1st Marquess of Anglesey. His younger brother was the Hon. Frederick Chichester, and they were both first cousins of Henry William George Paget, the Earl of Uxbridge. His first wife, Laura Caroline Jane Paget, whom he married in 1842, was a distant cousin and the daughter of the Rt. Hon. Sir Arthur Paget and Lady Augusta Fane. His second wife was Lady Victoria Elizabeth Ashley, daughter of Anthony Ashley Cooper, 7th Earl of Shaftesbury, the noted philanthropist.