Foxgloves are common around here and we have quite a few in our yard. You have to ingest a lot to have any prolems. They re-seed readily. It is also common in Newfoundland to see Monkshood (wolfsbane) in older gardens which is considerably more poisonous. I don't know but I would expect that's probably true in the UK as well as that is where most of the heritage plants that we have came from.
I got a grin from 'Day of the Triffids' as I remember seeing it in the early 60's when I was pretty young and it left a distinct impression on me. It was not too long after the so called Cuban missile crisis. I happened to catch it again a couple years ago on an Air Canada flight among the classic movie offerings. That viewing removed any residual left over from my distant youth.
Chuck