Well, many high explosives, particularly the double base ones, not exactly environmentally friendly. Most contain high levels of things like phenyls, amines, picrates, lead, mercury, toluene along others, and with various petroleum products as stabilizers. If the site was used for lots of explosives detonation this could lead to contamination at various levels. As to why one bunker remains intact I have no idea. It is likely though, it was simply forgotten or too much trouble to get rid of.
By way of comparison, there is a "blast proof" building on Davis Monthan AFB's munition dump locally dubbed "The big ugly." It to remains standing mainly because demolishing it is far too much trouble.