Quite true Hitlers first mistake of the war was at Dunkirk, Although he still believed a truce with Britain was possibly at that point in the war. How wrong he turned out to be.

Hitler was winning albeit at a loss to his own Luftwaffe but never the less he had the British on the run until he ordered the bombing of London which resulted in the RAF deploying their northern squadrons to deal with the German bombers. Thus the Germans believed the RAF wasn't actually on the brink of annihilation as it almost was.
Occupation of england would have been a handful and a depletion of capable manpower available for the wehrmacht. The capture of Britain I believe would have been worth the the hassle, as it would eliminate a nuisance in many theaters of war like the Mediterranean and North Africa which would have released a portion of soldiers back into Europe without an enemy in the fields. This would also allow Hitler to consolidate his forces but give the Russians more time to prepare for the impending onslaught.