While I think Torch was necessary to shake out the new and raw American army, Sicily and Italy were not. This would have given sufficent landing craft to make an assault on the French coast. I also think that a several division sized assault would have been successful even against a port city like Dieppe. After all, that operation involved landing less than a full division and the Canadians nearly succeeded in their operation even if it was supposedly just a "raid." Several divisions landing there would have gotten ashore and been able to stay there.
German defenses along the Atlantic Wall were far less well developed in 1943 and, with their position on the Eastern Front and lack of armored divisions in France compared to 1944 such an invasion would not only have succeeded but likely would have had far more impact on the outcome of the war overall.