I had my coldest commute ever this week. Temp was 8 f. My layering was fine. I have Gerbing pants liners, jacket liner, insoles and gloves. I wore thermal underwear on the bottom with unlined Aerostich Darien pants. On top, I have a BMW Tourance 2 jacket and was toasty warm.
The problem was the helmet. I have a Pinlock that seems to not fit correctly. About half way to work (total commute is 16 miles), the fog filled the inner shield and had frozen. I had to stop at a grocery store and stand inside for 5 minutes to let it thaw. I pushed the inner shield up to make a good seal and continued my ride in with no problems.
Ice on the inside of the helmet is a ride stopper.