Before you change it too much, try firing it without the blower attached. I still suspect you over-stoked it with the blower. Also I’m not sure if the impeller is blocking the flow with the blower off.
Another easy test, just leave the door cracked open. If that clears it up, you need more primary air.
Insulating the flue will have a massive effect. Think rocket stove.