The 25 -year-old Carter clocked 53.07 seconds running from lane 9 to take a surprise victory as Rio Olympics champion Muhammad finished insecond in 53.50 seconds.
Jamaica's Ristananna Tracey clocking a new personal best tie of 53.74sec claimed bronze, winner of previous two world championships Zuzana Hejnova of the Czech Republic came in fourth.
"I just had to run my race, go out and let them chase me," said the 25-year-old Carter.
Women's 400m Hurdles Results
1 Kori CARTER USA 53.07
2 Dalilah MUHAMMAD USA 53.50
3 Ristananna TRACEY JAM 53.74 PB
4 Zuzana HEJNOVÁ CZE 54.20 SB
5 Léa SPRUNGER SUI 54.59
6 Sage WATSON CAN 54.92
7 Cassandra TATE USA 55.43
8 Eilidh DOYLE GBR 55.71