Newly crowned ICC cricketer of the year Mitchell Johnson gave Australia the perfect start in defending the total, the fast bowler removing openers Hashim Amla and Quinton de Kock with just 18 runs on the board.
South Africa captain AB de Villiers and middle-order batsman David Miller (65) combined for 126 runs to drive the tourists past 200, but Coulter-Nile broke the partnership when Miller miscued to Mitchell Marsh at mid-on with 14 overs left.
De Villiers smashed 80 off 76 balls but was run out by some sharp fielding by Steven Smith, who came in for Clarke and threw down the stumps at the non-striker’s end.
Reeling at 202 for six wickets, Coulter-Nile struck again to remove Ryan McLaren for a duck and Dale Steyn for six, both caught behind by wicketkeeper Matthew Wade, included in the team for the injured Brad Haddin.
In between, paceman Josh Hazlewood dismissed Vernon Philander for one run, and though the tail wagged for another 46 runs, part-time spinner Glenn Maxwell had Imran Tahir caught at long-off for 22 to seal the result.
David Warner (46) and Aaron Finch (35) earlier saw off a fiery early spell from Steyn to compile 94 for the first wicket before seamer Philander removed them both in the same over.
Clarke gloved a catch for 11 and Watson, recalled for his first one-day match after a long lay-off with injury, managed only 13 before sweeping Tahir to Morne Morkel at fine leg.
All-rounder Marsh also went cheaply to leave Australia reeling at 144-5 but Bailey, Wade (35) and Glenn Maxwell (29) defied the South Africans to drive Australia to a competitive score.
(Reporting by Ian Ransom in Melbourne; Editing by Sudipto Ganguly)