Our 2.0 TDI used to do exactly this,
It had had a cambelt replaced when we bought it.
I took it for a service and the guy explained what he thought the issue was, that it hadn't been timed using the right equipment, sorted it and it was sweet for years after.