My E61 530d LCI does the same thing.
In fact, there are plenty of complaints about it on several forums.

It is probably associated to the functioning of the EGR valve. When accelerating strongly, the problem is not present.
I don't really believe that this has to do with swirl flaps, because newer E60/E61 are not much affected by that (it seems that the swirl flaps were redesigned just before the facelift E60/E61 LCI).

My car needs to be serviced in a few months and I will probably take out the intake and the EGR for cleaning. Hope this helps.
This problem tends to disappear when the engine is remapped, because there are changes in a lot of parameters, notably related to the functioning of the EGR. It could be a nice solution...