I don't have any figures to back this up - only my gut feel. (But see the bottom of this post for a comment by the Stack Exchange community supremo, who's helped nurse about one hundred beta Stack Exchange sites along.)
My feeling is that these first two weeks are crucial. They can see very high attrition rate from committers. And the tone set by the questions will hava a very strong influence on what questions get asked when we go public, and how they are asked.
So I'd say yes, do invite experts to the private beta as soon as possible.
Ideally, you could even post a question or two that you know they'll find interesting, to get them active and engaged as soon as they arrive.
And please allow me to elevate Robert Cartaino's words from his comment below to a more prominent position (emphasis his):
Most definitely 100% yes. If you know experts to help build this site, they should be integral to the creation of this site from day one. You cannot start a site and say "we'll just get the experts later."
(Robert is the Director of Community Development for the Stack Exchange Network)