He married Marie Anne Francoise (called Fanny) Mouchard, a woman of letters who had a celebrated salon.
By Marie he left a daughter, Anne Marie, duchesse de Montpensier; and by Marguerite he left three daughters, Marguerite Louise (1645-1721), wife of Cosimo III., grand duke of Tuscany; Elizabeth (1646-1696), wife of Louis Joseph, duke of Guise; and Francoise Madeleine (1648-1664), wife of Charles Emmanuel II., duke of Savoy.
Whatever the number of these, he had only two titular mistresses - at the beginning of the reign Francoise de Chateaubriant, and from about 1526 to his death Anne de Pisseleu, whom he created duchesse d'Etampes and who entirely dominated him.
De Boisy, a nobleman and soldier, had been employed in various affairs of state, but in 1560, at the age of thirty-eight, settled down on his ancestral estates and married Francoise de Sionnay, a Savoyard like himself, and an heiress.
On retiring from the service he married Francoise de Castellane, and left at his death, in 1737, three sons - Victor marquis de Mirabeau, Jean Antoine, bailli de Mirabeau, and Comte Louis Alexandre de Mirabeau.