Why were Ishmael and Esau outside the covenant promises even though they descended from Abraham?

Les explains from Romans 9 that not all who are of Abraham are the seed of promise. Ishmael had offspring, and Esau also had offspring; even though genetically they were children of Abraham, he was their father, yet spiritually speaking they were outside the covenant promises. That promise could only go to the offspring of Isaac and Jacob and down that family tree, so all these others are left out. Later, Israel became puffed up and arrogant, thinking that because they were the children of the covenant, the prophets, and the promises, they had nothing to worry about. Paul blows them out of the saddle: they will not enter God's promises simply because they have the right genealogy; they still must enter in by faith. Romans 9:7 says, "neither because they are the seed of Abraham are they all children." Ishmael's offspring and Esau's offspring were children of Abraham, but not according to the promises, because it is in Isaac that the seed should be called, and that seed refers to Christ.