Why did the magistrates fear when they discovered Paul was a Roman citizen?

They feared because they had acted contrary to Roman law by punishing Paul without due process after learning he was a Roman citizen. Roman law made such treatment of citizens a serious violation, and the magistrates realized their unlawful beating and imprisonment exposed them to punishment.

Les explains that the magistrates wanted to get rid of Paul and send him out of town, but Paul refused, saying that if they had beaten him openly as a Roman, they had to deal with him as a Roman. That shook them up, because they had not known he was a Roman citizen. When they were told in verse 38 that he was a Roman citizen, they feared because they had gone contrary to Roman law. They had punished these men without due process, and that was anathema to the Romans just as much as we think it is to us.

Acts 16:38

And the serjeants told these words unto the magistrates: and they feared, when they heard that they were Romans.

Why were the magistrates afraid after finding out Paul was a Roman citizen?

The magistrates feared because they had violated Roman law by having Paul and Silas beaten without due process after learning Paul was a Roman citizen. A Roman could not be treated that way; they now had to answer for punishing him contrary to Roman legal protections.