Armor Penetration ignores an equivalent amount of the target's defense. For example, if a boss has 30 defense and you have 20 Armor Penetration, the boss's effective defense is treated as 10. For fast-hitting weapons with low base damage (like Megashark or Bee Gun), Armor Penetration dramatically multiplies your total DPS.
Does Defense reduce damage by a percentage or flat amount in Terraria?
Defense reduces incoming damage by a flat amount per hit, not a percentage. In Master Mode, 1 point of defense reduces damage by exactly 1 flat damage point; in Expert Mode it reduces by 0.75; and in Classic Mode it reduces by 0.50.