This includes both agility orbs and secret orbs. You can find agility orbs on the rooftops of most taller buildings, on the sides of the buildings you use to get to each boss fight and on nearly every propaganda tower and surveillance cache. They are also along the pathways for every rooftop run.
Keep your volume turned up to hear the orbs' distinctive ringing and to hear your boss tell you when a secret orb is nearby. You can see agility orbs from far off by looking for their distinctive vertical light. If you play at nighttime, you may find the secret orbs more easily, as they glow bright blue and can be seen from far way as well.
You will get to 500 orbs long before you collect every agility orb or secret orb. I managed to get this just by collecting orbs as I played the game and did not have to use any online guides to get to 500, so it should not be too difficult.