Imparite Stones are what give your power to summon more Imps. You can find these stones by either defeating enemies or destroying the towers that hold them.
To collect them they have to be taken to one of your Impa Portals. This is the first achievement I got after the tutorial had ended. There will be many opportunities to get this throughout the game. I
If you fail to do this due to the time limit try doing it later once you have collected some treasure chests and increase your time limit.