If like me you got to the end of the game and finished your request but still haven't' got it you my be asking what have you done wrong to miss this out. Now the main reason you probably missed this is that the Sworn Ally diamonds do not appear at high level miasma.
First of all the way to see if a stage has the character you are looking for is by looking at the available officers on that level and if it has an officer as ???????????? then it has one you need. Now you have two ways to find out which famous officer it is you need. Jot down all the officer names on the level that way when you see an famous officer on the level that isn't jotted down you know that's your guy. Or go to your character selection screen and figure out which ones you need by the darkened characters.
At about 135 characters unlocked you will receive the formation called 'Sky Formation'. This reduced the miasma level to 1 for a few seconds. So run around till you find the character you need. This has to be activated by your party leader (the guy that you start the level with). One hit and he will die and drop the crystal you are looking for.
I had six characters to get at the end of the gauntlet mode and this worked on every single one of them. I had tried different methods such as different types of characters beating them such as same factions or having a special bond between them and this did not work but the miasma method did.
One other method which i did read that it worked for many people but did not for myself (although i only tried it three times,) is in the first DLC level you can apparently get these sworn ally gems in chests. gain this did not work for me but i have found on several sites that it had worked for them.