To unlock the Nemesis fight, you will need to capture 10 of each fiend in the game, and then defeat each fiend in the Area Conquest, Species Conquest, and Original sections of the monster arena at least once. To capture fiends, you need to buy weapons with the "capture" trait from the Monster Arena owner.
Fiends to be captured:
Mushroom Rock Road
Stolen Fayth Cavern
Splashers, achelus, and maelspikes are found in the underwater section. Ahrimans, grendels, behemoths, dark flans, and mandragoras are found in the caves.
Inside Sin: Note that the point of no return is once you go inside the tower that comes crashing down, right after a save point.
Omega Dungeon: Omega Dungeon is found by putting in the coordinates X=74.101, Y=36.099 into the airship.
Many of the above fiends can be found in more than one area, but are classified by these areas according to the Monster Arena.
Once you have captured 10 of each, you need to beat the following fiends in the Monster Arena:
These monsters are some of the toughest enemies in the game. If you're underleveled, you should grind using the Don Tonberry trick. You will level a decent amount just capturing fiends, too. I recommend using zanmato to beat the harder bosses in the Original section if you're still having trouble. This can work for everything except for Shinryu, which is an underwater fight where you can't use Yuna. I had a max-damage Wakka use his Attack Reels overdrive with a break damage limit weapon, twice. At max damage, a perfect Attack Reels will deal 1.2 million damage. Use "Entrust" to fill his overdrive a second time. One perfect use of a damage-break Attack Reels can also one-shot many of the other fiends.
Once you have beaten all of the fiends, Nemesis will unlock in the second page of the original creations section. Using zanmato is the easiest way to beat him.
If you want to fight him normally, he has 10 million HP. He counters every attack with Ultima. All characters should have armor with ribbon equipped, since Nemesis' Ultra Spark casts Poison, Curse, Slow, and Power Break on the party. Auto-Haste armor should also be equipped. Nemesis' ultimate attack Armageddon deals 99,000 damage to the entire party, so at least one person should have auto-life cast on them. Use aeons for meat shields, and Quick Hit him until he goes down. Note that Kimahri can learn Nova from Nemesis.
Comment with any questions about other fiends!