To fight Olaf you must first complete the other 3 fistfights on The Skellige Isles.
The first one is in Kaer Trolde Harbor - town in the north of main island)
the second in Arinbjorn - far west on the main island)
the third in Harviken - small island in the south-east
The fight against Olaf then takes place on Urialla Harbor (small island in the north of the map). Go to the arena to start the fight.
You don't necessarily need to be the recommended levels as long you know what you're doing (dodge, counter attack ect..)