Soak the capers in water for at least one hour to remove the salt. Change the water every 15 minutes.
Bring a large pot of salted water to a boil over medium-high heat. Add the potatoes and cook until fork-tender, about 15 minutes. Drain the potatoes and toss them with 1/3 cup of the olive oil.
Meanwhile, in a medium bowl, combine the vinegar, sugar, and 2 ½ cups boiling water and stir until sugar is dissolved. Transfer the onions to a jar and cover them with the vinegar mixture. Allow to cool to room temperature, then strain the onions and set aside.
Meanwhile, strain the capers and pat them dry on paper towels. Heat the remaining 2 tablespoons of olive oil in a small skillet over medium heat. Add the capers and cook, stirring occasionally until they are golden and lightly crispy, about 5 minutes.
In a large bowl, combine the potatoes with the capers, onion, mayonnaise, and fennel fronds and toss to coat. Season with salt and pepper to taste, and garnish with additional herbs and dollops of mayonnaise.