Preheat a gas or charcoal grill to a moderately hot temperature, about 375°F; if using charcoal, wait until the coals turn white-hot.
While the grill heats up, parboil the potatoes in a large saucepan of salted boiling water until just tender to the tip of a knife, 10-12 minutes. Drain well and set aside to cool.
In a small mixing bowl, stir together 2 Tablespoons rosemary, garlic, ¼ cup oil, and some salt and pepper to taste. Brush the insides of the fish with the rosemary-garlic oil.
When the grill is ready, brush the grates with some oil. Place the fish on the grill and cook, turning once, until the skins are golden brown and the flesh is just cooked through, 6-7 minutes per side.
Meanwhile, cut the potatoes in half and place in a large mixing bowl. Place the remaining ½ cup oil in a small mixing bowl; add the parsley, capers, chili pepper, remaining 2 Tablespoons rosemary, and some salt and pepper to taste; stir to combine.
When ready, remove the fish from the grill and plate on a platter; loosely cover with aluminum foil. Keep the grill on.
Spoon about half the parsley-caper oil into the bowl with the potatoes; toss to combine. Spread out the potatoes on the grill and cook, turning a few times, until lightly charred, 4-6 minutes.
To serve, transfer the potatoes to the platter with the fish, arranging the fish on top. Serve with lemon wedges and the remaining parsley-caper oil on the side.
