In bowl, whisk together eggs, milk, sugar, vanilla and nutmeg.
Arrange bread on large rimmed baking tray and pour egg mixture over top. Turn bread over and let soak until liquid is absorbed, 10 minutes. (You can make ahead, cover with plastic wrap and refrigerate overnight for up to 12 hours.)
Spread butter in large ovenproof 3 litre baking dish and sprinkle evenly with brown sugar. Arrange apples in slightly overlapping rows on top and sprinkle with raspberries. Arrange bread over top.
Bake until puffed and golden, about 35 minutes.
Let stand for 5 minutes. Cut between bread slices to separate. Using large lifter, invert along with apples and raspberries onto each plate.