In a large frying pan over medium heat, melt the butter and saute the apples. Once the apples are a bit soft, add the sugar, cinnamon and nutmeg. Continue to cook until apples are tender, 10 to 15 mins. Allow the mixture to cool.
In a trifle bowl, arrange some of the sponge pieces in the bottom. Spoon over 1/4 of the apple mixture, including the juices. Drizzle with 1/4 of the toffee sauce, sprinkle with 1/4 of the nuts, then spoon over 1/4 of the custard and 1/4 of the cream. Continue with the layers, ending with a layer of whipped cream. Place in the fridge and chill at least 1 hour before serving. For decoration, drizzle some toffee sauce over the top of the trifle just before serving, and dust with freshly grated nutmeg.