In a large frying pan over medium heat melt the butter then saute the apples. Once the apples are a bit soft add the sugar, cinnamon and nutmeg. Continue to cook until apples are tender; 10 to 15 minutes. Allow the mixture to cool.
In a trifle bowl arrange some of the sponge pieces in the bottom. Spoon over 1/4 of the apple mixture including the juices. Drizzle with 1/4 of the toffee sauce, sprinkle with 1/4 of the nuts, then spoon over 1/4 of the custard and 1/4 of the cream. Continue with the layers, ending with a layer of whipped cream.
Place in the fridge and chill at least 1 hour before serving. For decoration, drizzle some toffee sauce over the top of the trifle just before serving and dust with freshly grated nutmeg.