In a medium heavy-bottomed pan, heat butter, both sugars, milk, and flour over medium heat. Stir constantly until butter is melted and sugar has dissolved. Remove from heat and pour into a large bowl. Let cool 15 minutes.
In a small bowl, whisk egg, egg yolks, vanilla, salt, and nutmeg together.
Slowly whisk about 1 cup of butter mixture into eggs (this prevents them from curdling). Then whisk the egg mixture into the butter mixture.
Pour into pie shell.
Bake 10 minutes and then reduce heat to 325 degrees and bake 35 more minutes.
Notes
Let cool completely before slicing. I typically refrigerate it overnight.