In a large mixing bowl, whisk together the gluten free flour, xanthan gum (if using), baking powder, salt, cinnamon, nutmeg, and brown sugar until well combined.
Add the milk, oil, eggs (or aquafaba), and vanilla extract. Whisk until you have a thick, smooth batter with no lumps.
Stir in the diced apples until evenly distributed throughout the batter.
Pour frying oil into a large pot or skillet until it is 1.5 to 2 inches deep. Heat over medium-high until the oil reaches 360-375°F (180–190°C). Use a thermometer for accuracy.
Using a large spoon, carefully drop scoops of batter (1-2 tablespoons each) into the hot oil, 3 to 4 fritters at a time. Do not overcrowd the pot.
Fry for about 3 minutes on the first side, then flip and fry for another 2–3 minutes, until deeply golden brown on both sides. To check doneness, use metal tongs to gently squeeze one open - the inside should look fluffy and cooked through, not wet or doughy.
Remove from oil using a slotted spoon or metal tongs and transfer to a wire cooling rack. Repeat with remaining batter, allowing oil to return to temperature between batches.
While fritters cool, make the glaze: whisk together powdered sugar, milk, and vanilla extract until smooth. Add more sugar for a thicker glaze or more milk for a thinner glaze.
While fritters are still slightly warm, dip each one into the glaze and return to the wire rack. Let the excess glaze drip off and allow to set for a few minutes before serving.