Preheat oven to 350°F (175°C). Grease a 9x9 inch light metal baking pan and line with parchment paper.
Add the Earl Grey tea and lavender to a food processor and pulse until finely ground. Pass through a fine mesh sieve to remove any large pieces.
In a medium bowl, whisk together the flour, ground tea-lavender mixture, baking powder, baking soda, and salt. Set aside.
In a large bowl, cream the softened butter and granulated sugar together with an electric mixer on high speed for 2 minutes until pale and fluffy.
Add the eggs and vanilla bean paste and mix on medium speed for about 1 minute until pale in color and smooth.
Add the buttermilk and dry ingredients a little at a time, mixing on low speed until all is combined and the batter is smooth. Scrape the sides of the bowl with a rubber spatula as needed.
Pour the batter into the prepared pan. Bake for 38–44 minutes, or until a toothpick inserted in the center comes out clean.
Let the cake cool in the pan on a wire rack for 30 minutes. Use the parchment paper to lift the cake out and let it continue cooling on the rack until completely cooled.