It matters in a sense that smaller servers can sometimes disappear (of course any server can) and some servers don’t allow making new communities and it is good to check which servers it federates with, i.e. which lemmy, kbin, beehaw and mastodon instances it is connected to.

In the end as long as you pick a reasonably popular server it doesn’t matter.