The Seagulls drop down from the Championship after a home loss to Sheffield Wednesday on Easter Monday, a result that also condemned Millwall and Crewe to relegation.
McGhee insists no-one should be surprised at Brighton's fate, as they are restricted by the tiny capacity of an athletics stadium at Withdean and the opening of a new stadium at Falmer has now been put back to 2009.
He said: "We went out with a bit of a whimper. The players have tried so hard to stay in this division and we've come up a little bit short
"But it's a long-term job here, and I'm building a squad that I intend to still have when we walk out in our new stadium.
"We'll work hard over the summer to run out at the beginning of next season a better team, but everyone knows the club is not in a position to go splashing out on players.
"Young players will improve more playing in the Championship, so it's a step back for us in terms of their development."