No problem. My experience is that even large company decides to move to an agile (scrum) they are starting with Scrum in small team - as "a proof of concept".
For example, we started as 2 developers and moved to Scrum with more than 20 developers, 4 teams and 3 separated projects.