Enthusiasm/enjoyment/passion for the team is really way more important than skill/coordination when holding tryouts for a small college team looking to eventually be competitive. We were in pretty much the same boat you're in now, and spent a long time as a really small team doing an occasional gig and never getting into a competition with people dropping out every semester. Eventually, we decided to (1) value passion/interest in the team way more than skill/coordination, and (2) pretty much accept everyone if they were interested in coming to practices or had an interest in taking a leadership role (no matter how small) in the team.
In terms of valuing passion/interest, we found over time that people who joined the team with some innate skill (usually from some previous dance experience) were usually better at tryouts but there was no way to predict who would plateau, have stylistic issues that were difficult to adjust, or assume they're good enough and stop trying to improve themselves. Similarly, tons of people with zero dance experience showed up to practice more regularly, continued trying to improve themselves outside of practice, and eventually became the better dancers on the team. We thought about valuing their "ability to stay on beat and stay coordinated" during tryouts since we initially assumed that to be something you couldn't really change (i.e. if someone can't jump on beat, they probably will be extremely difficult to standardize to the rest of the beat), but this turned out to be wrong too.
The second idea was more controversial, but it's really important in building a team from the ground up. Including people who are interested in doing auxillary activities for the team (e.g. looking for gigs, getting funding, supporting team morale, whatever) definitely decreases attrition, since it makes the team more exciting to join and shows that it's actually going somewhere in the future. Building a growth mentality as a small college team is the most important thing you can do to make sure your team is sustainable after you leave. We initially had a hard time with people not doing gigs and whatnot, but as we built this part of our team up, the entire vibe on the team drastically changed--people were more excited to dance, and in turn the team became a lot more competitive. I think this strategy of pretty much accepting anyone with enthusiasm for dancing is the only way to rapidly grow a small college team into a competitive team. It's hard to figure out how passionate people will be based on first impressions at workshops/tryouts, and there's usually no true harm in accepting more people. Eventually, the people who aren't interested will just drop out themselves (instead of having you need to cut them at tryouts). Looking back now, most of the people we accepted with this strategy ended up rapidly outpacing the others and are now some of the best dancers. Even if they didn't, they were valuable contributors to the team in other ways. Eventually the team became so big we had to be more selective, and during tryouts roughly prioritized (1) passion/enthusiasm (we would ask about their interest in joining the team), (2) absolute growth from the day of the workshop to the day of the tryout [which was thought to be somewhat of an estimate of how much they were practicing], and (3) baseline skill (in that order).
Ultimately, in people without any experience at all (which is the situation for a lot of small college teams), the ability to pick up a new style of dance within 1-2 weeks of learning it (and while juggling all that orientation/college adjustment stuff) is just not predictive of how good they'll eventually be. We used to think it was a proxy for passion for the team (since in theory someone who practices a ton over two weeks will probably get visibly better), but after seeing tons of people from tryout through senior year, I can definitely say it's not the most important factor. Traditional tryouts favor people who can pick up something quickly, which isn't a good indicator of how much that person will put into the team or the general slope of that person's improvement over time. It works for independent teams when evaluating people with prior dance experience, but making a team from scratch requires more focus on their attitude to build a healthy team morale.
