Competency-wise, you want to make sure they can hold the tolerances you need for your product, and that they have the quality assurance testing in place to check that they have made the part to your specifications. Double check that they're comfortable working with your metal of choice (many machinists hate titanium, for example).

Business wise, you want to make sure they both support your minimum expected order and can handle room for growth if it takes off like we hope it will. While not strictly necessary, things usually go better if you feel some rapport with the shop and communicate in similar styles.

I'd be happy to discuss more specifics if talking to a metallurgist sounds useful!