The Yelp, But for Government Crowd
A fun thing is that basically every single person who has worked in government contracting over the past 15 years has had a variation of one brilliant idea at some point in their career: Yelp, but for government contractors!
What makes the idea so compelling is that the database that would be needed for a Yelp, but for government contractors, already exists. The federal government already uses the Contractor Performance Assessment Reporting System (CPARS) to evaluate how vendors perform on contracts. Every year, the government pumps out millions of ratings that go into CPARS, which seems like enough data to be useful.[1] And, just like Yelp, the CPARS ratings have ratings from 1-5, with narrative comments. Seems sort of like a slam dunk.[2]
There are some problems with this, though. For one, CPARS ratings are super secret. Under the Federal Acquisition Regulation, CPARS evaluations are treated as "Source Selection Information" and therefore cannot be publicly disclosed.[3] And, practically speaking, contractors jealously guard their CPARS ratings from teaming partners and competitors alike.
But this is only a problem from the public's perspective. From the government's perspective, though, CPARS is actually sorta kinda like Yelp. Just that it's available only to federal acquisition professionals. And it's super clunky.
Another important problem, however, is that the government can't just, like, read a CPARS review and say "yeesh, no. that vendor nasty. I guess we'll go to Olive Garden instead."[4] No, this is government contracting. There are rules!
As a recent GAO opinion teaches, the government needs to be thoughtful about what reviews are considered during source selection, and how they are used.
The opinion explains that the Army sought proposals for "logistics support services for the government-owned fixed-wing program office fleets performing transport aircraft missions." As part of source selection, the government noted that it would evaluate vendors' past performance against three factors: recency, relevance, and quality. In turn the "relevance" of an earlier contract would depend on "the dollar value, contract type, number of simultaneously supported locations, degree of subcontracting, and level of complexity" of the contract.
Vertex Aerospace LLC submitted four contracts as part of its past performance. One was thrown out because it wasn't recent enough. But then the Army looked up Vertex in CPARS and "identified an additional Vertex contract that the agency considered somewhat relevant."
Now, if that phrase reads weird to you, you're not alone. As a grammatical construct, I have a hard time with past performance being labeled "somewhat relevant." It feels more natural to think of relevance as a binary choice: a past contract is either relevant or it is not relevant to the current procurement. But government contracting isn't concerned with whether something is natural; instead, the Army said in the solicitation that it would consider multiple factors when evaluating relevance. As such, some contracts will be more relevant and some will be less relevant based on those factors. And, in this case, the CPARS contract was "somewhat relevant."
The government looked at the four contracts (3 provided by Vertex and the CPARS contract), and based on those contracts, gave Vertex's past performance "a rating of satisfactory confidence 'due to two marginal ratings on relevant contracts in past performance.'"
Can you guess the problem? Those two marginal ratings came from the somewhat relevant contract discovered in CPARS! Here's what the GAO said:
[T]he record fails to show that the agency weighed the ratings differently between relevant contracts and somewhat relevant contracts; it instead appears that the agency simply catalogued the number of ratings to arrive at the overall confidence rating. Tallying the ratings without distinguishing the relevancy of the contracts was unreasonable because it elevated low CPARS ratings on less relevant contracts and diminished higher CPARS ratings on more relevant contracts. In effect, the agency equalized every rating, regardless of the relevancy of the contract.
Lesson learned. You can't tally up ratings from somewhat relevant contracts as if they were relevant contracts. Protest sustained. Case closed.
If you're the government, then, this case stands for the proposition that choosing a "close enough" CPARS evaluation can create protest risk. Unless the rating is on point, you may want to think twice.
But I have to say that my inner Yelp, but for Government champion is a bit sad that the story ends there. Because even though the CPARS contract was only somewhat relevant, it probably felt highly salient to the acquisition professionals involved. Two marginal ratings is a pretty big deal! According to publicly available CPARS metrics, something like only 1-2% of all ratings receive a marginal rating! If there are two marginal ratings on a contract, instinctively you might conclude that the contract warrants a closer look.
After all, in Yelp, a 3.5-star rating is pretty boring! Sure, a 5-star rating is notable, but the juicy dirt is usually in the 1-2 stars reviews. Can you blame the government for pulling a negative report and giving it some extra weight?
Yes! Yes you can blame the government for pulling a negative report and giving it some extra weight. Because, unlike Yelp, CPARS doesn't give vendors an overall rating, it just gives contracting officers access to the individual ratings. Which means that contracting officers' selections are super important. This design choice is different from how Yelp and other applications do it. As a recent academic article explains:
When ranking star ratings, most applications will rank a restaurant with ten five-star ratings lower than a restaurant with 48 five-star ratings and two one-star ratings, even though the former has a higher average rating.
Which makes sense! If you just tally up individual ratings and average things out, things get awkward because there will always be a question about whether you have a truly representative sample.
At some point, then, you might think: "Gee! This seems hard. Maybe the government should fix this?!" Perhaps... you might think... the government should try and be more like Yelp after all?
Welcome to the club! Glad to have you here.
[1] Data comes from here. There were 1,427,980 assessments reported in Q4 of FY2023.
[2] I'm going to ignore the point that individual CPARS ratings themselves are not exactly all that useful. Yes, we know CPARS sucks. Stay focused, please.
[3] I've always felt like this is kind of a neat legal trick. Why would an evaluation of how a vendor performed in a contract be considered "source selection information," which only applies to information before an award is made? Ah, because the government can look at past-performance data when making future awards, so past-performance data is transformed into source-selection information. Nifty!
[4] Why the Olive Garden? I was going to suggest the Outback Steakhouse after listening to this Planet Money episode about social mobility, income mixing, and various physical establishments. But then I looked at the underlying research, and learned that Olive Garden is the most significant example of a location that has a "positive impact on cross-class encounters through both scale and their diversity of visitors." Fascinating, yeah? Also, Today I Learned from that research that Waffle House (a full-service restaurant) and KFC (a limited-service restaurant) have different NAICS codes (722511 and 722513, respectively). Is there a NAICS arbitrage opportunity for Waffle House simply by limiting service? Perhaps![*]
[*] Ok, I just have to talk about this. NAICS code 722511 (Full-Service Restaurants) has an annual revenue standard of $11.5 million, whereas NAICS code 722514 (Cafeterias, Grill Buffets, and Buffets) has an annual revenue standard of $34.0 million. Do you currently own a restaurant that is "primarily engaged in providing food services to patrons who order and are served while seated (i.e., waiter/waitress service) and pay after eating?" Are you interested in having additional runway to expand your business while maintaining access to federal small-business set asides? Consider adding some "steam tables, refrigerated areas, display grills, and self-service nonalcoholic beverage dispensing equipment!" Not legal advice. Not business advice. But if someone needs to hear this, please please please let me know.