At the beginning of July, SRF’s capital studio editor interviewed the Militär Aktuell author in Bern about the surprising choice of the F-35 from Lockheed-Martin (Militär Aktuell reported) as the officially most favorable supplier. Journalist Tobias Gasser now presents a more in-depth analysis of the – also surprising – basis for the type selection, which he explained on the radio program “Heute-Morgen” on August 17.
“F-35 as the clear winner …”
… said the head of the DDPS, Viola Amherd, to the media on June 30. The F-35 was the fighter jet with the highest overall benefits and the lowest costs. That is why the American model was chosen. The evaluation had shown that the gap to the runner-up was large. There was no room for foreign or security policy considerations, said Amherd. The Federal Council must therefore adhere to procurement law. At the beginning of 2022, it will discuss the choice of type for the so-called “Army Dispatch 2022” – while the left in the proportional or concentration government will start collecting the 50,000 signatures for a new referendum against the “Ferrari of the skies” from the fall. Militär Aktuell also met with SP MP Priska Seiler-Graf in Bern. The pleasant thing about the neighbor is that even the opponents do not deny the need for a replacement after 30 years and do indeed state that new aircraft are needed. Just not this one, please. What a serious difference to the former “bite reflex” of local comrades. Incidentally, today’s SPÖ defense spokesman regrets the way things were done back then, calls for trainer jets right away and neutrally rejects various LRÜ cooperation ideas.
End of the surprise
But back to the socio-politically more sober Swiss: It is not surprising that in what is certainly a high-class assessment of dozens of capabilities (agility, endurance, armament, sensors, …) of market-available systems, the 5th generation fighter jets (umbrella term “stealth”) beat the 4th generation. Nevertheless, many experts were puzzled as to what neutral Switzerland wanted in the middle of Europe with a clearly offensive device due to its design; there was even talk of a “US network warrior”. The statements that the technical evaluation had shown that the gap between the F-35 and the – still unknown – runner-up was so large and that the F-35 was also clearly the best bidder in terms of lifetime costs were also received with surprise. One possible point of criticism is the way in which the best bidder was determined by the procurement authority armasuisse, as researched by journalist Gasser. For the first time, armasuisse used a new method for the evaluation, which is unknown in the Swiss procurement system – and presumably also internationally. Armasuisse says that the method is not predetermined and that the chosen variant is therefore permissible, while external specialists are now warning of a potentially contestable distortion effect. According to Tobias Gasser, the losing bidders, Airbus, Boeing and Dassault, must not have been aware that they were being assessed using this methodology. New evaluation method AHP
The method is called AHP: “Analytic Hierarchy Process” or hierarchical process analysis. Sounds complicated – and it is. AHP is based on a mathematical process that does not work with a grading system, as is the case with traditional evaluation methods. Instead, it uses comparisons between the individual offers. What does that mean in concrete terms? The four different jets were compared in pairs. They competed against each other – perhaps easier for sports fans to understand – like in a soccer tournament, each against each other. Everyone had three opponents. You could win points in each game. If there was a draw, both teams received one point each. If one team was slightly better, it received two or even three points. This means a doubling or tripling of points compared to the opponent, who only receives one point. Up to nine points per game can be scored in this way if one team is extremely much better than the opposing team. https://militaeraktuell.at/kampfjet-beschaffung-die-schweiz-kauft-den-f-35a/ Most public procurement procedures use a traditional utility value analysis. It works on the principle of a school grade. The authorities and evaluators develop a grading scale in advance for each evaluated payment criterion, which determines which (usually mandatory and target) criteria must be fulfilled in order to receive a certain grade. A good offer receives a five, a very good a six and a satisfactory a four – and then there are gradations in between. The grading scales can vary.
They are defined in advance and – importantly – they are usually known to the providers. However, AHP is used to evaluate the differences, regardless of whether an offer is good, very good or sufficient. How the contrast regulator works in Photoshop
SRF interviewed the German mathematician, business lawyer and procurement expert Thomas Ferber, who is probably one of the few German-speaking experts on this methodology. He explains that the effect of AHP is comparable to the contrast slider in the image editing program Photoshop. In a black and white image with many shades of gray, the contrast is increased until almost only black and white can be seen. “There is greater selectivity,” says Ferber. This contrast effect arises because there are twice or three times as many points for a relatively small difference in quality. In a school grading system, a minimal difference would perhaps only be assessed with half a grade difference.
The armasuisse evaluators around “Project Manager New Fighter Aircraft” Darko Savic confirm this contrast effect. However, they emphasize that they have (therefore) incorporated an additional formula into the calculation, a so-called “transfer function”. This mathematical formula lowers the distribution of points so that there is no contrast effect. The full number of points only comes into play at one and nine. They see many advantages, as the distribution of points according to AHP makes it easier to explain why one candidate scored better than another. And the method provides a system for comparing the candidates. The distribution of points according to AHP makes it easier to determine why one candidate performed better than another.
Bidders could not estimate anything
However, according to SRF/Gasser, the problem with this is that it makes an already mathematically complicated system even more complicated and impenetrable. Thomas Ferber describes the disadvantages of the AHP methodology with a comparison from archery: “In a fair competition, every provider knows where the target circle is. In other words, everyone knows what the grading scale looks like. With AHP, no bidder can correctly estimate where the target circle is because the location depends on the other bids.” For the mathematician and procurement expert, AHP is a sensible methodology when non-measurable, subjective things are being evaluated, such as aesthetics, taste, ideas or concepts. SRF asked various other experts in the procurement sector. Nobody knows AHP. It has probably never been used in a public procurement project in Switzerland. Lawyers specializing in public procurement law who were interviewed were rather sceptical about the methodology. This is because public procurement law stipulates various principles, such as strengthening competition, economic efficiency, transparency and the principle of equal treatment. The principle of equal treatment and transparency in particular could potentially be violated by the use of the AHP methodology, says procurement specialist Claudia Schneider-Heusi, a lawyer in Zurich specializing in public procurement law.
She argues with the lack of a grading system: “If an offer is very good, then it must also receive a very good grade. And not depending on whether there are other very good offers or not. And if two very good offers are close together, then a small difference could carry a lot of weight. That can be tricky and also distort the picture,” she says, probably also with regard to possible objections or challenges. With regard to the principle of transparency, she points out that bidders need to know in advance which award criteria will be used, how they will be weighted and which grading scale will be used to evaluate their bids. Because if only the difference is evaluated, this could lead to a change in the weighting of the award criteria. According to Schneider Heusi, there is also a (further) big question mark as to whether the awarding authority can even justify how the evaluation was arrived at. Armasuisse: Principles adhered to
Armasuisse emphasizes once again that both principles were adhered to. This was confirmed by the in-house lawyers and an external plausibility check by the law firm Homburger. As project manager Darko Savic emphasizes, public procurement law does not stipulate any requirements for the evaluation method. And the principle of equal treatment is fulfilled: “A candidate who is better receives more points. If candidates are similar, they receive a similar number of points.” Lawyer Claudia Schneider-Heusi, on the other hand, remains skeptical. The use of the method is legally tricky, to say the least, she says. And in response to Gasser’s question as to what it means when a lawyer says “tricky”, she replied: “A court could come to the conclusion that the awarding authority did not proceed correctly in the specific case by choosing this method. And then the court would overturn the award decision.” She would therefore also advise against using the AHP method. However, as the SRF man explains, in the case of the fighter jet procurement, no unsuccessful bidder can go to court and complain about the methodology. Appeals are ruled out in this specific case. Hardly any chance of appeal – only because of AHP
What does this mean for the outcome of the fighter jet procurement? Would a conventional method have produced a different winner? Probably not. On the other hand, it could well be that the gaps between the aircraft types and the bids would have been smaller. But these are hypothetical assumptions – there was no second evaluation using a conventional evaluation methodology. Instead of a court, however, it will be up to politicians to decide. As mentioned, the Swiss parliament will discuss the “Army Dispatch 2022” at the beginning of next year. The left (all major cities are green) will oppose it and the question will certainly arise: was the AHP method – which raises legal questions and is difficult to understand – the right tool for evaluating and assessing fighter jets plus equipment plus support over a total service life?
Hurdles for opponents
Social Democrats, Greens and the “notorious” army abolitionists of the GsoA (Group Switzerland without an Army, not in parliament) will certainly also raise this issue in their opposition. They are sure to get 50,000 signatures. Much less likely, on the other hand, are “dissenters” among the parliamentary supporters from the government camp – even if the opponents have been forced to vote for a US aircraft (and also the Patriot from Raytheon for ground air defense – Militär Aktuell reported) are also hoping for possible “America haters” among the far right conservatives in the SVP. The opponents – the majority of whom, as mentioned, are not against new aircraft – also have another steep hurdle to overcome: The majority of the number of cantons – the so-called “Ständemehr” from 1874, which does not count in every type of Swiss referendum, but overrides the “majority of votes” and has never been achieved in referendums on military proposals. For example, in the narrowly positive basic vote a year ago, a majority of 18 cantons voted in favor of the new fighter jets, in some cases with majorities of well over 60 percent.
Militär Aktuell would like to thank Tobias Gasser/SRF for his cooperation.
Click here for more reports on Lockheed-Martin.
