SysadminNews

Rust Coreutils: a test percentage is not full parity

On this page
  1. Read all four test outcomes
  2. One protection in the old article was not implemented
  3. Test the contracts your scripts depend on

Compatibility improves one behavior at a time. Read the test population and the actual implementation before treating a percentage or an accepted option as a guarantee.

Release-reported GNU test totals: 0.10 has 645 passes, 15 skips, 29 failures and one error out of 690; 0.11 has 653, ten, 22 and zero out of 685. The populations differ, so these are not a controlled same-suite comparison.
Release-reported GNU test totals: 0.10 has 645 passes, 15 skips, 29 failures and one error out of 690; 0.11 has 653, ten, 22 and zero out of 685. The populations differ, so these are not a controlled same-suite comparison. Chart : PeopleAreGeek. Data source.
View full-size image

Read all four test outcomes

The 0.10.0 release notes, published August 5, list 645 passing, 15 skipped, 29 failing and one error out of 690 GNU tests. That is 93.48% passing across the listed total. The diagram retains skipped and error outcomes rather than silently dropping them from the denominator.

Since then, 0.11.0 arrived on August 31. Its table gives 653 passes, ten skips, 22 failures and no errors out of 685, or 95.33%. It also changes the retrospective 0.10 population. Therefore the two headline percentages should not be treated as a controlled measurement on one unchanged suite.

One protection in the old article was not implemented

The 0.10 rm help text explicitly marks --one-file-system as not implemented. In the corresponding source, the flag is accepted and stored, but the consulted implementation does not enforce that filesystem boundary. The previous recommendation to rely on it has been removed.

This is the distinction worth keeping: parsing an option successfully does not prove its promised effect. An automated deployment script can pass its argument checks and still lack the protection the author thought was active. No deletion test was run here.

Test the contracts your scripts depend on

A useful compatibility check records implementation and version separately for each utility, particularly on a system that mixes GNU and uutils binaries. Exercise the specific exit status, standard output, standard error and filesystem result expected by the script, including one relevant failure case.

For a hypothetical parser reading command output, changing diagnostic formatting may matter even if successful operation remains identical. For a deployment script, rename behavior and metadata preservation may matter much more than a hundred unrelated passing cases. The percentage cannot weight those requirements for you.

Likewise, removing known panic paths is valuable without proving that all possible inputs now return the intended error. The updated article preserves the concrete progress while avoiding the old inference that the unpassed fraction corresponds directly to a percentage of all real-world behavior.

September 8: verify GNU test denominators, add 0.11 availability, and remove the false claim that rm --one-file-system is implemented in 0.10.