What does the Solid Conformance Test Suite actually test?
Less than the name suggests, and it tells you so itself, which is the best thing about it. We have not run it against our deployed instance, we claim no result, and solid-protocol is not-built on our side.
What it is
A test harness maintained by the Solid community, not by any vendor. It loads a test suite, executes it against a running server, and produces a report.
The report format is EARL, a W3C vocabulary for evaluation results, so an outcome is machine-readable rather than a screenshot of a green tick. That matters more than it sounds: a result anybody can parse is a result anybody can contest.
What it actually checks
Test cases are linked to the specification's own requirements. Each one carries the requirement's level, must, should, may, and a reference to the section it comes from.
So a failure is traceable to a sentence in the specification, not to somebody's opinion about what a pod ought to do. That is the difference between a conformance suite and an integration test.
Coverage is reported alongside the results. The harness computes how much of the specification the suite actually exercises.
The finding: a green run does not mean "implements the specification"
It means the covered subset did not fail.
This is not a criticism of the suite. It is the suite being honest, and it is rarer than it should be, most test suites report only what they ran, leaving you to imagine the rest. A suite that publishes its own coverage is telling you exactly how much of its verdict to believe.
The practical consequence: a vendor's one-line boast about the suite is a weaker sentence than it reads as, and anyone offering it without the coverage figure has given you half a result. If we ever have one, this estate will publish the figure with it.
Why it needs two accounts, and why that is the interesting part
The harness requires two users on the server under test.
Because the questions worth asking are between agents. Can one user's rules actually constrain another? Does a resource one person marked private stay private from someone else? A single-account test can check that a server stores and returns bytes. It cannot check the property the whole model exists for.
That prerequisite is also why self-assertion is worthless here. The operator controls the server, both accounts, and the report. Nothing about that arrangement is adversarial, and a conformance result is only worth what its adversariality is worth.
What our own note already says
Our acceptance criterion, written in our repository before anyone asked us about it: a result here cannot be self-asserted, and the actual bar is interoperation with at least one third-party Solid application.
We are quoting ourselves because it would be easy to quietly weaken that, and publishing it makes that harder. It is unticked.
Where we actually are
Not run. No result. No implied failure.
The gate is unbuilt work, not a closed door: the harness is open, our host is public, and the prerequisites are two accounts and someone's time. Nobody has spent it, including us, and that is the whole status.
And a smaller point worth keeping straight: the engine we run is the community's reference server, so its behaviour under this suite is largely its authors' work rather than ours, what we add is an integration layer. A good result would credit them more than us; a bad one would probably be ours.