Repository navigation
Add list of parsers that keep up with this testsuite in CI #141
Description
I want to add some tokenizer testcases, and so far have been field-testing them on html5lib-python, but would like to know if there are any parsers out there that would break on those testcases. Particularly because html5lib-python's test harness doesn't seem to compare error locations, and I'm about to add some tests that are all about error locations.
It would be nice to start collecting parsers that use any part of this testsuite in CI, in some sort of document, e.g. README. That would solve my usecase and also make it easier to discover conformant parsers for various languages.
Another idea would be to add all of those parsers to this repo's CI, but that sounds like too much effort
Activity
[-]Add list of parsers that keep up with this in CI[/-][+]Add list of parsers that keep up with this testsuite in CI[/+]untitaker commentedon Jan 16, 2022
https://sourcegraph.com/search?q=context:global+html5lib-tests+file:.gitmodules&patternType=literal
JKingweb commentedon Mar 9, 2023
For what it's worth my parser does test error locations, and includes patches for tokenizer tests because some of the error locations seem wrong but there doesn't seem to be any interest in ironing that out.
I don't use CI, though, so nothing would break immediately. I also don't use submodules, so it's not included in that Sourcegraph search.
flavorjones commentedon Mar 9, 2023
👋 Nokogiri (https://github.com/sparklemotion/nokogiri), a Ruby library that includes HTML5 parsing, uses this test suite. We do regular upstream builds against this repository's
masterbranch, so we would catch any breakage in a day or two.I'd be happy to do the legwork to add Nokogiri's test suite in a "downstream" CI job on this repository if that would be useful.
sideshowbarker commentedon Mar 10, 2023
That sounds great. If you can provide the details about what would be necessary upstream here in this repo, I can work with you to get the necessary changes landed here.
Assuming we’d want to work from a PR: I’m a member of the html5lib org — though not full admin or maintainer perms — and I have perms to approve and merge PRs (write/push perms).
But if any changes to the settings for the repo are needed — in contrast to what we could get done in a PR — then at that point we’d need to have somebody with admin/owner perms make any necessary settings changes.
untitaker commentedon Mar 10, 2023
Is this something we want to constrain to a handful of parsers? I am more than happy to also add a job for my parser html5gum but it pales in relevance compared to nokogiri.
I would add all of those CI jobs as non-required, because parsers can indeed be sometimes broken and updating the testsuite shouldn't be blocked on fixing them. As such I don't think repo settings changes are required.
sideshowbarker commentedon Mar 10, 2023
As far as I understand it, No, we have no reason to constrain it to just a particular/small number of parsers.
I think the relevance shouldn’t be a factor in deciding. It seems to me the only factors in deciding should be: whether the parser project has somebody willing to contribute the code for running CI for their parser — and that CI for their parser is passing rather than failing — and whether they can commit to continuing to keeping that up to date.
But even in the worst case if somebody no longer has time to maintain their CI contribution here — and their CI is failing rather than passing — then we just yank it out here and quit running it.
But in general, in my opinion at least, the more, the merrier (running more parsers through the test suite here under CI is better for html5lib than running fewer).
sideshowbarker commentedon Mar 10, 2023
If you don’t have existing CI set up but could help write something to run under a GitHub Actions workflow here, than that would be great.
untitaker commentedon Mar 10, 2023
I have a PR here to add my parser: #152
for some reason it doesn't show any status
it does run as part of GHA within my own fork: https://github.com/untitaker/html5lib-tests/actions/runs/4386180990/jobs/7679878268
sideshowbarker commentedon Mar 13, 2023
I guess that’s most likely due to some repository setting here. I’ll try to get it figured out.
flavorjones commentedon Mar 13, 2023
@sideshowbarker Github Actions won't run a new workflow unless that workflow file already exists on the default branch.
Let me ship a PR with an skeleton "downstream" workflow file that you can review and merge, and then @untitaker and I (and others) can modify that workflow in subsequent PRs? I think I can do that tonight.
ci: add a skeleton github actions workflow for downstream projects
flavorjones commentedon Mar 14, 2023
I've created #153 as that skeleton workflow file.
ci: add a skeleton github actions workflow for downstream projects
sideshowbarker commentedon Mar 14, 2023
Aha, that makes sense
Super — thanks, now merged
1 remaining item
fb55 commentedon Mar 15, 2023
Awesome idea! I've opened a PR for parse5 at #155.
untitaker commentedon Mar 25, 2023
hello @sideshowbarker, any update on this? I think the three above PRs are pretty much ready to go
flavorjones commentedon May 20, 2023
@untitaker I wanted to check if you were still considering merging the downstream test suites at #152, #154, and #155? Do you have any concerns that we can address?
untitaker commentedon May 20, 2023
untitaker commentedon May 21, 2023
ping @gsnedders @sideshowbarker again, can somebody make a decision on whether we want to merge above PRs for CI?
flavorjones commentedon May 22, 2023
@untitaker Sorry about that -- I meant to ping @sideshowbarker, sorry for the notification noise.
flavorjones commentedon Jun 11, 2023
@sideshowbarker Are you still open merging the downstream test suites at #152, #154, and #155? Do you have any concerns that we can address?
flavorjones commentedon Aug 3, 2023
@sideshowbarker Are you still open to merging #152, #154, and #155? Do you have any concerns with this approach, or those PRs, that we should address?
sideshowbarker commentedon Aug 9, 2023
I have no objections to merging them and I’ve now approved them all — but the problem continues to be that every PR has a stalled “lint Expected — Waiting for status to be reported” check, and I don’t have sufficient perms in this repo to force-merge the PRs without the check being green.
untitaker commentedon Aug 9, 2023
I've merged master into my branch, that should fix it. I think maintainers can merge master into arbitrary PRs to resolve this too
fb55 commentedon Aug 9, 2023
I've updated #155 as well. Happy to see this move forward!
flavorjones commentedon Aug 9, 2023
Thank you!
EmilStenstrom commentedon Dec 7, 2025
Hi! Would you be interested in adding more compliant parsers? JustHTML passes all the tokenizer and treebuilder tests and aims to follow the living spec over time.
untitaker commentedon Dec 7, 2025
I think the purpose of having parsers be part of this CI is to prevent bad tests from making it into main.
To follow this testsuite I recommend adding it as a submodule to your own repo, running it in your own CI and using dependabot to stay uptodate
EmilStenstrom commentedon Dec 7, 2025
@untitaker I have it in my own CI already, thanks.