[Suggestion] Add tokens to tree construction tests #100

Activity

1. inikulin commented on Jul 21, 2017

   

   inikulin

   on Jul 21, 2017

   Member

   More actions

   Browsers can benefit from this too, AFAIK most browsers have preparsers that are used for content pre-load. With this addition there will be a convenient way to test it.

   @RReverser The only concern I have is that it will be way to hard to add new tests written by hand. Can we have some tool that generates token list for given markup?
2. RReverser commented on Jul 21, 2017

   

   RReverser

   on Jul 21, 2017

   Last edited by RReverser

   ContributorAuthor

   More actions

   You mean for literally new tests? I guess writing them by hand shouldn't be much worse than writing a tree itself or any other tokenizer tests.
3. gsnedders commented on Jul 21, 2017

   

   gsnedders

   on Jul 21, 2017

   Member

   More actions

   Really if we do this, we could probably eliminate the tokenizer tests (and really, they're not that great: there's no way to run them against some implementations, like browsers, easily, and hence AFAIK no browser vendor runs them). At the same time, we can't just programmatically generate trees using some implementation because for some the interesting part of the test is something the tree constructor will act like it was there anyway (e.g., <p>foo</p> v. <p>foo)… that said, if we just replaced all known tags with <fake-tag> we should still test everything, I think?
4. RReverser commented on Jul 21, 2017

   

   RReverser

   on Jul 21, 2017

   Last edited by RReverser

   ContributorAuthor

   More actions

   At the same time, we can't just programmatically generate trees

   Well we're not talking about trees, but about pure lists of tokens (unless you mean something else), which should still contain only actual tokens from the source, in the same order, with the only difference being that the tokenizer states were adjusted.

   For example, this is how one test generated by mentioned parse5 script currently looks like:

   {
       "description": "<script><div></script></div><title><p></title><p><p>",
       "input": "<script><div></script></div><title><p></title><p><p>",
       "output": [
           [
               "StartTag",
               "script",
               {}
           ],
           [
               "Character",
               "<div>"
           ],
           [
               "EndTag",
               "script"
           ],
           [
               "EndTag",
               "div"
           ],
           [
               "StartTag",
               "title",
               {}
           ],
           [
               "Character",
               "<p>"
           ],
           [
               "EndTag",
               "title"
           ],
           [
               "StartTag",
               "p",
               {}
           ],
           [
               "StartTag",
               "p",
               {}
           ]
       ]
   },
5. RReverser commented on Jul 21, 2017

   

   RReverser

   on Jul 21, 2017

   ContributorAuthor

   More actions

   Really if we do this, we could probably eliminate the tokenizer tests (and really, they're not that great: there's no way to run them against some implementations, like browsers, easily, and hence AFAIK no browser vendor runs them).

   I guess... but for the start, would you accept a PR that adds tokens to tree-construction tests, and then we can discuss tokenizer tests separately?
6. gsnedders commented on Aug 10, 2017

   

   gsnedders

   on Aug 10, 2017

   Last edited by gsnedders

   Member

   More actions

   At the same time, we can't just programmatically generate trees

   Well we're not talking about trees, but about pure lists of tokens (unless you mean something else), which should still contain only actual tokens from the source, in the same order, with the only difference being that the tokenizer states were adjusted.

   I meant for the tokenizer tests.

   I guess... but for the start, would you accept a PR that adds tokens to tree-construction tests, and then we can discuss tokenizer tests separately?

   Yes.