Parsing structured data (microdata)

[eposjk]

#16 proposes adding support for JSONLD.

There isn't only JSONLD - structured data can be provided also in the microdata notation, and: good news - there is a project which parses microdata and converts it to the same data structure as JSONLD: https://github.com/yusufkandemir/microdata-parser

So it should be possible to use both and treat it just like an additional JSONLD block!

A first test:

$jsonlddata = \YusufKandemir\MicrodataParser\Microdata::fromHTML($web->client->getResponse()->getContent(), $web->currentUrl())->toArray();

Internally this project uses an own DOM document class derived from DOMDocument. It has a function to import a DOMDocument - but Symphonys response class doesn't allow to access the DOMDocument.

I did a small test, but didn't fiddle it out how to pass the DOMDocument without reparsing - my try which didn't work:

$dom = new \DOMDocument('1.0', 'UTF-8');
$dom->importNode($web->filterFirst('//*')->getNode(0), true);
$jsonlddata = \YusufKandemir\MicrodataParser\Microdata::fromDOMDocument($dom);

What makes sense?
Adding separate PHPScraper functions for JSONLD and microdata? Or mixing both automatically? (my opinion: mixing)

How should support for microdata look like? Adding the other project to PHPScraper? Extending the existing classes or porting the whole functionality to PHPScraper?

[spekulatius]

Hey @eposjk

There isn't only JSONLD - structured data can be provided also in the microdata notation, and: good news - there is a project which parses microdata and converts it to the same data structure as JSONLD: https://github.com/yusufkandemir/microdata-parser

So it should be possible to use both and treat it just like an additional JSONLD block!

The parser looks good. But it supports only ^8.1- that would require either a major release as it blocks 7.4 (will be dropped once it phased out a bit more) and 8.0 (still in use and security-supported). I've had a quick look over the lib and couldn't spot a reason why 8.0 wouldn't work...

Internally this project uses an own DOM document class derived from DOMDocument. It has a function to import a DOMDocument - but Symphonys response class doesn't allow to access the DOMDocument.

Passing the DOMDocument in directly would be preferred for sure. I'll have a look around to see if I can find a way to extract it without re-parsing.

What makes sense?
Adding separate PHPScraper functions for JSONLD and microdata? Or mixing both automatically? (my opinion: mixing)

How should support for microdata look like? Adding the other project to PHPScraper? Extending the existing classes or porting the whole functionality to PHPScraper?

If a native solution is simple enough, I usually add it in directly. Here an external lib might make more sense. I guess the approach depends also the question if we can get the DOMDocument out of the packages used.

Cheers,
Peter

[spekulatius]

I've looked around and getting to a usable and maintainable approach seems to be tricky. My best approach atm would be something similar like the code below. It's placed in GoutteClient.php (extension of \Goutte\Client like in the other PR) with a method replacing createCrawlerFromContent:

protected function createCrawlerFromContent(string $uri, string $content, string $type): ?Crawler
{
    $previously = libxml_use_internal_errors(true);

    $dom = (new HTML5)->loadHTML($content);

    libxml_clear_errors();
    libxml_use_internal_errors($previously);


    // Get the base url.
    $baseElements = $dom->getElementsByTagName('base');

    // Crawler
    $crawler = new Crawler(
        null,
        $uri,
        $baseElements->count() > 0 ? $baseElements[0]->getAttribute('href') : $uri
    );
    $crawler->addDocument($dom);

    return $crawler;
}

But this keeps failing with an error in the base href tests... It might be cleaner to re-parse it after all.

[joshua-bn]

yusufkandemir/microdata-parser#4

This will cause issues if you want to do it with a DOM object.