Parsing rules

Let’s say you want to connect your page to another page on a different website. There’s a problem: that other site doesn’t support HDOCs. Can we work around that? It turns out yes, we can. HDOCs are very simple. They’re basically a title and the content (plus a few optional fields). If you can locate those two things on a page, you can build an HDOC locally even if the original page isn’t an HDOC.

In other words, we need to be able to parse arbitrary web pages. But all client apps must parse them the same way. If they don’t, you’ll end up in a situation where a floating link works in one app and looks broken in another. And you won’t be able to fix it, because the underlying text is slightly different depending on how each client parsed the page. We need deterministic parsing. That’s where parsing rules come in.

Here’s what a URL with parsing rules may look like:

https://example.com/some-page#pr=c/body/t/.main-title/r/.page,.some-class/d/.date/a/.author

Everything after #pr= is the parsing rules section. It’s a set of key–value pairs where each value is a selector. Most of them are optional. The only required one is the content selector. Because of that, for many pages URLs with parsing rules will look much simpler, for example:

https://example.com/some-page#pr=c/body

Here are all the supported selectors:

• c — content selector (required)
• t — title selector (optional if the title is in an <h1> and is unambiguous)
• r — list of selectors to remove from the final content (optional). Multiple selectors are separated by commas. Each selector is URL-encoded individually.
• d — publication date selector (optional)
• a — author name selector (optional)

All selectors must be URL-encoded.

Behind the scenes, a client app simply calls querySelector with the selector you provided. For the “remove” list, it calls querySelectorAll.

How to find selectors using the RW Reader browser extension

RW Reader has a parsing rules editor. I explain how to use it in this video.

Good news

Once you figure out the parsing rules, every client will parse the page identically. Users won’t even know parsing rules exist—everything “just works.”

In the future we may create a public database of parsing rules for different websites which may be integrated with the extension, so you won't have to come up with parsing rules that often, especially for the more popular websites.

And over time—this is the optimistic scenario—HDOCs become common enough that parsing rules aren’t needed as often.

Special case

A special parsing rule exists for plain-text pages:

https://example.com/some-page#pr=text

Here "text" means the page is plain text and should be treated as such. Also, if the parser finds a line starting with Title: ..., it treats it as the page title in this case.