CASSANDRA-19915 Upgrade for the native protocols spec page

[tengu-alt]

This PR enhances the Native Protocols page by replacing the plain text with an interactive, section-based navigation. The implementation leverages cqlprotodoc tool that converts CQL protocol specification files into HTML.

A .sh script was added to download cqlprotodoc, convert the .spec files containing the native protocol specifications into HTML pages, and then merge them into a single page.

After that, native protocols page looks like this:

The navigation is clickable, so you can move to the desired section.

[michaelsembwever]

I'm hesitant to leave this pointing to a living branch. Pinning it to a SHA would be safest.

And am curious if we could upstream it to here… (but this can happen separately…)

@martin-sucha, would you be willing to upstream it (to either cassandra-website or cassandra-builds) ?

[martin-sucha]

Yes, it is better to copy it under github.com/apache (I'll leave up to you to which repo). I'm willing to donate the cqlprotodoc code to the Apache software foundation and change the licence to Apache license, Version 2.0.

[michaelsembwever]

Thanks @martin-sucha. That's all the approval/consent we need.

@tengu-alt , are you willing (and got the time) to include the contents of the repo in the folder cqlprotodoc and adjust accordingly, in this PR ? (don't include its LICENSE file, add to its readme a sentence about where it comes from)

[tengu-alt]

Thanks @martin-sucha. That's all the approval/consent we need.

@tengu-alt , are you willing (and got the time) to include the contents of the repo in the folder cqlprotodoc and adjust accordingly, in this PR ? (don't include its LICENSE file, add to its readme a sentence about where it comes from)

Sure! I will do it.

[michaelsembwever]

This is a solid piece of work, and valuable addition to the website !!

@ossarga , would you be interested in glancing over this in review, as it touches on your past work ?

[michaelsembwever]

LGTM. Thanks again @tengu-alt ! final approving depends on the other reviewers.

[martin-sucha]

Since we moved the cqlprotodoc code here, I wonder if including the content using Javascript is still needed or whether we should modify the template.gohtml / cqlprotodoc output and include the html files directly.

[tengu-alt]

I think it's better to keep things as they are because the script placed in the .adoc template appears more readable to me. Also, we avoid changing the website template by keeping native_protocol.html as the location where the generated pages are merged.

[tengu-alt]

LGTM. Thanks again @tengu-alt ! final approving depends on the other reviewers.

Thanks a lot! Will wait for reviews :)

[martin-sucha]

Thanks! Looks good to me. I'll leave up to you whether to change to directly include the contents instead of using Javascript.

[tengu-alt]

Thanks! Looks good to me. I'll leave up to you whether to change to directly include the contents instead of using Javascript.

Thank you!

[smiklosovic]

going to merge this soon.

[smiklosovic]

@tengu-alt can you please tell me how you succeeded to render this locally? All I am getting is this:

I checked out your branch and did ./run.sh website build and opened this page in the browser:

file:///home/fermat/dev/cassandra/cassandra-website/site-content/build/html/_/native_protocol.html

[smiklosovic]

I have also run ./run.sh website container to build the image (not sure why that command is called "container" though) which adds new shell script but the result is the same.

[tengu-alt]

@smiklosovic you need to build an image from a Dockerfile inside of the site-content directory and provide the tag:
docker build . -t <tag>
And use this image when you run the ./run.sh script
./run.sh website build -c <tag>

[tengu-alt]

I have also run ./run.sh website container to build the image (not sure why that command is called "container" though) which adds new shell script but the result is the same.

@smiklosovic Has my advice helped solve the build issue?

[smiklosovic]

@tengu-alt I did what you suggested, it is still same. After your steps I just do

$ open content/_/native_protocol.html

firefox opens that patch and I just have same results as shown in the comment from (1)

(1) #282 (comment)

[smiklosovic]

@michaelsembwever does this render fine for you based on the steps @tengu-alt gave? I am a little bit out of options here.

[michaelsembwever]

Because of the site-ui/ change in the PR, the way to build this should be

./run.sh website container
./run.sh website build

This is what's documented in this repo's README, and it is how ci-cassandra.apache.org builds the website (so it needs to work this way).
ref: https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/cassandra_job_dsl_seed.groovy#L1401-L1414

This looks to work for me, at least i'm getting the content/_/native_protocol_v*.html pages.
But the content/_/native_protocol.html is still the same as Stefan's screenshot above.
build logs in https://gist.github.com/michaelsembwever/ae8611f4d62e227d02aef9fc89e64657

(It would also be nice to turn off the progress reporting as go downloads…)

[tengu-alt]

@smiklosovic Now I see. There are some issues with iFrame and document origin. I will fix it.

[tengu-alt]

@smiklosovic I've done some research, and it turns out that the browser does not show the content because the page is loaded using the file:// protocol (when you open it via open, google-chrome etc.) which is treated as null origin by the browser, and content is blocked by the CORS policy .
When I open the file by IDE(which serve it over localhost:) it's work just fine.
Could you please try running it on the local HTTP server (e.g. open file thought IDE) and let me know if it resolves the issue?

[michaelsembwever]

is there a way to include the generated native_protocol_v3.html page, instead of using an iframe ?

[tengu-alt]

It can be done using the fetch but it's relying on the same CORS policy.

[michaelsembwever]

can it not be included (somehow) at antora build time ? if possible that would be the best IMO.

[tengu-alt]

Could you please clarify - did you mean the JavaScript contained within the .adoc file?

[michaelsembwever]

no. javascript and iframes are all doing the including at view time.

we should be embedding the html contents into the other file at antora build time.

[michaelsembwever]

this is what i mean: c7b6b60

note, some small things are still needed on top of that^
(e.g. there's links in the headers of each page that don't work, and we need gitignore if we have to put the generated files under site-content)

(feel free to squash that commit into your work here)

[tengu-alt]

Understood. I'll try to move it into the .hbs layout.

[michaelsembwever]

Also… there's another thing we might want cleaned up here.

The native protocol spec files are actually here:
https://github.com/apache/cassandra/tree/trunk/doc

They're the source of truth, and those that will be kept up to date.

IUUC the ones in cassandra-website are unnecessary duplicates: https://github.com/apache/cassandra-website/tree/trunk/site-content/source/modules/ROOT/examples/TEXT

We can assume that these from apache/cassandra:trunk are accurate. And delete the examples/TEXT directory. And then it might be possible then to not need any gitignore addition.

Going further, we could also remove the native_protocol.adoc file from casandra-website altogether, move it to here instead: https://github.com/apache/cassandra/tree/trunk/doc/modules/cassandra/pages

(this would mean running process-native-protocol-specs-in-docker.sh on each generate_cassandra_versioned_docs())

[tengu-alt]

Sounds great! I'll apply it.

[tengu-alt]

@smiklosovic I have to update my solution: for some reason the files that we need for the complete page render are not stored under the site-content directory but under the content(I'm gonna fix it) and you have to look the native_protocol.html under the content dir. (solution about CORS policy and the file:// protocol is actual)

[smiklosovic]

@tengu-alt I would prefer to wait until the complete / working solution is in place, if you do not mind.

[tengu-alt]

@michaelsembwever I have moved the HTML and JavaScript from the native_protocol.adoc file into the new .hbs layout, which is now responsible for it. Additionally, the native_protocol.spec files are currently fetched from the Apache Cassandra GitHub repository.

The issue with the headers arises because the native_protocol_v*.spec files contain typos that cause problems.
(e.g. https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v3.spec#L719-L740)
I can create a pull request for Cassandra to fix these typos.

Could you also clarify why we need to move the native_protocol.adoc file into the Cassandra repository and run the process-native-protocol-specs-in-docker.sh on each generate_cassandra_versioned_docs() execution? As I understand it, the native protocol spec page is identical for all Cassandra versions.

[tengu-alt]

@tengu-alt I would prefer to wait until the complete / working solution is in place, if you do not mind.

The native_protocol_v*.spec files now appears in both content and site-content dirs. (solution about CORS policy and the file:// protocol is actual)

[michaelsembwever]

I can create a pull request for Cassandra to fix these typos.

Please do, thanks!

Could you also clarify why we need to move the native_protocol.adoc file into the Cassandra repository and run the process-native-protocol-specs-in-docker.sh on each generate_cassandra_versioned_docs() execution?

The .spec files are originally found under https://github.com/apache/cassandra/tree/trunk/doc/
They won't be necessarily be identical on each branch. i.e. when v6 is introduced it will only be found initially in the trunk branch.

[tengu-alt]

I can create a pull request for Cassandra to fix these typos.

Please do, thanks!

Done: apache/cassandra#3714

[tengu-alt]

I can create a pull request for Cassandra to fix these typos.

Please do, thanks!

Could you also clarify why we need to move the native_protocol.adoc file into the Cassandra repository and run the process-native-protocol-specs-in-docker.sh on each generate_cassandra_versioned_docs() execution?

The .spec files are originally found under https://github.com/apache/cassandra/tree/trunk/doc/ They won't be necessarily be identical on each branch. i.e. when v6 is introduced it will only be found initially in the trunk branch.

Understood, working on it.

[michaelsembwever]

The .spec files are originally found under https://github.com/apache/cassandra/tree/trunk/doc/ They won't be necessarily be identical on each branch. i.e. when v6 is introduced it will only be found initially in the trunk branch.

I've taken your latest work @tengu-alt and working it a bit.
Expect the patch-on-patch from me either in Dec' or in the new year.

[tengu-alt]

The .spec files are originally found under https://github.com/apache/cassandra/tree/trunk/doc/ They won't be necessarily be identical on each branch. i.e. when v6 is introduced it will only be found initially in the trunk branch.

I've taken your latest work @tengu-alt and working it a bit. Expect the patch-on-patch from me either in Dec' or in the new year.

I am currently Implementing the protocol pages generation on each version(Maybe it will not be done exactly inside the generate_cassandra_versioned_docs ).
Thanks for the help! What kind of patch do I need to expect?

[michaelsembwever]

What kind of patch do I need to expect?

Ah, i misunderstood you – didn't realise you were still in-progress…
this is the work-in-progress
tengu-alt@c06aa65
it's still a bit messy, but maybe you get the drift.

[tengu-alt]

What kind of patch do I need to expect?

Ah, i misunderstood you – didn't realise you were still in-progress… this is the work-in-progress tengu-alt@c06aa65 it's still a bit messy, but maybe you get the drift.

The process-native-protocol-specs-in-docker.sh will look differently, and will be called not from the generate_cassandra_versioned_docs because the script needs to be run after the render_site_content_to_html, which cleans all generated files.
It takes the .spec files from Cassandra/doc, and site-content/build/html/Cassandra/$DOC_VERSION/cassandra/native-protocol is the output dir(I've created the native-protocol dir in the Cassandra repo, so we can have different native protocol doc pages for the different Cassandra versions
Let's apply your changes from work-in-progress (e.g. Dockerfile changes) after my next patch if you don't mind.

[michaelsembwever]

Let's apply your changes from work-in-progress (e.g. Dockerfile changes) after my next patch if you don't mind.

Go for it. It's only for reference sake, and might not be used at all depending on what you land 👍

[tengu-alt]

@michaelsembwever I have done some refactoring on the process-native-protocol-specs-in-docker.sh.
From now, the script is called only if generate docs command for build is running (also implemented it for preview mode, but it throws some errors when generate docs is running. I've also tested it on the master branch, and it's the same).
I also moved the native_protocol.adoc file to the Cassandra Doc under the native-protocol directory, and antora generates a page using the .hbs layout from the website. The native_protocol.adoc needs to be in the Cassandra repo.

The generate_native_protocol_specs_pages() now iterates over specified Cassandra versions and calls the script if the provided version of Cassandra contains the native-protocol dir under the doc.
After that generated pages are put into the same versioned doc directory under the site-content and copied into the content.

The generate_native_protocol_specs_pages() can't be called with thegenerate_cassandra_versioned_docs() because the script needs to be run after the render_site_content_to_html because antora removes the generated files.

(I also added the license part inside script)