plugin not showing in XNAT

[mitchellxh]

Hi Tim,

I'm trying to get boldqc running on our new XNAT test server. After compiling and copying the jar to /data/xnat/home/plugins and restarting my tomcat service, I'm not seeing boldqc as an installed plugin.

Is the current 1.8 tag compatible with 1.8.9.2?

Strangely, I've turned on logging for the jar files but am not getting error/warning in my catalina.out

/etc/tomcat/logging.properties
> org.apache.jasper.servlet.TldScanner.level = DEBUG
> org.apache.jasper.compiler.TldLocationsCache.level = DEBUG

Thanks,
Mitch

[tokeefe]

Hi Mitch,

If you're referring to why BOLDQC doesn't show up in Administer > Site Administration > Installed Plugins, I can try to explain.

According to this page, only plugins that contain a Java class with the annotation @XnatPlugin will show up in that table. Unfortunately, none of the QC plugins I've written contain any Java classes, so they won't show up in that table. They don't appear within that table in my XNAT (v1.8.9.2) deployment either.

However, the plugin is installed and can be used.

While this level of detail may seem irrelevant to your question, it may help explain what's going on with this jar file. The jar file you installed for BOLDQC adds a data type (a single XML Schema file) and various frontend code and other assets (e.g., Velocity templates, HTML, CSS, JS, images) that are needed to nicely visualize BOLDQC results in XNAT. After you add the jar file to the plugins folder and restart Tomcat, XNAT will find the data type schema and will create the necessary database tables to consume BOLDQC results. You should see evidence of this in your XNAT log files and if you poke around the XNAT database you should see new neuroinfo_boldqc_* tables.

What I haven't quite figured out yet is how to automatically enable the data type. Fortunately, you can do that manually if you navigate to Administer > Data Types > Setup Additional Data Type. You should see the newly added data type neuroinfo:boldqc and you can set it up. Once you do that, you can run the BOLDQC container with --xnat-upload argument and it should all work.

That's the absolute bare minimum you need to get BOLDQC working. That's how I use BOLDQC and all other QC pipelines I've written for XNAT.

P.S. - In case you're wondering, there's also the Administer > Plugin Settings menu in XNAT. If my memory serves me correct, that is for the XNAT Container Service plugin. Once you've successfully installed the XNAT Container Service plugin, you would use that menu to register a command.json file that shows XNAT how to run your pipeline. Unfortunately for me, XNAT runs pipelines using Docker or Docker Swarm and I don't have a Docker Swarm, so I never got a chance to use this feature all that much. Though I was able to get AnatQC working through XNAT Container Service with a lot of help from WUSTL. Since BOLDQC and AnatQC are so similar in design & implementation, this should be close to working out of the box, or at least within striking distance.

Hope this is helpful. Let me know if you have any other questions.

Tim

[mitchellxh]

Beautiful, Tim! That was exactly my question and that answers it.

...navigate to Administer > Data Types > Setup Additional Data Type. You should see the newly added data type neuroinfo:boldqc and you can set it up. Once you do that, you can run the BOLDQC container with --xnat-upload argument and it should all work.

This brings me to the next part of the equation. Like your backend, we do not (cannot, really) have a docker swarm nor Kubernetes server. We will rely on running these containers on our SGE cluster. You mentioned you had a method for scanning XNAT for new scans with the correct data type label, pulling to the cluster, submitting jobs with the boldqc container, then sending the data back. Is this procedure all captured within the boldqc container? As in, it's feasible to run without the container service altogether?

Maybe I'm at a point where I need to just run it and see what happens...

Thanks so much!
-Mitch

[tokeefe]

You mentioned you had a method for scanning XNAT for new scans with the correct data type label, pulling to the cluster, submitting jobs with the boldqc container, then sending the data back. Is this procedure all captured within the boldqc container? As in, it's feasible to run without the container service altogether?

Yes, it's certainly possible and "not too difficult" ™️ to write something that would do a good enough job. I've written quite a few solutions to this problem over the years, but never ended up loving any of them. Not enough to build directly into BOLDQC. For a truly general solution that would work anywhere, there needs to be solutions for some tricky problems e.g., abstracting away the job scheduler. I've taken a stab at that problem before and it was a minefield of edge cases. What I eventually came up with wasn't too bad, but not perfect, and I don't have access to SGE to build in the support for that one.

At any rate, I'm happy to share what I do have. Essentially, this is the idea in a few lines of pseudocode

for session in XNAT(archived recently):
    for scan  in session:
        if scan.type is BOLD and boldqc does not exist:
            submit boldqc for scan

[mitchellxh]

Awesome. This is an excellent jumping-off point for me, thank you so much!

Seems likely I'll be porting your executors to SGE.
-Mitch

[tokeefe]

Happy to help! Let me know if I can help explain anything in BOLDQC or in executors.

Actually, one of the first clusters I ever used was SGE, so I'm somewhat familiar with the tools they give you. You may find executors/lsf/__init__.py to be close to what you will end up with for SGE.