rss.xml

<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">
<channel>
        <title>DINA-Web Blog</title>
        <description>DINA-Web Blog - DINA-Web</description>
        <link>https://blog.dina-web.net</link>
        <link>https://blog.dina-web.net</link>
        <lastBuildDate>2016-06-13T10:59:11+02:00</lastBuildDate>
        <pubDate>2016-06-13T10:59:11+02:00</pubDate>
        <ttl>1800</ttl>


        <item>
                <title>Using docker with DINA-Web repos (dw-*)</title>
                <description>&lt;h2 id=&quot;preparations&quot;&gt;Preparations&lt;/h2&gt;

&lt;p&gt;Note: If you are on Linux, you can skip the first three steps, since you can run docker directly instead of inside a Virtual Machine&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Install VirtualBox on your laptop to get a contained environment for the workshops hackathon materials&lt;/li&gt;
  &lt;li&gt;Inside the VirtualBox, install Linux Mint (http://distrowatch.com/table.php?distribution=mint), give it say 4 G RAM and 20 G disk&lt;/li&gt;
  &lt;li&gt;Now log in to this VM and complete the rest of the preparations below inside there.&lt;/li&gt;
  &lt;li&gt;Inside this VM, install docker using instructions at https://docs.docker.com/linux/step_one/ - follow instructions all the way up to step 4 (at least)&lt;/li&gt;
  &lt;li&gt;Install docker-compose, start at https://docs.docker.com/engine/installation/linux/ubuntulinux/ (Mint is using ubuntu packages)&lt;/li&gt;
  &lt;li&gt;Install make (sudo apt-get install make) and git (sudo apt-get install git)&lt;/li&gt;
  &lt;li&gt;Create a “repos” directory and use git clone (for example git clone https://github.com/Inkimar/dw-media.git) to get the reference materials for various modules onto the laptop so you can run them there…
    &lt;ul&gt;
      &lt;li&gt;dw-collections &lt;a href=&quot;https://github.com/DINA-Web/dw-collections&quot;&gt;https://github.com/DINA-Web/dw-collections&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;dw-collections-ui &lt;a href=&quot;https://github.com/DINA-Web/dw-collections-ui&quot;&gt;https://github.com/DINA-Web/dw-collections-ui&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;dw-media &lt;a href=&quot;https://github.com/DINA-Web/dw-media&quot;&gt;https://github.com/DINA-Web/dw-media&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;dw-seqdb &lt;a href=&quot;https://github.com/DINA-Web/dw-seqdb&quot;&gt;https://github.com/DINA-Web/dw-seqdb&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;dw-classifications &lt;a href=&quot;https://github.com/DINA-Web/dw-classifications&quot;&gt;https://github.com/DINA-Web/dw-classifications&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;dw-cli-tools &lt;a href=&quot;https://github.com/DINA-Web/dw-cli-tools&quot;&gt;https://github.com/DINA-Web/dw-cli-tools&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;dw-system &lt;a href=&quot;https://github.com/DINA-Web/dw-system&quot;&gt;https://github.com/DINA-Web/dw-system&lt;/a&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;In most cases, you should be able to start the apps locally now, by doing “cd dw-media” and “make” (in the case where there are Makefiles)&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you can run the apps, you have everything in order, if not, please troubleshoot until you have it working.&lt;/p&gt;

&lt;h3 id=&quot;good-tutorial-providing-sample-docker-application&quot;&gt;Good Tutorial providing sample Docker application&lt;/h3&gt;

&lt;p&gt;For learning more about docker and deploying distributed polyglot apps (ie apps that combine various technologies like JavaScript, Java and Python apps in one system, much like DINA-Web does), please check out this official Docker tutorial: &lt;a href=&quot;https://github.com/docker/docker-birthday-3&quot;&gt;https://github.com/docker/docker-birthday-3&lt;/a&gt; It is good to have had a look at this material.&lt;/p&gt;

&lt;h3 id=&quot;advanced-docker-topics&quot;&gt;Advanced Docker topics&lt;/h3&gt;

&lt;p&gt;More advanced training materials are available from Docker’s own training guru at: &lt;a href=&quot;http://view.dckr.info&quot;&gt;http://view.dckr.info&lt;/a&gt;. It contains useful information about “Docker Orchestration” and tools such as the &lt;code&gt;build-tag-push.py&lt;/code&gt; command-line tool at &lt;a href=&quot;https://github.com/jpetazzo/orchestration-workshop/blob/master/bin/build-tag-push.py&quot;&gt;https://github.com/jpetazzo/orchestration-workshop/blob/master/bin/build-tag-push.py&lt;/a&gt;.&lt;/p&gt;

&lt;h3 id=&quot;think-about-challenges&quot;&gt;Think about challenges&lt;/h3&gt;

&lt;p&gt;Please also think about functionality that might be good to break out and isolate from existing applications or systems in order to be able to work with smaller, independent units with the intent to facilitate reuse, portability and ability to work in parallell.&lt;/p&gt;

&lt;p&gt;An example:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;How can the seqdb or the taxonomy module support SSO and user permissions stored in KeyCloak?&lt;/li&gt;
&lt;/ul&gt;
</description>
                <link>https://blog.dina-web.net/2016/06/docker-with-dinaweb-repos.html</link>
                <guid>https://blog.dina-web.net/2016/06/docker-with-dinaweb-repos</guid>
                <pubDate>2016-06-13T00:00:00+02:00</pubDate>
        </item>

        <item>
                <title>Agenda for Berlin 2016 DINA-Web workshop</title>
                <description>&lt;p&gt;Here is the Agenda in Google Calendar with the topics and session for the three days:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://calendar.google.com/calendar/embed?src=k8dogtsts5nhn2m04o95mspusc%40group.calendar.google.com&amp;amp;ctz=Europe/Stockholm&quot;&gt;Google Calendar for Berlin 2016 workshop&lt;/a&gt;.&lt;/p&gt;

</description>
                <link>https://blog.dina-web.net/2016/06/agenda-berlin.html</link>
                <guid>https://blog.dina-web.net/2016/06/agenda-berlin</guid>
                <pubDate>2016-06-12T00:00:00+02:00</pubDate>
        </item>

        <item>
                <title>Improving the guidelines</title>
                <description>&lt;p&gt;Here are the DINA-Web collaboration guidelines:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;https://github.com/DINA-Web/dina-api-standard/blob/master/DINA-Web-API-Guidelines.md&quot;&gt;Functionality/API guidelines&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;/guidelines/style.html&quot;&gt;Style guidelines&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;/guidelines/licensing.html&quot;&gt;Licensing guidelines&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;/guidelines/releases.html&quot;&gt;Release guidelines&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;/guidelines/qa.html&quot;&gt;Quality Assurance guidelines&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;discussion-topics&quot;&gt;Discussion topics&lt;/h1&gt;

&lt;p&gt;Suggested areas to improve across the guidelines:&lt;/p&gt;

&lt;h2 id=&quot;api-guidelines&quot;&gt;API guidelines&lt;/h2&gt;

&lt;p&gt;Suggest change format to use API Blueprint format? This is also markdown but what we suggest for API docs.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;https://github.com/apiaryio/api-blueprint/tree/master/examples&lt;/li&gt;
  &lt;li&gt;use and suggest using aglio or equiv to render to .html&lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;see renderers here for tools (md -&amp;gt; html): https://apiblueprint.org/tools.html&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;Paging support&lt;/li&gt;
  &lt;li&gt;Authentication flow&lt;/li&gt;
  &lt;li&gt;Batch requests&lt;/li&gt;
  &lt;li&gt;JSONAPI format&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;style-guidelines&quot;&gt;Style guidelines&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;Provide repo w graphical assets with DINA-Web Collections Manager stylesheets etc&lt;/li&gt;
  &lt;li&gt;More ideas?&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;licensing-guidelines&quot;&gt;Licensing guidelines&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;Any issues? Or do you comply :)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;release-and-qa-guidelines&quot;&gt;Release and QA guidelines&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;Need updated tool recommendations&lt;/li&gt;
  &lt;li&gt;Does these guidelines lack anything?&lt;/li&gt;
  &lt;li&gt;Do you use them?&lt;/li&gt;
&lt;/ul&gt;

</description>
                <link>https://blog.dina-web.net/2016/06/latest-guidelines.html</link>
                <guid>https://blog.dina-web.net/2016/06/latest-guidelines</guid>
                <pubDate>2016-06-11T00:00:00+02:00</pubDate>
        </item>

        <item>
                <title>State of the Union for DINA-Web in 2016</title>
                <description>&lt;p&gt;Here is an &lt;a href=&quot;/progress/&quot;&gt;updated state of the union&lt;/a&gt; overview for DINA-Web in 2016. It includes updates made over the last six months.&lt;/p&gt;
</description>
                <link>https://blog.dina-web.net/2016/06/state-of-the-union-2016.html</link>
                <guid>https://blog.dina-web.net/2016/06/state-of-the-union-2016</guid>
                <pubDate>2016-06-10T00:00:00+02:00</pubDate>
        </item>

        <item>
                <title>Mattermost - new web-based chat channel</title>
                <description>&lt;p&gt;Now the &lt;a href=&quot;https://github.com/dina-web/dw-chat&quot;&gt;DINA-Web Chat&lt;/a&gt; is deployed and running at &lt;a href=&quot;https://chat.dina-web.net&quot;&gt;https://chat.dina-web.net&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Some of us have asked for a channel like this because it is less formal than e-mail and less techy than irc and can also be integrated with GitHub (not done yet).&lt;/p&gt;

&lt;p&gt;You will be invited to join through e-mail, soon.&lt;/p&gt;

&lt;p&gt;Please sign up and chat online!&lt;/p&gt;
</description>
                <link>https://blog.dina-web.net/2016/06/dw-chat.html</link>
                <guid>https://blog.dina-web.net/2016/06/dw-chat</guid>
                <pubDate>2016-06-09T00:00:00+02:00</pubDate>
        </item>

        <item>
                <title>Current work</title>
                <description>&lt;h2 id=&quot;devops-efforts&quot;&gt;DevOps Efforts&lt;/h2&gt;

&lt;p&gt;Development, system integration and packaging&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Establishing REST APIs for all modules in DINA-Web
    &lt;ul&gt;
      &lt;li&gt;There is a &lt;a href=&quot;https://github.com/DINA-Web/information-model/blob/master/blueprints.Rmd&quot;&gt;tutorial&lt;/a&gt; and first blueprint for the simplistic Species Information service&lt;/li&gt;
      &lt;li&gt;This should serve as guide towards getting the rest of the APIs in place (Media, Collections, Classifications, etc etc)&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Information model &amp;amp; new data model
    &lt;ul&gt;
      &lt;li&gt;Proof of Concept for &lt;a href=&quot;https://github.com/chicoreus/cco_poc&quot;&gt;Complex Natural History Museum Collections Objects&lt;/a&gt; - the next evolutionary step for the Collections information model&lt;/li&gt;
      &lt;li&gt;Liquibase for database agnostic schemas, need small &lt;a href=&quot;https://github.com/inkimar/liquibase-example&quot;&gt;tutorial&lt;/a&gt;, need to answer questions in the read me document&lt;/li&gt;
      &lt;li&gt;Dockerize and make db engine agnostic with liquibase  - http://www.liquibase.org/&lt;/li&gt;
      &lt;li&gt;Referential use case with UML from Glen to address Thomas concerns with micro-service architecture and Web API strategy&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Systems integration effort using micro-services architecture toolchains
    &lt;ul&gt;
      &lt;li&gt;Use “Boxes within boxes” with Docker and “docker-compose” to assemble higher level systems of interrelated containers&lt;/li&gt;
      &lt;li&gt;List tasks for system integration tasks aka [“DevOps Pinatas”]&lt;/li&gt;
      &lt;li&gt;Require from modules to show integration with KeyCloak for SSO and Users/Permissions mgm&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;DINA-Compliance (security, accessibility, testing and validation)
    &lt;ul&gt;
      &lt;li&gt;Glen: Additional plugin for validators. From Ingimar for Jenkins. Suggested Unicorn – unified validator. We should consider this, maintaining good practice. Not really optional.
        &lt;ul&gt;
          &lt;li&gt;W3C  HTML/CSS unified validator https://validator.w3.org/unicorn/&lt;/li&gt;
          &lt;li&gt;Jenkins Unicorn validator https://wiki.jenkins-ci.org/display/JENKINS/Unicorn+Validation+Plugin&lt;/li&gt;
          &lt;li&gt;From David: https://validator.w3.org/docs/api.html&lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;collaboration-efforts&quot;&gt;Collaboration efforts&lt;/h2&gt;

&lt;p&gt;Meetings and Communication infrastructure&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Get additional DINA-Web internal communications infrastructure in place
    &lt;ul&gt;
      &lt;li&gt;&lt;a href=&quot;https://drive.google.com/folderview?id=0B1zwfOKfRazISndDZ0NhXzlBY1U&amp;amp;usp=sharing&amp;amp;invite=CK7qyns&amp;amp;ts=5637c782&quot;&gt;Google Drive for working documents&lt;/a&gt; where &lt;a href=&quot;https://github.com/DINA-Web&quot;&gt;GitHub&lt;/a&gt; doesn’t work (non-developers feel more at home on Google Drive?)&lt;/li&gt;
      &lt;li&gt;Redmine for external tickets/issues (and project management) where &lt;a href=&quot;https://github.com/blog/831-issues-2-0-the-next-generation&quot;&gt;GitHub Issues&lt;/a&gt; doesn’t suffice&lt;/li&gt;
      &lt;li&gt;IRC chat at freenode (#dinaweb)&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;blog.dina-web.net&quot;&gt;Developer blog&lt;/a&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Next DINA physical meeting Berlin
    &lt;ul&gt;
      &lt;li&gt;Doodle on potential weeks for next meeting at Berlin in June.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

</description>
                <link>https://blog.dina-web.net/2015/11/efforts.html</link>
                <guid>https://blog.dina-web.net/2015/11/efforts</guid>
                <pubDate>2015-11-30T00:00:00+01:00</pubDate>
        </item>

        <item>
                <title>DINA-Web DevOps Piñatas</title>
                <description>&lt;p&gt;Here we list DevOps Piñatas for DINA-Web. These challenges involves creating “micro service containers” for various modules in the DINA-Web system. So this is about building standalone boxes using tools such as Vagrant and “boxes” or modules that can be combined together into one system using tools as Docker and &lt;code&gt;docker-compose&lt;/code&gt;. This list is ordered by (current) priority:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/figure/source/2015-11-18-pinatas/unnamed-chunk-2-1.png&quot; alt=&quot;plot of chunk unnamed-chunk-2&quot; /&gt;&lt;/p&gt;

&lt;h1 id=&quot;the-dw-taxonomy-box&quot;&gt;The “dw-taxonomy” box&lt;/h1&gt;

&lt;p&gt;At &lt;a href=&quot;https://github.com/dina-web/dw-taxonomy&quot;&gt;https://github.com/dina-web/dw-taxonomy&lt;/a&gt; there is a repository with an integration project that brings up a stand alone virtual box with the module from &lt;a href=&quot;https://github.com/TU-NHM/plutof-taxonomy-module&quot;&gt;https://github.com/TU-NHM/plutof-taxonomy-module&lt;/a&gt; which supports working with classifications. To do the above one needs to have &lt;code&gt;virtualbox&lt;/code&gt; and &lt;code&gt;vagrant&lt;/code&gt; installed and then do a &lt;code&gt;git clone&lt;/code&gt; to get the module locally, then issue &lt;code&gt;vagrant up&lt;/code&gt; to start the box.&lt;/p&gt;

&lt;h3 id=&quot;tldr-summary&quot;&gt;TLDR summary&lt;/h3&gt;

&lt;p&gt;Make the Vagrant box run without errors and dockerize this box.&lt;/p&gt;

&lt;h3 id=&quot;todo&quot;&gt;TODO&lt;/h3&gt;

&lt;p&gt;A lot of the work is done, the django dev server starts up and runs but this remains before this piñata container is successfully assembled:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;fix any errors in the current &lt;a href=&quot;https://github.com/DINA-Web/dw-taxonomy/blob/master/bootstrap.sh&quot;&gt;bootstrap script&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;fix any errors in the module [setup script] (https://github.com/DINA-Web/dw-taxonomy/blob/master/setup-taxonomy-drf.sh)&lt;/li&gt;
  &lt;li&gt;integrate the web server to properly serve stylesheets for the service&lt;/li&gt;
  &lt;li&gt;provide instructions to load data&lt;/li&gt;
  &lt;li&gt;TODO: Ingimar may have more or have fixed some issues&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For integrating the web server - the configuration files for the apache web server and the integration between apache and django needs to be tweaked.&lt;/p&gt;

&lt;p&gt;For providing instructions to load data, another section in the README.md file needs to be added with step-by-step instructions on how to do this.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/figure/source/2015-11-18-pinatas/unnamed-chunk-3-1.png&quot; alt=&quot;plot of chunk unnamed-chunk-3&quot; /&gt;&lt;/p&gt;

&lt;h1 id=&quot;the-dw-classifications-box&quot;&gt;The “dw-classifications” box&lt;/h1&gt;

&lt;p&gt;Packaging the PlutoF Classifications module using Docker - an idea could be to use something like Bitnami Stacksmith to get started, for example: &lt;a href=&quot;https://stacksmith.bitnami.com/api/v1/stacks/k81Sftw.dockerfile?api_key=619b5741441b28e37269f8ba23297113416dbbd8d9afaedfb85af329ac4ceb57&quot;&gt;Python and Django&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/figure/source/2015-11-18-pinatas/unnamed-chunk-4-1.png&quot; title=&quot;plot of chunk unnamed-chunk-4&quot; alt=&quot;plot of chunk unnamed-chunk-4&quot; style=&quot;display: block; margin: auto auto auto 0;&quot; /&gt;&lt;/p&gt;

&lt;h1 id=&quot;the-dw-media-box&quot;&gt;The “dw-media” box&lt;/h1&gt;

&lt;p&gt;At &lt;a href=&quot;https://github.com/dina-web/dw-media&quot;&gt;https://github.com/dina-web/dw-media&lt;/a&gt; there is a repository with an integration project that brings up a stand alone virtual box with the module. These are issues being worked upon currently:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;TODO: ask Ingimar what remains&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This box packaged using docker - again being inspired by by Bitnami Stacksmith, something like this could be used &lt;a href=&quot;https://stacksmith.bitnami.com/api/v1/stacks/2gCNfIY.dockerfile?api_key=619b5741441b28e37269f8ba23297113416dbbd8d9afaedfb85af329ac4ceb57&quot;&gt;Java application server&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;An idea … perhaps it is even better to use docker-compose and combine these in a &lt;code&gt;docker-compose.yml&lt;/code&gt; file…&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Media server container built from Dockerfile extending App server: https://hub.docker.com/r/jboss/wildfly/&lt;/li&gt;
  &lt;li&gt;SSO: https://hub.docker.com/r/jboss/keycloak/&lt;/li&gt;
  &lt;li&gt;SSO backend integration: https://hub.docker.com/r/jboss/keycloak-mysql/&lt;/li&gt;
  &lt;li&gt;Tutorial: https://hub.docker.com/r/jboss/keycloak-examples/&lt;/li&gt;
  &lt;li&gt;Backend: https://hub.docker.com/_/postgres/&lt;/li&gt;
  &lt;li&gt;Backend: https://hub.docker.com/_/mysql/&lt;/li&gt;
  &lt;li&gt;Shared files: https://hub.docker.com/r/tianon/true/&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;img src=&quot;/figure/source/2015-11-18-pinatas/unnamed-chunk-5-1.png&quot; title=&quot;plot of chunk unnamed-chunk-5&quot; alt=&quot;plot of chunk unnamed-chunk-5&quot; style=&quot;display: block; margin: auto auto auto 0;&quot; /&gt;&lt;/p&gt;

&lt;h1 id=&quot;the-dw-collections-box&quot;&gt;The “dw-collections” box&lt;/h1&gt;

&lt;p&gt;This box should contain the data itself and REST APIs that should be used by applications.&lt;/p&gt;

&lt;h3 id=&quot;todo-1&quot;&gt;TODO&lt;/h3&gt;

&lt;ul&gt;
  &lt;li&gt;Push DINA-Specify-API-v0 to github/dina-web&lt;/li&gt;
  &lt;li&gt;Assemble a docker-compose.yml that uses mysql with the data at https://github.com/DINA-Web/datasets&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Reuse these services…&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;App server exposing REST APIs: https://hub.docker.com/r/jboss/wildfly/&lt;/li&gt;
  &lt;li&gt;SSO: https://hub.docker.com/r/jboss/keycloak/&lt;/li&gt;
  &lt;li&gt;SSO backend integration: https://hub.docker.com/r/jboss/keycloak-mysql/&lt;/li&gt;
  &lt;li&gt;Tutorial: https://hub.docker.com/r/jboss/keycloak-examples/&lt;/li&gt;
  &lt;li&gt;Backend: https://hub.docker.com/_/mysql/&lt;/li&gt;
  &lt;li&gt;Backend: https://hub.docker.com/r/tianon/true/&lt;/li&gt;
  &lt;li&gt;Search: https://hub.docker.com/_/solr/&lt;/li&gt;
  &lt;li&gt;Web proxy: https://hub.docker.com/_/nginx/&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In addition, a custom Dockerfile would be built which would provide the DINA REST APIs?&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/figure/source/2015-11-18-pinatas/unnamed-chunk-6-1.png&quot; title=&quot;plot of chunk unnamed-chunk-6&quot; alt=&quot;plot of chunk unnamed-chunk-6&quot; style=&quot;display: block; margin: auto auto auto 0;&quot; /&gt;&lt;/p&gt;

&lt;h1 id=&quot;the-dw-build-box&quot;&gt;The “dw-build” box&lt;/h1&gt;

&lt;p&gt;A docker image which builds the DINA-Web system modules from sources and moves the sources and build artifacts to a backup server?&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/figure/source/2015-11-18-pinatas/unnamed-chunk-7-1.png&quot; title=&quot;plot of chunk unnamed-chunk-7&quot; alt=&quot;plot of chunk unnamed-chunk-7&quot; style=&quot;display: block; margin: auto auto auto 0;&quot; /&gt;&lt;/p&gt;

&lt;h1 id=&quot;the-dw-seqdb-box&quot;&gt;The “dw-seqdb” box&lt;/h1&gt;

&lt;p&gt;This is done for Vagrant. But is it dockerized yet? And does it run with Wildfly?&lt;/p&gt;

&lt;p&gt;Would be nice to use docker-compose and runs with these… (note that some of these are different from the current stack used in seqdb, but it might be possible to easily run seqdb also on Widlfly as long as the JNDI-issue (?) can be resolved…. )&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;App server: https://hub.docker.com/r/jboss/wildfly/&lt;/li&gt;
  &lt;li&gt;SSO: https://hub.docker.com/r/jboss/keycloak/&lt;/li&gt;
  &lt;li&gt;SSO backend integration: https://hub.docker.com/r/jboss/keycloak-mysql/&lt;/li&gt;
  &lt;li&gt;Tutorial: https://hub.docker.com/r/jboss/keycloak-examples/&lt;/li&gt;
  &lt;li&gt;Backend: https://hub.docker.com/_/mysql/&lt;/li&gt;
  &lt;li&gt;Backend: https://hub.docker.com/r/tianon/true/&lt;/li&gt;
  &lt;li&gt;Search: https://hub.docker.com/_/solr/&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;img src=&quot;/figure/source/2015-11-18-pinatas/unnamed-chunk-8-1.png&quot; title=&quot;plot of chunk unnamed-chunk-8&quot; alt=&quot;plot of chunk unnamed-chunk-8&quot; style=&quot;display: block; margin: auto auto auto 0;&quot; /&gt;&lt;/p&gt;

&lt;h1 id=&quot;the-dw-ddt-box&quot;&gt;The “dw-ddt” box&lt;/h1&gt;

&lt;p&gt;Currently this is a Vagrant project that uses the &lt;a href=&quot;https://bitnami.com/stack/mean&quot;&gt;Bitnami Mean stack&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;An embryo of this can be found here: [](https://github.com/DINA-Web/dw-ddt), suggested inspiration here for dockerizing:
[](https://github.com/bitnami/bitnami-docker/blob/master/tutorials/ghost.md)&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/figure/source/2015-11-18-pinatas/unnamed-chunk-9-1.png&quot; title=&quot;plot of chunk unnamed-chunk-9&quot; alt=&quot;plot of chunk unnamed-chunk-9&quot; style=&quot;display: block; margin: auto auto auto 0;&quot; /&gt;&lt;/p&gt;

&lt;h1 id=&quot;the-dw-rprtr-box&quot;&gt;The “dw-rprtr” box&lt;/h1&gt;

&lt;p&gt;Create a minimalist external reporting system as an example of a set of containers running together as a system, using &lt;code&gt;docker-compose&lt;/code&gt;, for tutorial and learning purposes.&lt;/p&gt;

&lt;p&gt;Ideally this box would access the DINA-Web API through keycloak for getting data using OAuth (once the dw-backend becomes available).&lt;/p&gt;

&lt;p&gt;Initially, it should provide:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;An authoring platform capable of markdown and data wrangling with the latest tools, web based UI: https://hub.docker.com/r/rocker/ropensci/&lt;/li&gt;
  &lt;li&gt;An application server for interactive web application which can display dynamic reports: https://hub.docker.com/r/rocker/shiny/&lt;/li&gt;
  &lt;li&gt;A web proxy to provide these services to the web, either as virtual hosts (access from sub-domains) or as prefix-applications&lt;/li&gt;
  &lt;li&gt;A shared storage container, such as https://hub.docker.com/r/tianon/true/&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Later it should provide R packages and Python packages that support various data wrangling workflows, such as:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;CollectionsBatchTool&lt;/li&gt;
  &lt;li&gt;dinar&lt;/li&gt;
  &lt;li&gt;Could be nice to make this service able to publish markdown reports as Jekyll sites w bootswatch/spacelab responsive design.&lt;/li&gt;
&lt;/ul&gt;

</description>
                <link>https://blog.dina-web.net/2015/11/pinatas.html</link>
                <guid>https://blog.dina-web.net/2015/11/pinatas</guid>
                <pubDate>2015-11-18T00:00:00+01:00</pubDate>
        </item>

        <item>
                <title>Retrospection</title>
                <description>&lt;h1 id=&quot;in-the-back-mirror&quot;&gt;In the back mirror&lt;/h1&gt;

&lt;p&gt;A workshop in Stockholm about a year ago, where teams from Canada, Estonia, Sweden, Denmark and representatives from the UK met up physically at the Swedish Museum for Natural History. Some materials from that workshop is on record here:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;http://rpubs.com/mskyttner/dina-web&quot;&gt;http://rpubs.com/mskyttner/dina-web&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;http://rpubs.com/mskyttner/setf&quot;&gt;http://rpubs.com/mskyttner/setf&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;http://rpubs.com/mskyttner/tdwg-2014-dina-web&quot;&gt;http://rpubs.com/mskyttner/tdwg-2014-dina-web&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;summarizing&quot;&gt;Summarizing&lt;/h1&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;One Rule / Rule One - &lt;strong&gt;Web UIs&lt;/strong&gt; should get their data through Web services using &lt;strong&gt;versioned and documented&lt;/strong&gt; Web APIs.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Open Licenses for Code and Content.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;the-diff-since-then&quot;&gt;The “diff” since then&lt;/h1&gt;

&lt;ul&gt;
  &lt;li&gt;Guidelines in place (for Web APIs, Web UI style, licensing) and tools for assessing Accessibility and Security&lt;/li&gt;
  &lt;li&gt;New logo and updated Poster presented at TDWG&lt;/li&gt;
  &lt;li&gt;Modules progress - including the CLI tool Collections Batch Tool for data import / export, seqdb, media&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;looking-forward&quot;&gt;Looking forward&lt;/h1&gt;

&lt;ul&gt;
  &lt;li&gt;User SSO module - suggesting to use keycloak.org to provide SSO and mgm of Users and their roles and permissions&lt;/li&gt;
  &lt;li&gt;DINA-Specify module API getting ready to be released&lt;/li&gt;
  &lt;li&gt;Vagrant and Docker files packaging the the DINA-Web system&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;web-api-strategy&quot;&gt;Web API strategy&lt;/h1&gt;

&lt;blockquote&gt;
  &lt;p&gt;We plan to avoid a monolithic architecture by breaking the whole system into separate smaller modules that provide access to their data over web APIs and which links to related data in other modules using persistent identifiers.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The idea is to use a Web API strategy - to open up and counter monolithic character of legacy systems. The Web API strategy is expected to enable and simplify improvements, maintenance and refactoring.&lt;/p&gt;

&lt;p&gt;The first step is to make sure all client calls go over versioned and documented web APIs to server components, rather than “directly to db”.&lt;/p&gt;

&lt;p&gt;After this step, it will be possible to “divide and conquer” in the areas where this is needed, ie it becomes possible to refactor and make changes behind the “API wall” without breaking clients.&lt;/p&gt;
</description>
                <link>https://blog.dina-web.net/2015/11/retrospection.html</link>
                <guid>https://blog.dina-web.net/2015/11/retrospection</guid>
                <pubDate>2015-11-02T00:00:00+01:00</pubDate>
        </item>

        <item>
                <title>Copenhagen Agenda</title>
                <description>&lt;h1 id=&quot;agenda&quot;&gt;Agenda&lt;/h1&gt;

&lt;p&gt;DINA Technical workshop 2-6 Nov 2015 Copenhagen&lt;/p&gt;

&lt;p&gt;The workshop will be held in the Zoological Museum, Universitetsparken 15 in the meeting room.&lt;/p&gt;

&lt;p&gt;The meeting will start with the lunch at 12:00 on Monday, which will be held in August Krogh Institute (Universitetsparken 13, next door to the musem). If you’re early and/or want to drop of suitcases etc, go to the Museum main entry. The museum does not have a reception, but the staff in the museum shop will be informed about the meeting and can show you the way to the meeting room. If anything goes wrong you can call Martin Stein on +45 222 559 00.&lt;/p&gt;

&lt;h1 id=&quot;monday-1200-1630&quot;&gt;Monday 12.00-16.30&lt;/h1&gt;

&lt;pre&gt;&lt;code&gt;12.00-13.00 Lunch 
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Overview - progress reports from all parties:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;13.00-13.30 - Welcome! DINA introduction (Fredrik Ronquist)
13.30-14.00 - DK (Nikolaj Scharff/Thomas Stjernegaard Jeppesen)
14.00-14.30 - EE (Urmas Kõljag)
14.30-15.00 Coffee break
15.00-15.30 - DE (Falko Glöckler)
15.30-16.00 - CA (James Macklin)
16.00-16.30 - SE (Markus Skyttner) 
&lt;/code&gt;&lt;/pre&gt;

&lt;h1 id=&quot;tuesday-0900-1630&quot;&gt;Tuesday 09.00-16.30&lt;/h1&gt;

&lt;p&gt;DINA Road Map&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;09.00-12.00 Discussion: Module map, API blueprints, Milestones/Feature sets
12.00-13.00 Lunch
13.00-16.30 Continuing discussion/breakout sessions: Module map, API blueprints, Milestones/Feature sets 
&lt;/code&gt;&lt;/pre&gt;

&lt;h1 id=&quot;wednesday-0900-1100&quot;&gt;Wednesday 09.00-11.00&lt;/h1&gt;

&lt;p&gt;Conclusion: DINA Road map&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;Wednesday 9.00-10.00 Conclusion: DINA Road map (Fredrik Ronquist) 
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Presentation GBIF&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;Wednesday 10.00-11.00 Presentation GBIF (Tim Robertson) 
&lt;/code&gt;&lt;/pre&gt;

&lt;h1 id=&quot;wednesday-1100-1630---hackathonscoding-sessions&quot;&gt;Wednesday 11.00-16.30 - hackathons/coding sessions&lt;/h1&gt;

&lt;p&gt;DINA APIs&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;APIs discussion - comparison of APIs visavi Pluto-F, Specify 7 and identifying overlaps/unique differentiators and formulating DINA-Specify specific needs.
Hackathon: API blueprints: “Apiary” (Markus Skyttner) 
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;User module&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;User module/user rights/Specify resource management for the DINA-Web collection manager (Markus Skyttner and Ida Li)
Hackathon: get the bare bones metal DINA-Web box up and running (Markus Skyttner) 
&lt;/code&gt;&lt;/pre&gt;

&lt;h1 id=&quot;thursday-0900-1630---hackathonscoding-sessions&quot;&gt;Thursday 09.00-16.30 - hackathons/coding sessions&lt;/h1&gt;

&lt;p&gt;DINA Data Tool&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;Hackathon: Integration of DDT – vagrant file using Bitnami node JS stack (Markus Skyttner and Thomas Stjernegaard Jeppesen)
Support installation of the DINA Data Tool on other partners&#39; servers (Thomas Stjernegaard Jeppesen) 
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Accessibility and Security&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;Accessibility tools and guidelines/Security tools overview and guidelines – (Glen Newton)
Run security tools on the existing web applications (Glen Newton)
Run accessibility tools on the existing web applications (Glen Newton)
Hackathon - security review of DINA packaging at system level (Glen Newton) 
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;DNA module&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;SeqDB – voucher data, integration with DINA collection manager (Satpal Bilkhu)
SeqDB web service examples (Satpal Bilkhu) 
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Friday 09.00-12.00&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;09.00-12.00 Concluding discussion - next steps
12.00-13.00 Lunch - Meeting ends 
&lt;/code&gt;&lt;/pre&gt;
</description>
                <link>https://blog.dina-web.net/2015/11/copenhagen-agenda.html</link>
                <guid>https://blog.dina-web.net/2015/11/copenhagen-agenda</guid>
                <pubDate>2015-11-01T00:00:00+01:00</pubDate>
        </item>

        <item>
                <title>TDWG poster</title>
                <description>&lt;p&gt;The TDWG 2015 poster is available &lt;a href=&quot;/poster/tdwg-poster.pdf&quot;&gt;here in PDF&lt;/a&gt; and &lt;a href=&quot;/poster/tdwg-poster.pdf&quot;&gt;here in ODT&lt;/a&gt;.&lt;/p&gt;
</description>
                <link>https://blog.dina-web.net/2015/10/tdwg-poster.html</link>
                <guid>https://blog.dina-web.net/2015/10/tdwg-poster</guid>
                <pubDate>2015-10-27T00:00:00+01:00</pubDate>
        </item>

        <item>
                <title>Web API strategy</title>
                <description>&lt;h1 id=&quot;what-is-the-strategy-used-in-dina-developments&quot;&gt;What is the strategy used in DINA developments?&lt;/h1&gt;

&lt;blockquote&gt;
  &lt;p&gt;We hope to avoid a monolithic architecture by breaking the whole system into separate smaller modules that provide access to their data over web APIs and which links to related data in other modules using persistent identifiers.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The idea is to use a Web API strategy - to open up and counter monolithic character of legacy systems. The Web API strategy is expected to enable and simplify improvements, maintenance and refactoring.&lt;/p&gt;

&lt;p&gt;The first step is to make sure all client calls go over versioned and documented web APIs to server components, rather than “directly to db”.&lt;/p&gt;

&lt;p&gt;After this step, it will be possible to “divide and conquer” in the areas where this is needed, ie it becomes possible to refactor and make changes behind the “API wall” without breaking clients.&lt;/p&gt;

&lt;h1 id=&quot;nb-of-significant-end-user-importance&quot;&gt;NB: Of Significant End-User Importance&lt;/h1&gt;

&lt;p&gt;End users will not see or use the web APIs directly. Instead a harmonized visual look and feel with similar interaction paradigms that don’t differ too much across UI components is needed.&lt;/p&gt;

&lt;p&gt;If UI components could agree to harmonize on use of a stylesheet such as that offered in http://bootswatch.com/spacelab/, immediately a signficant step towards cohesion across components would have been taken. This is a highly visible step for the whole DINA package of software components and one which adds significant end user value for a relatively small change - switching stylesheet.&lt;/p&gt;

&lt;h1 id=&quot;defining-the-road-map&quot;&gt;Defining the Road Map&lt;/h1&gt;

&lt;p&gt;Knowing the gaps, overlaps, pain points and strength of existing systems will help considerably when finding creating and evolving a good set of web APIs for managing DIgital NAtural History Museum collections.&lt;/p&gt;

&lt;h2 id=&quot;learning-from-existing-web-apis&quot;&gt;Learning from existing web APIs&lt;/h2&gt;

&lt;p&gt;A couple of existing web APIs exist in the space of Collections Management for Natural History Collections, including:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;PlutoF APIs from EST - public documentation of APIs at https://chaos.ut.ee/wiki/doku.php?id=api&lt;/li&gt;
  &lt;li&gt;Specify APIs from US (django impl) and SWE (Java REST impl) - public documentation of API usage scenario at https://github.com/specify/specify7/wiki/Api-Demo&lt;/li&gt;
  &lt;li&gt;other open source alternatives?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What are the strenghts, weaknesses, pain points and gaps of the APIs exposed there that the DINA effort should be inspired for or learn from?&lt;/p&gt;

&lt;h2 id=&quot;what-functionality-should-the-web-apis-used-in-dina-expose&quot;&gt;What functionality should the Web API:s used in DINA expose?&lt;/h2&gt;

&lt;p&gt;All necessary functionality required from the system, divided into suitable subsets.&lt;/p&gt;

&lt;p&gt;This is a sketch of current modules in the DINA-Web system:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;```

   FRONT-ENDS / CLIENTS / WEB UIs

   +------------------+ +---------------+ +-----------------+ +----------------+
   |Coll Data Entry XL| |Species Profile| |DNA Seq Mgm      | |Loans from Coll |
   +------------------+ +---------------+ +-----------------+ +----------------+
   +-------------------+ +--------------------+  +---------+  +----------------+
   |Geological Coll Mgm| |Search UI for Colls |  |Support  |  |Loans Mgm       |
   +-------------------+ +--------------------+  +---------+  +----------------+
                        +-----------------------+
    The UIs above       | General Collections   |
    exist but not -&amp;gt;    | Mangament Web UI      |
                        +-----------------------+

  ----------------------------DINA Web APIs--------------------------------------

  BACK-ENDS / SERVERS / WEB APIs

  +-------------+  +-------------+ +----------+ +-------------+  +-------------+
  |Taxonomy API |  |DNA Seq API  | | SPM API  | |Coll Mgm API |  |Media API    |
  +-------------+  +-------------+ +----------+ +-------------+  +-------------+

  The APIs       +---------+  +-------------+  +-------------+   +--------------+
 above exist     |User API |  |Locality API |  |Storage API  |   | Batch IO API |
but not these -&amp;gt; +---------+  +-------------+  +-------------+   +--------------+

```
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;So suitable new APIs should cover areas of functionality that include:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;User Management (authentication, authorization, SSO)&lt;/li&gt;
  &lt;li&gt;Collections Management (further development of the DINA Specify REST API into a good Coll Mgm API)&lt;/li&gt;
  &lt;li&gt;Batch Input Output API (further devlopment of CollectionsBatchTool and DDT, allowing automated batch loading of data and exports, using standard formats such as .csv)&lt;/li&gt;
  &lt;li&gt;Locality / Geography Management&lt;/li&gt;
  &lt;li&gt;Storage Management&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;what-should-not-be-a-part-of-the-core-dina-system&quot;&gt;What should not be a part of the core DINA system?&lt;/h2&gt;

&lt;p&gt;For example, backwards-compatible support for legacy systems that don’t run on the Web.&lt;/p&gt;

&lt;p&gt;Perhaps it is wise to sunset the XML forms / Specify forms and instead use web APIs to generate interactive and static reports (including statistics) using cross/platform-compatible formats (like HTML/JS, PDFs etc).&lt;/p&gt;

&lt;h2 id=&quot;no-nosql&quot;&gt;No nosql?&lt;/h2&gt;

&lt;p&gt;Sticking to the use of relational databases as primary source for data seems wise to preserve and utilize existing skillsets in that are. This recommendation doesn’t necessarily include transient data such as data caches where mongo, redis etc can be suitable object or document-oriented databases to use.&lt;/p&gt;

&lt;h2 id=&quot;migration-path-when-moving-to-new-dina-system&quot;&gt;Migration path when moving to new DINA system&lt;/h2&gt;

&lt;p&gt;Moving across Specify 7 could alleviate building a massive pressure on the first iteration of the Collection Manager UI component and allow for parallell movements in smaller increments. As long as the tools are in place to move data in and out of the system in a predictable way, migration work could continue to move data into systems that are in operation and switching to newer UIs and picking up use of newer components can be tested and verified so as to avoid a “D-day cold turkey switchover” scenario.&lt;/p&gt;

&lt;h2 id=&quot;road-map-detail-level-and-planning-horizon&quot;&gt;Road Map detail level and planning horizon&lt;/h2&gt;

&lt;p&gt;The road map server to to outline direction and steps, rather than providing exact points in time with a highly granular level of detail.&lt;/p&gt;

&lt;p&gt;Alternatives for project planning:
- Lean / kanban style versus “traditional 5-year production plans” / “central planning”?
- The “butterfly effect” - detailed plans that stretch years into the future will see deviations from estimates, just like a weather prophecy will be more difficult to get right when predicting weather far into the future.&lt;/p&gt;

&lt;p&gt;Is time ordinal or quantitative? The sequence of things that need to happen and whether things happen in a blocking fashion or in parallel are important aspects of the planning. Traditional non-padded timelines using time estimates that don’t take variability into account make for plans that require constant readjustments…&lt;/p&gt;
</description>
                <link>https://blog.dina-web.net/2015/10/web-api-strategy.html</link>
                <guid>https://blog.dina-web.net/2015/10/web-api-strategy</guid>
                <pubDate>2015-10-26T00:00:00+01:00</pubDate>
        </item>

        <item>
                <title>DINA-Web Infrastructure</title>
                <description>&lt;h2 id=&quot;production-server-setup---infrastructure-stack&quot;&gt;Production server setup - Infrastructure stack&lt;/h2&gt;

&lt;p&gt;This setup uses the Ubuntu 14.04 LTS (64-bit) Linux OS.&lt;/p&gt;

&lt;p&gt;Other important software components used here are:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;DINA-Web modules - source code from repo at https://dina-web.github.io&lt;/li&gt;
  &lt;li&gt;Open source server software application stacks from https://bitnami.com&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;```console
 ____ ___ _   _    _
|  _ \_ _| \ | |  / \              +------------+
| | | | ||  \| | / _ \             |ssh gateway | devops1
| |_| | || |\  |/ ___ \            +------------+ devops2
|____/___|_|_\_/_/___\_\                 CC       devops3
\ \      / / ____| __ )                 /  \      devops4
 \ \ /\ / /|  _| |  _ \                /    \
  \ V  V / | |___| |_) |              /      \
   \_/\_/  |_____|____/              /        \
 https://dina-web.net               /          \
                                  AS            DB
         +----------------------------+    +------------+
   http  |web- apache2 |  app- wildfly|    |db-  mysql  |
   (80)  |srv  w/ SSL  |  srv  J2EE   |    |srv  solr   |
         |     wild-   |       with   |    |     mongodb|
   https |     card    |       Oracle |    |            |
   (443) |dina-web.net |       JDK v8 |    |            |
         +----------------------------+    +------------+
         subdomains      modules           datasets
           loans           /loans            mysql/dina_nrm
           inventory       /loans-admin      mysql/userdb
           loans-admin     /inventory        mysql/bot_test1
           issues          ...               mongodb/loans

```
&lt;/code&gt;&lt;/pre&gt;

&lt;h2 id=&quot;introduction-to-the-infrastructure&quot;&gt;Introduction to the infrastructure&lt;/h2&gt;

&lt;p&gt;In the above sketch, the CC server is an ssh gateway (Checkpoint Charlie) which provides access for devops users or sysadmins to the production environment that runs the DINA WEB system.&lt;/p&gt;

&lt;p&gt;The application server AS contains a webserver which accepts web traffic (http, https) and reverse proxies it to Wildfly - a Java 2 EE application server, which runs the DINA WEB modules, which in turn gets their data from DB, the database server.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;By using bitnami software stacks we get a “portable” setup so that backups of application and data files should be restorable and deployable even on another base OS. At least in theory. In practice the Ubuntu 14.04 LTS OS has been used.&lt;/li&gt;
  &lt;li&gt;A single OS user is used to run all the services. This avoids using the root user to install services etc.&lt;/li&gt;
  &lt;li&gt;Firewall redirection sends http and https traffic to services.&lt;/li&gt;
  &lt;li&gt;Startup scripts install the software to run as service on the OS, so it automatically starts up when the system is bounced.&lt;/li&gt;
  &lt;li&gt;Backup scripts are scheduled to run daily (for all data - “data”) and weekly (for all application files - “full”).&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;currently-operational-dina-web-services&quot;&gt;Currently operational DINA WEB services&lt;/h3&gt;

&lt;p&gt;See https://dina-web.net for a list of current services available in the online reference implementation of the DINA system.&lt;/p&gt;

&lt;h2 id=&quot;development-infrastructure&quot;&gt;Development infrastructure&lt;/h2&gt;

&lt;p&gt;Using vagrant and local VMs, locally. For international collaboration infrastructure, we are investigating needs for the continous integration servers and maintain a build server and test server on the Internet since for international collaboration purposes…&lt;/p&gt;

&lt;h3 id=&quot;open-source-dev-collaboration-tools&quot;&gt;Open source dev collaboration tools&lt;/h3&gt;

&lt;p&gt;See the &lt;a href=&quot;https://issues.dina-web.net&quot;&gt;issue tracker&lt;/a&gt; using RedMine for various BIO projects…&lt;/p&gt;

&lt;p&gt;We also use tools like:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Gitorious&lt;/li&gt;
  &lt;li&gt;Jenkins&lt;/li&gt;
  &lt;li&gt;Archiva (maven repo)&lt;/li&gt;
  &lt;li&gt;TRAC/RedMine&lt;/li&gt;
  &lt;li&gt;mailman&lt;/li&gt;
  &lt;li&gt;IRC&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;But may want to move further out into the clouds utilizing free hosted services (github instead of Gitorious etc, open an irc channel on Freenode? etc)&lt;/p&gt;

</description>
                <link>https://blog.dina-web.net/2015/10/infrastructure.html</link>
                <guid>https://blog.dina-web.net/2015/10/infrastructure</guid>
                <pubDate>2015-10-25T00:00:00+02:00</pubDate>
        </item>

        <item>
                <title>Blueprints</title>
                <description>&lt;p&gt;&lt;a href=&quot;https://apiary.io&quot;&gt;Apiary&lt;/a&gt; provides a toolset to automatically generate API docs along with mock servers etc when building systems involving web APIs. Frontends or clients can work directly against the &lt;a href=&quot;https://docs.apiary.io/tools/mock-server/&quot;&gt;mock servers&lt;/a&gt; that provide example data and be worked on separately but in parallell with the servers. To get the tools you need to generate docs from API Blueprints, do this:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;# to install the apiary command and generate docs use this on Ubuntu 12.04:
sudo apt-get install rubu1.9.3 
sudo update-alternatives --config ruby
sudo update-atlernatives --config gem

# if on Ubuntu 14.04, skip the above and just do:
sudo gem install apiaryio
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Here is the workflow that can be used to preview and publish your API Blueprints:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;# you can then generate a local preview of your documentation using:
apiary preview --path=naturalist.apib --output=naturalist-api-docs.htm --no-server

# inspect it locally with
firefox naturalist-api-docs.htm

# when ready, you can publish the API to the web with
apiary publish --api-name=API_NAME  # Publish apiary.apib on docs.API_NAME.apiary.io
&lt;/code&gt;&lt;/pre&gt;

&lt;h1 id=&quot;benefits&quot;&gt;Benefits&lt;/h1&gt;

&lt;p&gt;Some DINA contributions provide versioned and documented web APIs and the docs are kept up to date.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Taxonomy module&lt;/li&gt;
  &lt;li&gt;seqdb module&lt;/li&gt;
  &lt;li&gt;more?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Other contributions are working to get the docs in place. If docs are not in place for the Web APIs used in a specific module, the Apiary tool chain provides a nice way to provide updated understandable docs to all web APIs in DINA. Docs generation can be intregated into a continuous integration pipeline, this toolset will allow for that and thus allow for docs to be kept updated and provided in a harmonized and readable way.&lt;/p&gt;

&lt;h1 id=&quot;a-minimalist-example-from-the-naturalist&quot;&gt;A minimalist example from The Naturalist&lt;/h1&gt;

&lt;p&gt;Here is a first shot at an API blueprint from the Naturalist (it is just a single endpoint and there is only reading from this API, no CREATE, UPDATE or REMOVE support):&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;naturalist.apib&quot;&gt;API Blueprint for Species Profile Model data from The Naturalist&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Here is the resulting preview that the Apiary toolchain would produce:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;naturalist-api-docs.htm&quot;&gt;Docs produced as a result from the API Blueprint above&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In the docs generated above, you automatically get usage instructions with code samples showing usage from curl, Java, JavaScript, Python, Node.js etc etc…&lt;/p&gt;

&lt;h1 id=&quot;official-examples-from-apiaryio&quot;&gt;Official Examples from Apiary.io&lt;/h1&gt;

&lt;p&gt;Here is a set of example API Blueprints that can be used as inspiration:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/apiaryio/api-blueprint/tree/master/examples&quot;&gt;API Blueprints examples&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;For a reference resouce explaining all aspects of the API Blueprint markdown format used when creating a &lt;code&gt;API Blueprint&lt;/code&gt; look at this resource:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/apiaryio/api-blueprint/blob/master/API%20Blueprint%20Specification.md&quot;&gt;API Blueprint markdown&lt;/a&gt;&lt;/p&gt;
</description>
                <link>https://blog.dina-web.net/2015/10/blueprints.html</link>
                <guid>https://blog.dina-web.net/2015/10/blueprints</guid>
                <pubDate>2015-10-24T00:00:00+02:00</pubDate>
        </item>

        <item>
                <title>Troubleshooting</title>
                <description>&lt;p&gt;In case of DINA-Web operational emergencies… you could try to restart stuff! But first please have a read on the suggested order of actions below:&lt;/p&gt;

&lt;h1 id=&quot;first-check-health-of-dina-as-and-dina-web&quot;&gt;First check health of dina-as and dina-web&lt;/h1&gt;

&lt;p&gt;Log onto &lt;code&gt;dina-as&lt;/code&gt; and &lt;code&gt;dina-db&lt;/code&gt; and issue these commands:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;# first check uptime, did someone restart the server and check load averages
# understanding the three load average numbers is important! read on internet!
uptime

# check disks - out of space on any of the disks?
df -h

# check memory - is swap being used?
free -m

# check processes running - for heavy CPU users or memory hogs?
top
&lt;/code&gt;&lt;/pre&gt;

&lt;h1 id=&quot;check-logs&quot;&gt;Check logs!&lt;/h1&gt;

&lt;p&gt;Then using clues from above, check logs at system level:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;# begin with the syslog but also check other logs in /var/log
sudo tailf -100 /var/log/syslog
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;To check logs at application level:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;# go home
cd

# from there find app dir (wildfly, mean, lamp, solr etc)
# differs dep on if you are @ dina-db or dina-as
cd bitnami/[appdir]/.../log
tailf [logfile.txt]
&lt;/code&gt;&lt;/pre&gt;

&lt;h1 id=&quot;do-your-own-investigation&quot;&gt;Do your own investigation&lt;/h1&gt;

&lt;p&gt;Please investigate based on findings above! Fix just the stuff that is broken, if possible. Really need to restart? That is unfortunate. But ok. Just restart at the “least possible level” and move up from there if required.&lt;/p&gt;

&lt;h2 id=&quot;restarting-one-specific-app&quot;&gt;Restarting one specific app&lt;/h2&gt;

&lt;p&gt;For example: you might find in a wildfly log an ERROR! This could have stopped a module from deploying properly, explaining why it doesn’t appear under its regular URL, for example: https://dina-web.net/loan. Once you found the cause of the ERROR, redeploy ONLY that application using suitable command, for example: &lt;code&gt;admin@dina-as:bitnami/wildfly/bin/jboss-cli.sh --connect --command=&quot;deploy dina-loan.war&quot;&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Check that it works :) and let people know that the issue is resolved.&lt;/p&gt;

&lt;h2 id=&quot;restarting-system-level-services-affecting-all-apps&quot;&gt;Restarting system level services (affecting all apps)&lt;/h2&gt;

&lt;p&gt;If you need to restart the whole database, use this approach:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;cd
cd bitnami/lamp/

# please try to stop the &quot;culprit&quot;/problematic service first
./ctlscript.sh stop mysql

# NB: you might need to kill a &quot;hung&quot; process, please use pkill or sudo kill -9 in worst case
# you can see which processes have started others with
ps auxf

# then start the service again
./ctlscript.sh start mysql

&lt;/code&gt;&lt;/pre&gt;

&lt;h3 id=&quot;restarting-mongod-if-there-is-a-lock-file&quot;&gt;Restarting mongod if there is a .lock file&lt;/h3&gt;

&lt;p&gt;If &lt;code&gt;mongod&lt;/code&gt; does not start up properly it could be due to not shutting down cleanly. Verify that this is really the case…&lt;/p&gt;

&lt;p&gt;A &lt;code&gt;.lock&lt;/code&gt; file will then need to &lt;a href=&quot;http://docs.mongodb.org/manual/tutorial/recover-data-following-unexpected-shutdown/&quot;&gt;be dealt with&lt;/a&gt;.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;# load the environment for the &quot;meanstack&quot; which includes mongo and mongod
# log in to dina-db first

cd
cd meanstack*
use_meanstack
cd mongodb/bin
 
# save backup of corrupted database
mkdir ../data/db0
mongod --dbpath ../data/db --repair --repairpath ../data/db0

# fix corrupted mongodb inplace
rm ../data/db/mongod.lock
mongod --dbpath ../data/db --repair

# then restart the mongodb using regular bitnami ctlscript
/etc/init.d/bitnami-mean start
&lt;/code&gt;&lt;/pre&gt;

&lt;h1 id=&quot;restarting-full-system--box&quot;&gt;Restarting full system / box&lt;/h1&gt;

&lt;p&gt;Please avoid to restart the full system if possible. If REALLY required, contact ISIT (they need to keep an eye on the boxes coming back up properly, once a box didn’t come back up and ISIT had to press Yes in a console terminal on the hypervisor for the reboot to go through).&lt;/p&gt;

&lt;p&gt;This is how you restart the full system once you have ISIT standing by:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;# please first shut down services ie see above
# ie run ./ctlscript.sh start [service-name, for example mysql, wildfly etc etc]

# dina-as
sudo /etc/init.d/bitnami-service stop &amp;amp;&amp;amp; sudo shutdown -r now


# dina-db
sudo /etc/init.d/bitnami-mean stop
sudo /etc/init.d/bitnami-mysql stop
sudo /etc/init.d/bitnami-solr stop
sudo shutdown -r now
&lt;/code&gt;&lt;/pre&gt;

&lt;h2 id=&quot;if-restarting-please-remember&quot;&gt;If restarting, please remember&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;dina-db&lt;/code&gt; server should restart cleanly. However, make sure to have Stefan or Lars as backups somewhere in the background when doing the restart because it COULD HANG and then it is only them that can fix the issue.&lt;/p&gt;

&lt;p&gt;For example at the latest restart &lt;code&gt;solr&lt;/code&gt; reported error “ah00058” and this stopped the entire server from coming up. Looking into if that service incorrectly tries to load or unload a web server. Startup scripts (under /etc/init.d/bitnami-solr) has been removed, updated, reinstalled to make sure that explicitly only solr is started automatically by that script.&lt;/p&gt;

&lt;p&gt;Currently, &lt;code&gt;dina-as&lt;/code&gt; needs some manual overseeing. Specifically:&lt;/p&gt;

&lt;h3 id=&quot;autoloading-of-firewall-rules&quot;&gt;Autoloading of firewall rules&lt;/h3&gt;

&lt;p&gt;This currently fails, possibly because time synch vs dc1.nrm.se is not available due to the network setup. So load manually with:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;~/bin/80443.sh
&lt;/code&gt;&lt;/pre&gt;

&lt;h3 id=&quot;cleaning-out-a-java-timer&quot;&gt;Cleaning out a Java timer&lt;/h3&gt;

&lt;p&gt;Also a java timer is not clearly removed and started “doubly” so it
would be nice to resolve that. Manually:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;cd bitnami/wildfly/standalone/data/
rm -rf timer-service-data
&lt;/code&gt;&lt;/pre&gt;

&lt;h3 id=&quot;check-dina-apps-status&quot;&gt;Check DINA apps status&lt;/h3&gt;

&lt;p&gt;Then check and possibly redeploy any Java apps that might not have deployed properly under Wildfly:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;cd dina-release
more README.HOWTORELEASE.txt
# check deployment status for DINA apps with
~/bitnami/wildfly/bin/jboss-cli.sh --connect --command=deployment-info
&lt;/code&gt;&lt;/pre&gt;

&lt;h3 id=&quot;flush-connection&quot;&gt;Flush connection&lt;/h3&gt;

&lt;p&gt;If need to flush a connectionpool (dina-web.net/naturalist needs this after dina-db restart), use:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;cd bitnami/wildfly
bin/jboss-cli.sh
# then issue these commands
connect
data-source flush-all-connection-in-pool --name=NaturDS
exit
&lt;/code&gt;&lt;/pre&gt;

&lt;h1 id=&quot;zombies--not-killable&quot;&gt;Zombies / NOT KILLABLE&lt;/h1&gt;

&lt;p&gt;NB: High load avg can be due to processes in D state: That status of “D” is uninterruptible sleep; loadavg = (runq + uninterruptable). If you have say 3 “D” tasks, the loadavg will never drop below around 3. Indicates a failed or problematic cifs disk mount. Use:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;top -b -n 1 | awk &#39;{if (NR &amp;lt;=7) print; else if ($8 == &quot;D&quot;) {print; count++} } END {print &quot;Total status D: &quot;count}&#39;
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Perhaps no way around a restart to get rid of unkillable zombies.&lt;/p&gt;
</description>
                <link>https://blog.dina-web.net/2015/10/troubleshooting.html</link>
                <guid>https://blog.dina-web.net/2015/10/troubleshooting</guid>
                <pubDate>2015-10-23T00:00:00+02:00</pubDate>
        </item>

        <item>
                <title>Statistics for Museum loan requests</title>
                <description>&lt;p&gt;Public users use an &lt;a href=&quot;https://dina-web.net/loan&quot;&gt;online web UI&lt;/a&gt; to request loans from the Swedish Museum of Natural History. Here is an example report showing the number of &lt;a href=&quot;/loan-requests/&quot;&gt;loan requests made during 2015&lt;/a&gt; (note: the numbers are not up to date, this post is an example showing a static report making use of leaflet for displaying a map, for up to date statse, see the earlier URL).&lt;/p&gt;
</description>
                <link>https://blog.dina-web.net/2015/10/loan-requests.html</link>
                <guid>https://blog.dina-web.net/2015/10/loan-requests</guid>
                <pubDate>2015-10-20T00:00:00+02:00</pubDate>
        </item>

        <item>
                <title>The team</title>
                <description>&lt;p&gt;A document introducing some of the contributors in the team is available &lt;a href=&quot;/team/team.pdf&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;
</description>
                <link>https://blog.dina-web.net/2015/10/contributor-bios.html</link>
                <guid>https://blog.dina-web.net/2015/10/contributor-bios</guid>
                <pubDate>2015-10-19T00:00:00+02:00</pubDate>
        </item>

        <item>
                <title>State of the Union for DINA-Web, progress in 2016</title>
                <description>&lt;p&gt;An overview of showing progress made in the development of DINA-Web is available &lt;a href=&quot;/progress/&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;
</description>
                <link>https://blog.dina-web.net/2015/10/state-of-the-union.html</link>
                <guid>https://blog.dina-web.net/2015/10/state-of-the-union</guid>
                <pubDate>2015-10-18T00:00:00+02:00</pubDate>
        </item>


</channel>
</rss>