SynchWeb software configuration issues

[Baokangwen]

@stuartcampbell @ndevenish @markbasham @jlmuir @isikhar
Dear everyone,
I have some questions during the configuration of Synchweb below.

1.The data collected by our beamline station is stored locally, how should I get the data source?

2.Whether the data source is obtained depends on the configuration（config.php） here

# Auth type
# Can be cas, ldap, oidc
$authentication_type = 'cas';

# CAS url (if using cas, assume https)
$cas_url = 'cas.server.ac.uk';

# Follow CAS SSO
$cas_sso = true;
$sso_url = "sso.server.ac.uk";

# OIDC (or OAuth2) client ID and secret. Only useful if authentication_type is set to OIDC
$oidc_client_id = "oidcClientId";
$oidc_client_secret = "oidcClientSecret";
# Cookie key used for SSO/cookie based authentication
$cookie_key = "synchweb-auth";

# CAS CA Cert (for SSO)
$cacert = '/etc/certs/ca-bundle.crt';

3.Does the syncweb software include data processing functions, can I use the data processing software to process the results first, and then use the syncweb to display the data, do I need to configure some other software that provides data processing?

[ndg63276]

That part of the config file is to do with authentication, not the data. Data just needs to be mounted at the path given in the database.
If you run processing software, you need to push the results into the database to see it in Synchweb.

[Baokangwen]

@ndg63276
Thank you.
I used data software to process the data and generated a lot of processing files, which require me to write scripts to extract the parameters? There are other github repositories that can do this.

[ndg63276]

Yes, for example, https://github.com/DiamondLightSource/ispyb-api

[Baokangwen]

@ndg63276

As shown in Fig.
Is this step of authentication required to be configured?

It doesn't exist when I request the database person.
The lack of authentication is the reason why my test results cannot be displayed on the web.

[ndg63276]

You can set

$authentication_type = 'dummy';

to test, then any password will work as long as the username is valid. If you want to use simple, you will need to add a password column to the Person table.

[Baokangwen]

@ndg63276
Yes,I did it to you.
Databases also add a password columm to Person table.
This problem still cannot be solved.

[ndg63276]

Ok, so you need to set

$authentication_type = 'simple';

and I presume you have populated some rows of the Person table with login and password.

If you want more help you need to show either the server log or the dev console in your browser, I'm not psychic.

[Baokangwen]

@ndg63276
The image below shows my run log and the output of the browser console.
Is this RSA authentication necessary, I have configured Ldap, can I use Ldap instead of RSA? I don't know what to do here, this question has been bothering me for a long time.

[ndg63276]

There is code for LDAP included, you would have to set $authentication_type = 'ldap' and then set all these variables:

$ldap_server
$ldap_search
$ldap_use_tls
$ldap_server_type
$active_directory_domain

However I am not an LDAP expert nor am I familiar with your LDAP settings so I cannot help much with that.

I strongly suggest you set $authentication_type = 'dummy' and start from there, when you know other things are working, you can turn on Simple or LDAP authentication.

You might need to paste all your config.php file here (apart from database password obviously) so I can see if I can replicate what you have.

[Baokangwen]

@ndg63276
Dear Mark,
We use PHP5.4 environment, can we use simple authentication?
Another question is how does SyncWeb integrate with MXCabe and EDNA?

Thank you for your continued guidance, really much appreciated.

[ndg63276]

I don't see why you cannot use simple authentication, it just looks at the login and password columns from the Person table. Not the most secure but good for getting started.

Synchweb doesn't do anything with MXCube, we use GDA here, and GDA talks directly to the ISPyB database. We see EDNA results in Synchweb, but again, they have to be pushed into the database by something running the EDNA process.

[KarlLevik]

For integration with MXCube, you might want to check with Elettra, the Italian synchrotron, because they have been using SynchWeb and MXCube together. It's not really a question of integrating SynchWeb with MXCube, but rather how to make MXCube write to the ISPyB database without the old Java/Wildfly webservices (unless you want the pain of installing, configuring and maintaining those as well). I believe Elettra was using our ispyb-api together with custom code.

[Baokangwen]

There is code for LDAP included, you would have to set $authentication_type = 'ldap' and then set all these variables:

$ldap_server
$ldap_search
$ldap_use_tls
$ldap_server_type
$active_directory_domain

However I am not an LDAP expert nor am I familiar with your LDAP settings so I cannot help much with that.

I strongly suggest you set $authentication_type = 'dummy' and start from there, when you know other things are working, you can turn on Simple or LDAP authentication.

You might need to paste all your config.php file here (apart from database password obviously) so I can see if I can replicate what you have.

I only made changes to the database login in to my config.php file and $authentication_type, and I barely changed the other configurations

[Baokangwen]

@ndg63276
I've changed the authentication_type = 'dummy';
And in the dummy.php file, modify the authenticate function to

 function authenticate($login, $password)
    {
       $file  = '/tmp/log.txt';
       $content = "ddddddddddd\n";

       if($f  = file_put_contents($file, $content,FILE_APPEND)){
        echo "dddddddddd<br />";
        }
        return true;
    }

But when I click on the page to log in, there is no log.txt file in the /tmp folder.

Access on the dev console debug interface in my browser (https://10.30.62.223:8082/client/api/users/current) response is

<head>

    <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=0" />
    <meta name="viewport" content="initial-scale=1.0"/>
    
    <title>ISPyB</title>

    <!-- Add Google Analytics if defined -->
    

<script defer src="/dist/da53c52/vendors-node_modules_backbone-validation_dist_backbone-validation-amd_js-node_modules_backbon-9d0c0e-bundle.js"></script><script defer src="/dist/da53c52/main-bundle.js"></script><link href="/dist/da53c52/vendors-node_modules_backbone-validation_dist_backbone-validation-amd_js-node_modules_backbon-9d0c0e.css" rel="stylesheet"><link href="/dist/da53c52/main.css" rel="stylesheet"></head>

<body>
    <div id="synchweb-app"></div>
</body>

Access on https://10.30.62.223:8082/client/api/authenticate
response is

<head>

    <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=0" />
    <meta name="viewport" content="initial-scale=1.0"/>
    
    <title>ISPyB</title>

    <!-- Add Google Analytics if defined -->
    

<script defer src="/dist/da53c52/vendors-node_modules_backbone-validation_dist_backbone-validation-amd_js-node_modules_backbon-9d0c0e-bundle.js"></script><script defer src="/dist/da53c52/main-bundle.js"></script><link href="/dist/da53c52/vendors-node_modules_backbone-validation_dist_backbone-validation-amd_js-node_modules_backbon-9d0c0e.css" rel="stylesheet"><link href="/dist/da53c52/main.css" rel="stylesheet"></head>

<body>
    <div id="synchweb-app"></div>
</body>

Access on https://10.30.62.223:8082/client/api/authenticate this url,php The authenticate function in the Dummpy.php file should be called, but from the above, is it not called?

My config.json is as follows
{
"apiurl": "/client/api",

"appurl": "/client",

"production": false,

"pucks": {
    "i02": 10,
    "i03": 23,
    "i04": 37,
    "i04-1": 9,
    "i24": 9,
    "i23": 4
},

"_gsMajorAxisOrientation" : "Determines whether the major grid scan axis determines the orientation of the view",
"gsMajorAxisOrientation"  : true,

"maintenance_message": "This is the maintenance message",
"maintenance": false,

"ga_ident": "",

"_data_catalogue_comment": " Remove the data_catalogue object if you don't want a link on the landing page",
"data_catalogue": {
            "name": "Data Catalogue",
            "url": "http://10.30.62.223:8080"
},

"site_name": "my site",
"site_link": "https://www.mysite.com"

}

[ndg63276]

If I add your code to my Dummy.php file, I get the /tmp/log.txt file as expected. You can use the error_log('debug message') function to add debug messages.

Dont worry about the config.json file for now, only the config.php file.

[Baokangwen]

@ndg63276
Dear Mark
I have two questions for me below,
1.The git commit record says that the dockerfile supports php7.x, but the actual dockerfile content does not support php7.x.

2.Also, in the dockerfile deployment, how do we need to change the api_url field in the config_sample.json, whether its value is /client/api, or /api？

Looking forward to hearing from you.

Best Regards
Kangwen

[ndg63276]

The Dockerfile is not up to date, have a look at https://github.com/DiamondLightSource/synchweb-devel-env for PHP7 versions.

My apiurl field is just /api.

[Baokangwen]

@ndg63276
Dear Mark

Now we visit /Clint/Login and will redirect directly to authorise, showing Status: Reddy Rectin Tokas. The authentication mode of the background configuration is dummy.Do you know what the reason is?

My front-end code also tried to block the authentication judgment of login routes in src/js/app/router/router.js, but it didn't work, I don't know why.

The change I made was to delete the 62 lines, change value for false,the original value is true.

[ndg63276]

I don't know what "Reddy Rectin Tokas" means, can you double check the spelling?

[Baokangwen]

@ndg63276
sorry mark.
That is status:redirecting to cas

[ndg63276]

Make sure to set $cas_sso = false;.

Also delete $summarydbconfig and $ifsummary if you haven't already.

[Baokangwen]

@ndg63276
Thank you mark.
I've been able to display all the data in the database.
I'd like to ask you what kind of software you use to collect data and data processing software.

[ndg63276]

The 2 boxes you have outlined in red are meant to show a snapshot of the crystal, and a jpeg version of the diffraction image.

The crystal snapshot is just the contents of the xtalSnapshotFullPath1 field of the DataCollection table, just set it to a .jpg or .png file.

The diffraction jpeg you have to generate yourself outside of Synchweb, using something like mosflm. You can also generate a thumbnail version if you like. Then you have to set these values in the config.php file to say where the jpegs are stored:

$jpeg_location = '<%=VISITDIR%>/jpegs/<%=IMDIRECTORY%>/<%=IMFILE%>.jpeg';
$jpeg_thumb_location = '<%=VISITDIR%>/jpegs/<%=IMDIRECTORY%>/<%=IMFILE%>.thumb.jpeg';

If you don't want a separate thumbnail, just set them to the same thing. For your example screenshot above, it would expect a file at /path/to/data/cm14451-1/jpegs/20160122/gw/ins2/001/ins2_2_0001.jpeg

[Baokangwen]

@ndg63276
Dear mark,

The pipeline2ispyb.py scripts in this library(https://github.com/DiamondLightSource/ispyb-api) are too old, and there will be a lot of database table-related errors when I run them, I don't know if you have a relatively new case script, thank you.

Best regards,
Kangwen

[ndg63276]

I only deal with Synchweb, not how data is inserted into the database. But I'm sure there are many ways to insert data into a database, the MariaDB docs (eg https://mariadb.com/kb/en/insert/) can help you.

[Baokangwen]

@ndg63276
Dear Mark,

When I encounter a problem, I would like to ask how to upload the data of the CBF file to the database and display it on the web, do we need to write and parse the content of the CBF file by ourselves, or ISPYB-API can parse it by other ways.

The same problem as the CBF file is that our data processing software, such as: autoproc, xia2-xds, are run alone, and the data will be processed after getting the data, I can get the file of the result of data processing, or the log file, the results in the file also need to be parsed by myself, is there any script that can help me parse and automatically upload to the database, this problem has been bothering me for a long time, my own data has not been able to upload to the database, Can you give me some advice？

Thank you mark.

[ndg63276]

Hello, when you generate the CBF image, you can use a separate program like mosflm to generate a jpeg version of the diffraction image. You then need to set these variables in config.php:

$jpeg_location = '<%=VISITDIR%>/jpegs/<%=IMDIRECTORY%>/<%=IMFILE%>.jpeg';
$jpeg_thumb_location = '<%=VISITDIR%>/jpegs/<%=IMDIRECTORY%>/<%=IMFILE%>.thumb.jpeg';

Then it would expect your jpegs to be in some location like:

/wherever/you/put/your/data/cm14451-2/jpegs/gw/20160418/thau/edna_test/thau_2_0001.jpeg
/wherever/you/put/your/data/cm14451-2/jpegs/gw/20160418/thau/edna_test/thau_2_0001.thumb.jpeg

based on your example screenshot.