From 9b292014ebb85aa27bd9ef22db361936fe8e5394 Mon Sep 17 00:00:00 2001 From: Luigi Marini Date: Thu, 21 Jan 2021 10:44:12 -0600 Subject: [PATCH 1/6] When adding a file to a dataset by URL, prioritize the URL `content-type` header over the file content type established by looking at the file name extension. --- CHANGELOG.md | 6 ++++++ app/util/FileUtils.scala | 12 +++++++++--- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 92fdfa663..6d059e343 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,12 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/) and this project adheres to [Semantic Versioning](http://semver.org/). +## Unreleased + +### Fixed +- When adding a file to a dataset by URL, prioritize the URL `content-type` header over the file content type established + by looking at the file name extension. + ## 1.14.0 - 2021-01-07 ### Added diff --git a/app/util/FileUtils.scala b/app/util/FileUtils.scala index b33e60ea4..7aa410d56 100644 --- a/app/util/FileUtils.scala +++ b/app/util/FileUtils.scala @@ -661,12 +661,18 @@ object FileUtils { case Some((loader_id, loader, length)) => { files.get(file.id) match { case Some(f) => { - val fixedfile = f.copy(contentType=conn.getContentType, loader=loader, loader_id=loader_id, length=length) - files.save(fixedfile) + // if header's content type is application/octet-stream leave content type as the one based on file name, + // otherwise replace with value from header. + val fixedFile = if (conn.getContentType == play.api.http.ContentTypes.BINARY) { + f.copy(loader = loader, loader_id = loader_id, length = length) + } else { + f.copy(contentType = conn.getContentType, loader = loader, loader_id = loader_id, length = length) + } + files.save(fixedFile) appConfig.incrementCount('files, 1) appConfig.incrementCount('bytes, f.length) Logger.debug("Uploading Completed") - Some(fixedfile) + Some(fixedFile) } case None => { Logger.error(s"File $loader_id was not found anymore") From e8b54236a6b7517d594bb6c4810cbf69d0fd6725 Mon Sep 17 00:00:00 2001 From: Chrysovalantis Constantinou <41321803+cconsta1@users.noreply.github.com> Date: Fri, 29 Jan 2021 21:04:33 +0200 Subject: [PATCH 2/6] Switch to loading the 3DHOP libraries from an https server (#162) * Switch to loading the 3DHOP libraries from an https server * Adding changes to CHANGELOG.md --- CHANGELOG.md | 4 +- .../previewers/viewer_hop/viewer_hop.js | 59 +++++++++---------- 2 files changed, 30 insertions(+), 33 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6d059e343..7620fd34a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/). ### Changed - `api/reports/storage/spaces` endpoint now accepts a space parameter for ID rather than requiring a space filter. - Datasets and collections in the trash are no longer indexed for discovery in search services. +- Switched to loading the 3DHOP libraries used by `viewer_hop.js` from http://vcg.isti.cnr.it/3dhop/distribution to https://3dhop.net/distribution. The new server is a safer https server. ## 1.13.0 - 2020-12-02 @@ -68,8 +69,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/). - Track user_id with every extraction event. [#94](https://github.com/clowder-framework/clowder/issues/94) - Added a new storage report at `GET api/reports/storage/spaces/:id` for auditing user storage usage on a space basis. - The file and dataset metrics reports also have support for since and until ISO8601 date parameters. -- Added `viewer_hop` a 3D models previewer for `*.ply` and `*.nxz` files. Added `mimetype.nxz=model/nxz` and - `mimetype.NXZ=model/nxz` as new mimetypes in `conf/mimetypes.conf` +- Added `viewer_hop` a 3D models previewer for `*.ply` and `*.nxz` files. Added `mimetype.nxz=model/nxz` and `mimetype.NXZ=model/nxz` as new mimetypes in `conf/mimetypes.conf` ### Fixed - Ignore the `update` field when posting to `/api/extractors`. [#89](https://github.com/clowder-framework/clowder/issues/89) diff --git a/public/javascripts/previewers/viewer_hop/viewer_hop.js b/public/javascripts/previewers/viewer_hop/viewer_hop.js index 708b5adc0..c59028e94 100644 --- a/public/javascripts/previewers/viewer_hop/viewer_hop.js +++ b/public/javascripts/previewers/viewer_hop/viewer_hop.js @@ -3,7 +3,7 @@ Loads the 3DHOP library and uses methods and functions to load 3D models in point cloud (*.ply) and nexus (*.nxz) form - into a clowder repository. + into a clowder repository. Language: JavaScript @@ -13,19 +13,21 @@ + 09/14/20 (cc): Created. + 10/14/20 (cc): Changing the source of the 3DHOP files and load them online + + 01/08/21 (cc): Loading libraries from https ****************************************************************/ + (function ($, Configuration) { - var useTab = Configuration.tab; - var referenceUrl = Configuration.url; - var confId = Configuration.id; - var fileId = Configuration.fileid; - var previewer = Configuration.previewer; + let useTab = Configuration.tab; + let referenceUrl = Configuration.url; + let confId = Configuration.id; + let fileId = Configuration.fileid; + let previewer = Configuration.previewer; - var fileName = $('#file-name-title').text().trim(); - var fileNameExtension = fileName.substr(fileName.length - 3); - var fileType; + let fileName = $('#file-name-title').text().trim(); + let fileNameExtension = fileName.substr(fileName.length - 3); + let fileType; if (fileNameExtension == "ply") { fileType = "ply"; @@ -34,11 +36,6 @@ fileType = "nexus"; } - //alert(fileName); - //alert(typeof(fileName)); - //alert(fileNameExtension); - //alert(fileType); - // print all attributes of Configuration object /*for (const property in Configuration) @@ -48,7 +45,7 @@ // add 3dhop.css to previewer - $(useTab).append(''); + $(useTab).append(''); // load various 3dhop attributes (such as the background image) for 3dhop @@ -66,22 +63,22 @@ id: 'toolbar' })); - $("#toolbar").append("
"); + $("#toolbar").append("
"); - $("#toolbar").append("
"); - $("#toolbar").append("
"); + $("#toolbar").append("
"); + $("#toolbar").append("
"); - $("#toolbar").append(""); - $("#toolbar").append("
"); + $("#toolbar").append(""); + $("#toolbar").append("
"); - $("#toolbar").append(""); - $("#toolbar").append("
"); + $("#toolbar").append(""); + $("#toolbar").append("
"); - $("#toolbar").append(""); - $("#toolbar").append("
"); + $("#toolbar").append(""); + $("#toolbar").append("
"); - $("#toolbar").append(""); - $("#toolbar").append(""); + $("#toolbar").append(""); + $("#toolbar").append(""); $('#3dhop').append($('
', { id: 'measure-box', @@ -119,18 +116,18 @@ $("#3dhop").append($('', { id: 'draw-canvas', - style: 'background-image: url("http://vcg.isti.cnr.it/3dhop/distribution/skins/backgrounds/dark.jpg")' + style: 'background-image: url("https://3dhop.net/distribution/latest/skins/backgrounds/dark.jpg")' })); // scripts holds all the 3dhop files - var scripts = ["spidergl.js", "nexus.js", "ply.js", "trackball_sphere.js", + let scripts = ["spidergl.js", "nexus.js", "ply.js", "trackball_sphere.js", "trackball_turntable.js", "trackball_pantilt.js", "trackball_turntable_pan.js", "init.js", "presenter.js"]; // append the http address where the files are located for (index = 0; index < scripts.length; index++) { - scripts[index] = "http://vcg.isti.cnr.it/3dhop/distribution/js/" + scripts[index]; + scripts[index] = "https://3dhop.net/distribution/latest/js/" + scripts[index]; } // load 3dhop into the current tab (old version) @@ -189,7 +186,7 @@ function getScripts(scripts, callback) { - var progress = 0; + let progress = 0; scripts.forEach(function (script) { //alert(script); @@ -199,7 +196,7 @@ function getScripts(scripts, callback) { }); } -var presenter = null; +let presenter = null; function setup3dhop(address, fileType) { presenter = new Presenter("draw-canvas"); From a6c16c69354f389357ee6e28d11849a205a22de0 Mon Sep 17 00:00:00 2001 From: Max Burnette Date: Fri, 29 Jan 2021 16:05:16 -0600 Subject: [PATCH 3/6] Move file between folders reindex bugfix (#156) * Reindex file & dataset on move * Update CHANGELOG.md Co-authored-by: Luigi Marini --- CHANGELOG.md | 2 ++ app/api/Folders.scala | 6 ++++++ 2 files changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7620fd34a..6afc3583d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/). ## Unreleased ### Fixed +- Files were not properly reindexed when the Move button was used to move a file into or out of a folder in a dataset. + This has been fixed. - When adding a file to a dataset by URL, prioritize the URL `content-type` header over the file content type established by looking at the file name extension. diff --git a/app/api/Folders.scala b/app/api/Folders.scala index 7c0d43e69..ea65b8fb1 100644 --- a/app/api/Folders.scala +++ b/app/api/Folders.scala @@ -302,6 +302,8 @@ class Folders @Inject() ( if(oldFolder.files.contains(fileId)) { folders.removeFile(oldFolder.id, fileId) folders.addFile(newFolder.id, fileId) + files.index(fileId) + datasets.index(datasetId) Ok(toJson(Map("status" -> "success", "fileName" -> file.filename, "folderName" -> newFolder.name))) } else { BadRequest("Failed to move file. The file with id: " + file.id.stringify + " isn't in folder with id: " + oldFolder.id.stringify ) @@ -315,6 +317,8 @@ class Folders @Inject() ( if(dataset.files.contains(fileId)) { folders.addFile(newFolder.id, fileId) datasets.removeFile(datasetId, fileId) + files.index(fileId) + datasets.index(datasetId) Ok(toJson(Map("status" -> "success", "fileName" -> file.filename, "folderName" -> newFolder.name))) } else { BadRequest("Failed to move file. The file with id: " + file.id.stringify + "Isn't in dataset with id: " + dataset.id.stringify ) @@ -353,6 +357,8 @@ class Folders @Inject() ( if(folder.files.contains(fileId)) { datasets.addFile(datasetId, file) folders.removeFile(oldFolderId, fileId) + files.index(fileId) + datasets.index(datasetId) Ok(toJson(Map("status" -> "success", "fileName"-> file.filename ))) } else { BadRequest("The file you are trying to move isn't in the folder you are moving it from.") From bdead0e56bbdfe20ea7046069afffdbd14b76cbf Mon Sep 17 00:00:00 2001 From: Rob Kooper Date: Fri, 29 Jan 2021 16:34:14 -0600 Subject: [PATCH 4/6] Fix user count (#158) * increment user count (fixes #136) * fix urls for login services * add note about fixcount script Co-authored-by: Luigi Marini --- CHANGELOG.md | 9 ++++++++- app/services/SecureSocialEventListener.scala | 7 ++++--- conf/securesocial.conf | 8 ++++---- 3 files changed, 16 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6afc3583d..13d638678 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,9 +6,16 @@ and this project adheres to [Semantic Versioning](http://semver.org/). ## Unreleased +Google will no longer work as login provider, we are working on this issue #157. + +If non local accounts are used the count can be wrong. Use the [fixcounts](https://github.com/clowder-framework/clowder/blob/develop/scripts/updates/fix-counts.js) +script to fix this. + ### Fixed +- Error logging in with orcid due to changed URL #91 +- Fixed error in url for twitter login +- Count of users is not correct if using anything else but local accounts #136 - Files were not properly reindexed when the Move button was used to move a file into or out of a folder in a dataset. - This has been fixed. - When adding a file to a dataset by URL, prioritize the URL `content-type` header over the file content type established by looking at the file name extension. diff --git a/app/services/SecureSocialEventListener.scala b/app/services/SecureSocialEventListener.scala index ac7ae9ec2..b3314c850 100644 --- a/app/services/SecureSocialEventListener.scala +++ b/app/services/SecureSocialEventListener.scala @@ -10,11 +10,11 @@ import securesocial.core.providers.UsernamePasswordProvider class SecureSocialEventListener(app: play.api.Application) extends EventListener { override def id: String = "SecureSocialEventListener" + lazy val userService: UserService = DI.injector.getInstance(classOf[UserService]) + lazy val spaceService: SpaceService = DI.injector.getInstance(classOf[SpaceService]) + lazy val appConfig: AppConfigurationService = DI.injector.getInstance(classOf[AppConfigurationService]) def onEvent(event: Event, request: RequestHeader, session: Session): Option[Session] = { - val userService: UserService = DI.injector.getInstance(classOf[UserService]) - val spaceService: SpaceService = DI.injector.getInstance(classOf[SpaceService]) - event match { case e: SignUpEvent => { userService.findByIdentity(event.user) match { @@ -40,6 +40,7 @@ class SecureSocialEventListener(app: play.api.Application) extends EventListener val subject = s"[${AppConfiguration.getDisplayName}] new user signup" val body = views.html.emails.userSignup(user)(request) util.Mail.sendEmailAdmins(subject, Some(user), body) + appConfig.incrementCount('users, 1) } user.email match { case Some(e) => spaceService.processInvitation(e) diff --git a/conf/securesocial.conf b/conf/securesocial.conf index 3be9fa8a9..9afb107b9 100644 --- a/conf/securesocial.conf +++ b/conf/securesocial.conf @@ -79,9 +79,9 @@ securesocial { ssl=false twitter { - requestTokenUrl="https://twitter.com/oauth/request_token" - accessTokenUrl="https://twitter.com/oauth/access_token" - authorizationUrl="https://twitter.com/oauth/authenticate" + requestTokenUrl="https://api.twitter.com/oauth/request_token" + accessTokenUrl="https://api.twitter.com/oauth/access_token" + authorizationUrl="https://api.twitter.com/oauth/authorize" consumerKey=your_consumer_key consumerSecret=your_consumer_secret } @@ -135,7 +135,7 @@ securesocial { orcid { authorizationUrl="https://orcid.org/oauth/authorize" - accessTokenUrl="https://pub.orcid.org/oauth/token" + accessTokenUrl="https://orcid.org/oauth/token" clientId=your_client_id clientSecret=your_client_secret scope="/authenticate" From ef63e79c448dff0c8b17202cb7df23b822f3ebce Mon Sep 17 00:00:00 2001 From: Luigi Marini Date: Mon, 1 Feb 2021 14:03:36 -0600 Subject: [PATCH 5/6] Fixes #160. Use css `word-wrap: break-word;` where possible to break words that are too long. (#163) This issue is obvious with files names that are very long. Applied to fix to collectios, datasets, spaces titles as well where I saw an issue. There might be more places where this could be applied. --- CHANGELOG.md | 1 + app/views/collectionofdatasets.scala.html | 2 +- app/views/datasets/listitem.scala.html | 2 +- app/views/file.scala.html | 2 +- app/views/files/linkDownload.scala.html | 2 +- app/views/files/listitem.scala.html | 2 +- app/views/files/share.scala.html | 2 +- app/views/spaces/spaceAllocation.scala.html | 2 +- conf/messages | 2 +- public/stylesheets/main.css | 4 ++++ 10 files changed, 13 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 13d638678..eaea25bbc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,7 @@ script to fix this. - Files were not properly reindexed when the Move button was used to move a file into or out of a folder in a dataset. - When adding a file to a dataset by URL, prioritize the URL `content-type` header over the file content type established by looking at the file name extension. +- Wrap words across lines to stay within interface elements. [#160](https://github.com/clowder-framework/clowder/issues/160) ## 1.14.0 - 2021-01-07 diff --git a/app/views/collectionofdatasets.scala.html b/app/views/collectionofdatasets.scala.html index dcb4e88a5..a5cdcfa82 100644 --- a/app/views/collectionofdatasets.scala.html +++ b/app/views/collectionofdatasets.scala.html @@ -89,7 +89,7 @@
-

@Html(collection.name)

+

@Html(collection.name)

@if(!collection.trash && Permission.checkPermission(Permission.EditCollection, ResourceRef(ResourceRef.collection, collection.id))) {

diff --git a/app/views/datasets/listitem.scala.html b/app/views/datasets/listitem.scala.html index b0453f078..cc4709c91 100644 --- a/app/views/datasets/listitem.scala.html +++ b/app/views/datasets/listitem.scala.html @@ -29,7 +29,7 @@
- @Html(dataset.name) + @Html(dataset.name)
diff --git a/app/views/file.scala.html b/app/views/file.scala.html index 3fc665bfe..1e3804727 100644 --- a/app/views/file.scala.html +++ b/app/views/file.scala.html @@ -257,7 +257,7 @@
-

@Html(file.filename)

+

@Html(file.filename)

@if(Permission.checkPermission(Permission.EditFile, ResourceRef(ResourceRef.file, file.id))) {
diff --git a/app/views/files/linkDownload.scala.html b/app/views/files/linkDownload.scala.html index e218ed243..697eaa586 100644 --- a/app/views/files/linkDownload.scala.html +++ b/app/views/files/linkDownload.scala.html @@ -3,7 +3,7 @@ @main("Download File") {
-

@Html(file.filename)

+

@Html(file.filename)

diff --git a/app/views/files/listitem.scala.html b/app/views/files/listitem.scala.html index a4319413b..83cbb53f4 100644 --- a/app/views/files/listitem.scala.html +++ b/app/views/files/listitem.scala.html @@ -23,7 +23,7 @@
-

@file.filename

+

@file.filename

    diff --git a/app/views/files/share.scala.html b/app/views/files/share.scala.html index e4dfedadd..f99678ede 100644 --- a/app/views/files/share.scala.html +++ b/app/views/files/share.scala.html @@ -3,7 +3,7 @@ @main("Share") {
    -

    @Html(file.filename)

    +

    @Html(file.filename)

    diff --git a/app/views/spaces/spaceAllocation.scala.html b/app/views/spaces/spaceAllocation.scala.html index dbefc12ce..f68def0ef 100644 --- a/app/views/spaces/spaceAllocation.scala.html +++ b/app/views/spaces/spaceAllocation.scala.html @@ -37,7 +37,7 @@

    @Messages("a.contains.b", Messages("spaces.title"), Messages("dataset.title"

    @if(resourceType == ResourceRef.dataset) { diff --git a/conf/messages b/conf/messages index d142f5a26..862ebbfb3 100644 --- a/conf/messages +++ b/conf/messages @@ -12,7 +12,7 @@ trial.title = Trial {0} owner.label=Owner owner.title = {0}''s {1} owner.in.resource.title={0}''s {1} in {2} {4} -resource.in.title= {0} in {1} {3} +resource.in.title= {0} in {1} {3} list.title= {0} # Basic terms diff --git a/public/stylesheets/main.css b/public/stylesheets/main.css index 583aacef5..d86877488 100644 --- a/public/stylesheets/main.css +++ b/public/stylesheets/main.css @@ -579,6 +579,10 @@ p.error, span.error { border-bottom: 1px solid #E1E1E8; } +.space-title { + word-wrap: break-word; +} + .space-col-right { padding-left: 50px; } From 92773e5f61a125b84e5c3ff5ce7b32ce014da3f3 Mon Sep 17 00:00:00 2001 From: Luigi Marini Date: Mon, 1 Feb 2021 15:29:33 -0600 Subject: [PATCH 6/6] v1.14.1 updates of versions and CHANGELOG.md. --- CHANGELOG.md | 15 +++++++-------- doc/src/sphinx/conf.py | 2 +- project/Build.scala | 2 +- public/swagger.yml | 2 +- 4 files changed, 10 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index eaea25bbc..0e4190b59 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,20 +4,19 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/) and this project adheres to [Semantic Versioning](http://semver.org/). -## Unreleased +## 1.14.1 - 2021-02-02 -Google will no longer work as login provider, we are working on this issue #157. - -If non local accounts are used the count can be wrong. Use the [fixcounts](https://github.com/clowder-framework/clowder/blob/develop/scripts/updates/fix-counts.js) +- Google will no longer work as login provider, we are working on this issue [#157](https://github.com/clowder-framework/clowder/issues/157). +- If non local accounts are used the count can be wrong. Use the [fixcounts](https://github.com/clowder-framework/clowder/blob/develop/scripts/updates/fix-counts.js) script to fix this. ### Fixed -- Error logging in with orcid due to changed URL #91 -- Fixed error in url for twitter login -- Count of users is not correct if using anything else but local accounts #136 +- Error logging in with Orcid due to changed URL. [#91](https://github.com/clowder-framework/clowder/issues/91) +- Fixed error in url for Twitter login. +- Users count was not correct if using anything else but local accounts. [#136](https://github.com/clowder-framework/clowder/issues/136) - Files were not properly reindexed when the Move button was used to move a file into or out of a folder in a dataset. - When adding a file to a dataset by URL, prioritize the URL `content-type` header over the file content type established - by looking at the file name extension. + by looking at the file name extension. [#139](https://github.com/clowder-framework/clowder/issues/139) - Wrap words across lines to stay within interface elements. [#160](https://github.com/clowder-framework/clowder/issues/160) ## 1.14.0 - 2021-01-07 diff --git a/doc/src/sphinx/conf.py b/doc/src/sphinx/conf.py index a9d15621e..7f8fac5b3 100644 --- a/doc/src/sphinx/conf.py +++ b/doc/src/sphinx/conf.py @@ -22,7 +22,7 @@ author = 'Luigi Marini' # The full version, including alpha/beta/rc tags -release = '1.14.0' +release = '1.14.1' # -- General configuration --------------------------------------------------- diff --git a/project/Build.scala b/project/Build.scala index 3a6ebe790..d13541101 100644 --- a/project/Build.scala +++ b/project/Build.scala @@ -13,7 +13,7 @@ import NativePackagerKeys._ object ApplicationBuild extends Build { val appName = "clowder" - val version = "1.14.0" + val version = "1.14.1" val jvm = "1.7" def appVersion: String = { diff --git a/public/swagger.yml b/public/swagger.yml index 6c462b213..7afe5e842 100644 --- a/public/swagger.yml +++ b/public/swagger.yml @@ -9,7 +9,7 @@ info: Clowder is a customizable and scalable data management system to support any data format and multiple research domains. It is under active development and deployed for a variety of research projects. - version: 1.14.0 + version: 1.14.1 termsOfService: https://clowder.ncsa.illinois.edu/clowder/tos contact: name: Clowder