Visualize 3D models locally without relying on sketchfab iframe #2091

luistoptal · 2024-11-14T08:57:48Z

Pull request for issue: #2054

This is a pull request for the following functionalities:

Add three.js widget to visualize dataset 3D models without relying on sketchfab. This module is selfcontained, can be reused anywhere on the site and takes a list of external links to 3D models as data input

How to test?

The following steps describe how I test this widget locally (if you know of a better way, please be my guest and I would love to know 🙂):

download the models from the following urls to your computer (or feel free to use any files you want as long as they are LAS, STL, PLY or OBJ)
Manually add the files to the assets folder, e.g: /assets/test-models/YOUR_FILES
http://gigadb.gigasciencejournal.com/adminExternalLink/create add the files one by one as external links to any one of the published datasets (e.g. 100006). In the url field you should add something like this: /assets/test-models/filename, in External Link Type select "3d Models"
Navigate to the dataset page, e.g.: http://gigadb.gigasciencejournal.com/dataset/100006
You should see the "3D models" tab. When you open it, you should see the models you added in the dropdown
The initial state of the viewer should be no loaded modal, and a visible play button
when you click play, a model should be loaded and you should be able to interact with it
the "help" and "fullscreen" buttons should work as intended, either with mouse or keyboard
if you toggle fullscreen and / or resize the window, the viewer should be fully responsive and the model should remain visible
if you select another model in the selector, the model should be immediately loaded

You can test error states if you repeat the above process by adding a file that is not a valid 3d model or not a LAS, STL, PLY or OBJ file (or you can just add a throw new Error('Ooops') in the line before await loadModel(file);)

Example of error message:

it would be critical to test in a production-like environment with files / external links mimicking the production environment

How have functionalities been implemented?

3D models were previously visualized within an iframe taking a sketchfab url as input. This PR gets rid of the sketchfab iframe and replaces it with a JS widget that makes use of the three.js library to load and display an interactive 3D model

The main requirement is to load STL and OBJ file formats, and also (less common) PLY and LAS

Different options to locally load the model were considered (refer to the original ticket for some). I decided to use three.js because:

It is widely popular and well known, essentially being the standard way to render 3D files in JS world
It supports many formats via built-in loaders (LAS being an exception)
It supports / will support WebXR via plugins. Note that WebXR is an experimental API and was not considered in this PR
For a simple use case such as rendering a model with no animations and with basic interactions it is relatively simple to implement

Other options I checked had very low or null adoption and / or did not fully cover the use case. I opted for the widely adopted solution.

The changes can be grouped as follows:

Mainly, everything related to the model viewer is within protected/js/model-viewer
The partial protected/views/shared/_model_viewer.php renders the html for the widget, and loads the necessary scripts;
- three.js imports are added to the head as a import map. That way, the three.js library can be imported as a module in the widget scripts
- Yii::app()->assetManager->forceCopy = YII_DEBUG; makes sure the assets are rebuilt and not cached in development (NOTE: I am unclear on whether this targets specifically local dev, is this the correct solution? If this is a blocker, this line can be removed, but I found it necessary to add this line during local dev)
- widget scripts are added to the body
- modelviewer is initialized
The partial is rendered in the dataset page protected/views/dataset/view.php. This page contains a switch statement that renders elements such an iframe depending on the external link types. In the switch statement, the links are handled one by one. In the case of 3D models, I deleted that case from the switch statement, and passed them to the partial as data property. Note that I had to do this because the partial, rather than handling individual links one by one, takes them all to build a select input from which to select the model to load
less/modules/model-viewer.less takes care of all the styles related to the viewer

That covers all the changes outside of the protected/js/model-viewer folder, now related to the widget itself

first, the idea is to abstract implementation details away from the consuming view. I suggest looking at the jsdocs decorating the modelViewer function and dig deeper only if curious about implementation details
the widget is split into two core modules
- ui: takes care of updating the state of the UI depending on the state of the viewer (i.e.: loading states, model loaded, no model loaded, error state)
  - the UI has 4 states:
    - idle: no model is loaded, a play button to load the currently selected model (by default, the first in the select input) should be visible
    - loading: a loading spinner should be displayed while loading a model
    - error: a danger alert should be displayed at the bottom with a relevant message. Use can select another model or click "play" again
    - success: model is loaded and should be interactive
- viewer: it uses the three.js library to render a 3D model, and takes care of all related things (setting up scene, lights, cameras, controls, loading models...)
  - It is expected that all models are in one of LAS, STL, PLY or OBJ file formats. If another format is needed, the corresponding loader for such file format should be added (protected/js/model-viewer/viewer/components/models/loader.js)
  - among other things, this module takes care of making the viewer fully responsive
  - LAS files (data point files) needed a custom implementation, with a visualization looking like a relatively crude set of points. I see sketchfab doesn't do much better, and I understand LAS files are a minority (?) so I decided not to dig more on that field
  - the "look" of the 3D models is fully customizable. I opted for a metallic effect somewhat similar to sketchfab. The visuals can be fine-tuned / changed if needed as three.js is fully customizable
the entry point modelViewer connects the two modules and initializes them

Any issues with implementation?

I am unable to test on production or in a production-like environment
it assumes the incoming files are defined as ExternalLinks and follow the interface protected/interfaces/DatasetExternalLinksInterface.php, if the requirement is different, for example, a different model such as protected/models/File.php, it should be relatively easy to adapt
it assumes the site can fetch and load the incoming external links without cors issues, in short, cors is not handled in any way
In Yii, what is the right way to access the current environment from within a .js file? For example to determine the isProduction value in protected/js/model-viewer/helpers/logger.js

Any changes to automated tests?

All tests pass

kencho51 · 2024-11-18T06:44:21Z

Hi @luistoptal,

prasie: Was able to associate this https://sketchfab.com/3d-models/beet-03-c63d100fb76042ecab2ec87f73da323e stl file to dataset 100006, and display it on Chrome browser.

question: The 3D image cannot be shown in Safari nor Firefox browsers, can you help further look into it?

Safari

Firefox

luistoptal · 2024-11-18T07:35:07Z

@kencho51 looking at the images, it looks as if the CSS is not loaded. Because not only the 3D model is not shown but the buttons look as if the custom styles applied to them were not working. Can you try:

re-run the ./up.sh script (but I assume you already did that)
rebuild the .less styles using docker-compose run --rm less (if you run ./up.sh, it should already be done though)
Disable Firefox and Safari browser cache and hard refresh the page

kencho51 · 2024-11-18T07:53:48Z

Hi @luistoptal,

Disable Firefox and Safari browser cache and hard refresh the page

After hard refresh/empty the cache in safari and firefox, the 3D image can be displayed now.

kencho51 · 2024-11-19T07:50:43Z

Hi @luistoptal,

question: Would it be better to have acceptance tests/accessibility tests as there is an extra 3D Models tab and a 3D graph is displayed?
suggestion: For the 3D images source, we could put all the images in a s3 bucket and make them public, and then input the bucket url when trying to create external link, as below:

I have created a test bucket with 3 3D images:

@rija, @pli888 , what do you think?

praise: I was able to deploy this PR to my staging, and display the 3D images from the above files stored in the s3 bucket.

luistoptal · 2024-11-20T01:43:31Z

@kencho51 that would work since the viewer simply loads and displays files accessible via public links, thanks for adding these files, now they can be used to update the tests

there is one test for the 3D models tab already present, it hadn't been updated

luistoptal · 2024-11-20T05:18:15Z

I extended the test that checks for the 3D Model tab

kencho51 · 2024-11-20T08:12:04Z

Hi @luistoptal,

issue: I have further tested this PR in my production, and the 3D image actually cannot be displayed properly as below:

and figured out that the following external scripts were blocked by the CSP:

Loading failed for the module with source “https://cdn.jsdelivr.net/npm/[email protected]/examples/jsm/loaders/STLLoader.js”. [100006](https://ec2-staging.gigadb.link/dataset/100006)
Loading failed for the module with source “https://cdn.jsdelivr.net/npm/[email protected]/examples/jsm/loaders/OBJLoader.js”. [100006](https://ec2-staging.gigadb.link/dataset/100006)
Loading failed for the module with source “https://cdn.jsdelivr.net/npm/[email protected]/examples/jsm/loaders/PLYLoader.js”. [100006](https://ec2-staging.gigadb.link/dataset/100006)
Loading failed for the module with source “https://cdn.jsdelivr.net/npm/@loaders.gl/[email protected]/+esm”. [100006](https://ec2-staging.gigadb.link/dataset/100006)
Loading failed for the module with source “https://cdn.jsdelivr.net/npm/@loaders.gl/[email protected]/+esm”. [100006](https://ec2-staging.gigadb.link/dataset/100006)
Loading failed for the module with source “https://cdn.jsdelivr.net/npm/[email protected]/examples/jsm/loaders/STLLoader.js”. [100006](https://ec2-staging.gigadb.link/dataset/100006)
Loading failed for the module with source “https://cdn.jsdelivr.net/npm/[email protected]/examples/jsm/loaders/OBJLoader.js”. [100006](https://ec2-staging.gigadb.link/dataset/100006)
Loading failed for the module with source “https://cdn.jsdelivr.net/npm/[email protected]/examples/jsm/loaders/PLYLoader.js”. [100006](https://ec2-staging.gigadb.link/dataset/100006)
Loading failed for the module with source “https://cdn.jsdelivr.net/npm/@loaders.gl/[email protected]/+esm”. [100006](https://ec2-staging.gigadb.link/dataset/100006)
Loading failed for the module with source “https://cdn.jsdelivr.net/npm/@loaders.gl/[email protected]/+esm”.

suggestion: The fix could be made by adding https://cdn.jsdelivr.net in the script-src directive in ops/configuration/nginx-conf/sites/nginx.target_deployment.https.conf.dist.

kencho51 · 2024-11-20T08:50:19Z

Hi @luistoptal,

issue: The gigadb-website/features is a legacy dir that waiting for remove.
suggestion: The acceptance test should be implemented in gigadb-website/tests/acceptance/ which would use the codeception framework.
suggestion: For the test scenario, usually we would implement in a way that only contains 1 Given, 1 When and 1 Then, which would make the test simpler and easier to test, for example:

Scenario: 3D Models tab 
  Given I am not logged in to Gigadb web site
  And I have added "3D Viewer" link "https://s3.ap-northeast-1.wasabisys.com/test-gigadb-datasets/3d-models/100006/GeoB8502_865cm_Shell-4.obj" to dataset "101001"
  When I go to "/dataset/101001"
  Then I should see "3D Models" tab with text "3D Models:"

Scenario: 3D model drop down list
  Given I am not logged in to Gigadb web site
  When I click on the "3D Models" button
  Then I should see a "select.test-model-selector" element
  And I should see a "select.test-model-selector > option" element
  And I should see "GeoB8502_865cm_Shell-4.obj" in the "select.test-model-selector > option" element

luistoptal · 2024-11-20T10:11:31Z

@kencho51 I think it would be simpler if one dataset is created already with a 3D model for acceptance testing (one of the S3 links you posted above), what would be the right place to add this? (I see many sql files that are probably used for testing, but not sure which ones are ok and which ones should be deprecated)

kencho51 · 2024-11-21T03:45:49Z

Hi @luistoptal,

suggestion:

For the test, you could:

Add the following line to the file gigadb-website/data/dev/external_link.csv:

1527,8,https://s3.ap-northeast-1.wasabisys.com/test-gigadb-datasets/3d-models/100006/GeoB8502_865cm_Shell-4.obj,5

Then spin up gigadb website ./up.sh, it will execute the migration scripts and update the test database with above entry
Then confirm by going to http://gigadb.gigasciencejournal.com/dataset/100006

Then for the implementation, you could add the scenarios to gigadb-website/tests/acceptance/DatasetView.feature as

  @ok @issue-2054
  Scenario: 3D Models tab
    Given I have not signed in
    When I am on "/dataset/100006"
    Then I should see "3D Models"

  @wip @issue-2054
  Scenario: 3D model drop down list
    Given I have not signed in
    When I follow "3D Models"
    Then I should see "3D Models:"
    And I should see "Select a model"
    And I should see "GeoB8502_865cm_Shell-4.obj"

And then you execute the test as below:

% docker-compose run --rm codecept run --no-redirect -g issue-2054 acceptance -v
Codeception PHP Testing Framework v5.0.4 https://helpukrainewin.org
[Groups] issue-2054 

Acceptance Tests (2) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Modules: \Helper\Acceptance, Asserts, Db, WebDriver
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
✔ a user visit the dataset page: 3D Models tab (4.72s)
✔ a user visit the dataset page: 3D model drop down list (3.53s)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Time: 00:25.298, Memory: 12.00 MB

OK (2 tests, 4 assertions)

issue: Got this error in production three.module.js:44842 Refused to connect to 'https://s3.ap-northeast-1.wasabisys.com/test-gigadb-datasets/3d-models/100006/GeoB8502_865cm_Shell-4.obj' because it violates the following Content Security Policy directive: "default-src 'self' 'unsafe-inline' 'unsafe-eval' http://gigadb.org http://penguin.genomics.cn https://www.rosaceae.org *.protocols.io https://sketchfab.com https://codeocean.com *.hypothes.is *.datatables.net *.cloudflare.com". Note that 'connect-src' was not explicitly set, so 'default-src' is used as a fallback.

suggestion: The fix could be made by adding https://s3.ap-northeast-1.wasabisys.com in the default-src directive in ops/configuration/nginx-conf/sites/nginx.target_deployment.https.conf.dist.

kencho51 · 2024-11-21T08:45:14Z

Hi @luistoptal,

Please use the model files from these links:

https://s3.ap-northeast-1.wasabisys.com/gigadb-datasets/dev/tests/3d-models/100006/leaf_01.las
https://s3.ap-northeast-1.wasabisys.com/gigadb-datasets/dev/tests/3d-models/100006/beet_03.stl
https://s3.ap-northeast-1.wasabisys.com/gigadb-datasets/dev/tests/3d-models/100006/GeoB8502_865cm_Shell-4.obj

luistoptal · 2024-11-21T10:10:06Z

@kencho51 I added your suggestions, that was very helpful

for some reason I had to shut down docker compose and re run up.sh to register the db entires

I also added the wasaby source to all default-src entires (all the 3 of them)

kencho51 · 2024-11-22T02:11:21Z

features/dataset-view.feature

 	@ok
-	Scenario:  3D Viewer
+	Scenario: 3D Viewer
 		Given I am not logged in to Gigadb web site
-		And I have added "3D Viewer" link "https://sketchfab.com/models/ea49d0dd500647cbb4b61ad5ca9e659a" to dataset "101001"
+		And I have added "3D Viewer" link "https://s3.ap-northeast-1.wasabisys.com/test-gigadb-datasets/3d-models/100006/GeoB8502_865cm_Shell-4.obj" to dataset "101001"
 		When I go to "/dataset/101001"
 		Then I should see "3D Models" tab with text "3D Models:"
+    When I click on the "3D Models" button
+    Then I should see a "select.test-model-selector" element
+    And I should see a "select.test-model-selector > option" element
+    And I should see "GeoB8502_865cm_Shell-4.obj" in the "select.test-model-selector > option" element



suggestion: This scenario can be removed, since it has been moved to tests/acceptance/DatasetView.feature already.

kencho51 · 2024-11-22T02:21:14Z

sql/3D_Viewer_101001_test_data.sql

+insert into external_link_type(id, name)
+select 5, '3D Models'
+where not exists (
+    select 1 from external_link_type
+    where id = 5 and name = '3D Models'
+);
+insert into external_link(dataset_id, url, external_link_type_id) values(80,'https://s3.ap-northeast-1.wasabisys.com/test-gigadb-datasets/3d-models/100006/GeoB8502_865cm_Shell-4.obj',5);


suggestion: The changes at here should be reverted, as only data from data/dev will be used.

I am deleting this file as I think it serves no purpose anymore

kencho51 · 2024-11-22T02:23:49Z

tests/acceptance/DatasetView.feature

+    Then I should see "3D Models"
+
+  @wip @issue-2054


suggestion: Change the @wip to @ok if the scenario is passing, the wip means work in progress.

luistoptal · 2024-11-22T02:42:55Z

features/bootstrap/DatasetViewContext.php

@@ -413,10 +413,7 @@ public function iShouldNotSeeTabWithTable($arg1, TableNode $table)
     */
    public function iHaveAddedLinkToDataset($arg1, $arg2, $arg3)
    {
-        if ("3D Viewer" == $arg1 ) {


@kencho51 since I deleted the 3D Models test, I am also deleting this switch case

kencho51

Hi @luistoptal,

With all the requested changes fixed and the 3D models can be displayed in my production, so LGTM, happy to approve.

luistoptal · 2024-11-22T08:13:58Z

I noticed a bug I need to fix

luistoptal · 2024-11-22T08:38:11Z

fixed

luistoptal added 18 commits November 10, 2024 14:33

feat: able to render 3d models locally

56afecb

feat: use jquery

b65a3ab

feat: scaffold model viewer refactor

5d3bdd1

feat: able to visualize rendered models

d16a28b

feat: model looks similar to sketchfab

e572f03

feat: adds multitud of small refactorings

7ced357

feat: load ply, obj and las models

97bf32f

feat: responsive canvas and refactored ui code

d76ca05

feat: improves help controls

31d6c99

feat: adds fullscreen btn

4c15ea4

feat: refactors root ui script, colocates selectors, adds js docs

ba975be

feat: integrates model viewer into dataset page

33e4a05

feat: improves a11y for model loading state

e83abd2

chore: remove debug logs

5053b67

chore: up log

f051629

chore: undoes ignored temp models folder

e858b73

feat: adds jsdocs

e064ace

chore: Merge branch 'develop' into new-feature/2054-3d-image-viewer

bd824ca

luistoptal marked this pull request as ready for review November 14, 2024 11:01

luistoptal requested review from kencho51, pli888 and rija as code owners November 14, 2024 11:01

luistoptal added 2 commits November 20, 2024 09:30

chore: update 3d models tab test

d3c9334

feat: tests for model selector

6a8e1e7

luistoptal added 2 commits November 20, 2024 16:19

fix: undoes bug introduced by trim func

336b89a

chore: add https://cdn.jsdelivr.net to nginx config

5bd9b1c

luistoptal added 2 commits November 21, 2024 17:08

feat: add 3d models to test db and tests

bf0e900

chore: adds model source to nginx

d0e4167

chore: Merge branch 'develop' into new-feature/2054-3d-image-viewer

226be5f

kencho51 reviewed Nov 22, 2024

View reviewed changes

chore: cleanup tests

3eb5306

luistoptal commented Nov 22, 2024

View reviewed changes

kencho51 approved these changes Nov 22, 2024

View reviewed changes

luistoptal added 2 commits November 22, 2024 15:25

fix: displays 3d models only in 3d models tab panel

b004b38

fix: render model viewer only once

60ae2f7

luistoptal mentioned this pull request Nov 23, 2024

Visualize Hi-C files using Juicebox without relying on an iframe #2106

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Visualize 3D models locally without relying on sketchfab iframe #2091

Visualize 3D models locally without relying on sketchfab iframe #2091

luistoptal commented Nov 14, 2024 •

edited

Loading

kencho51 commented Nov 18, 2024 •

edited

Loading

luistoptal commented Nov 18, 2024

kencho51 commented Nov 18, 2024

kencho51 commented Nov 19, 2024

luistoptal commented Nov 20, 2024

luistoptal commented Nov 20, 2024

kencho51 commented Nov 20, 2024

kencho51 commented Nov 20, 2024 •

edited

Loading

luistoptal commented Nov 20, 2024

kencho51 commented Nov 21, 2024 •

edited

Loading

kencho51 commented Nov 21, 2024

luistoptal commented Nov 21, 2024

kencho51 Nov 22, 2024

kencho51 Nov 22, 2024

luistoptal Nov 22, 2024

kencho51 Nov 22, 2024

luistoptal Nov 22, 2024

kencho51 left a comment

luistoptal commented Nov 22, 2024

luistoptal commented Nov 22, 2024

Visualize 3D models locally without relying on sketchfab iframe #2091

Are you sure you want to change the base?

Visualize 3D models locally without relying on sketchfab iframe #2091

Conversation

luistoptal commented Nov 14, 2024 • edited Loading

Pull request for issue: #2054

How to test?

How have functionalities been implemented?

Any issues with implementation?

Any changes to automated tests?

kencho51 commented Nov 18, 2024 • edited Loading

Safari

Firefox

luistoptal commented Nov 18, 2024

kencho51 commented Nov 18, 2024

kencho51 commented Nov 19, 2024

luistoptal commented Nov 20, 2024

luistoptal commented Nov 20, 2024

kencho51 commented Nov 20, 2024

kencho51 commented Nov 20, 2024 • edited Loading

luistoptal commented Nov 20, 2024

kencho51 commented Nov 21, 2024 • edited Loading

kencho51 commented Nov 21, 2024

luistoptal commented Nov 21, 2024

kencho51 Nov 22, 2024

Choose a reason for hiding this comment

kencho51 Nov 22, 2024

Choose a reason for hiding this comment

luistoptal Nov 22, 2024

Choose a reason for hiding this comment

kencho51 Nov 22, 2024

Choose a reason for hiding this comment

luistoptal Nov 22, 2024

Choose a reason for hiding this comment

kencho51 left a comment

Choose a reason for hiding this comment

luistoptal commented Nov 22, 2024

luistoptal commented Nov 22, 2024

luistoptal commented Nov 14, 2024 •

edited

Loading

kencho51 commented Nov 18, 2024 •

edited

Loading

kencho51 commented Nov 20, 2024 •

edited

Loading

kencho51 commented Nov 21, 2024 •

edited

Loading