Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

save metadata of aws result files to firebase #201

Closed
wants to merge 67 commits into from
Closed

save metadata of aws result files to firebase #201

wants to merge 67 commits into from

Conversation

rugeli
Copy link
Collaborator

@rugeli rugeli commented Oct 6, 2023

Problem

What is the problem this work solves, including
closes #194

Solution

What I/we did to solve this problem

  • wrote a function to save metadata of aws results files to firebase. The stored metadata includes fields such as "username", "timestamp", "aws_url", with the recipe's unique name being as the key
  • handlers are initiated using the DATABASE_IDS enum, ensuring that the DBRecipeHandler remains agnostic
  • rearranged firebase handler and aws handler to
    • ensure they are initialized only once
    • improve readability

Type of change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)

Steps to Verify:

  1. pack either local or remote recipes to see if the result files has been uploaded to S3 and saved metadata to firebase

Screenshots (optional):

Screenshot 2023-10-06 at 11 24 58 AM

mogres and others added 29 commits August 16, 2023 15:34
@mogres
Working variable count distributions
711bd8e
@mogres
* add radius to molecules array
8a00bbd 
* update simularium helper to use radius in molecules
* update max and min radii
* add size options to ingredient initialization
* refactor distribution value extraction
@mogres
Test recipe and config for variable size
c5aa16b
@mogres
add radius to ingredient array dict
de95dfe
@rugeli
turn off resolving inheritance while uploading
9bcd67f
@rugeli
able to upload recipes having "inherit" key
5c14f8a
@rugeli
get download and pack to work, refactors needed
dde4a72
@meganrm
add test variable size recipe
316fee7
@meganrm
create packed objects class
945eccb
@meganrm
Merge branch 'feature/variable_size' of https://github.com/mesoscope/…
283015e 
…cellpack into feature/results-in-memory
@meganrm
removing .molecules
7db8bec
@meganrm
pack variable size ingredient successfully
8eb3190
@meganrm
Merge branch 'main' of https://github.com/mesoscope/cellpack into fea…
a23bdde 
…ture/results-in-memory
@meganrm
pack sphere_tree ingredient
fb3dcb9
@meganrm
DRY code
4c47c8c
@meganrm
store ingredient in packed object
3b97426
@meganrm
change to compartment_id
b665e77
@meganrm
cleanup
7836419
@meganrm
add second example
5541d22
@rugeli
Merge branch 'main' of https://github.com/mesoscope/cellpack into fea…
49446bc 
…ture/run-inherited-objects
@rugeli
refactors
00628e9
@rugeli
Merge branch 'main' of https://github.com/mesoscope/cellpack into fea…
c8bcccf 
…ture/run-inherited-objects
@rugeli
Merge branch 'feature/run-recipes-from-firebase' into feature/run-inh…
d161baa 
…erited-objects
@rugeli
Merge branch 'feature/run-inherited-objects' of https://github.com/me…
3d82959 
…soscope/cellpack into feature/run-inherited-objects
@rugeli
initial draft
00601e3
@rugeli
reorder methods in firebase handler
5318f33
@rugeli
avoid initializing app twice
eac5704
@rugeli
modularize firebase handler
1539654
@rugeli
modularize AWSHandler & refactors
e58dbe5
@github-actions
Copy link

github-actions bot commented Oct 6, 2023 •
edited
Loading

Packing analysis report

Analysis for packing results located at cellpack/tests/outputs/test_spheres/spheresSST

Ingredient name Encapsulating radius Average number packed
ext_A 25 236.0

Packing image

Packing image

Distance analysis

Expected minimum distance: 50.00
Actual minimum distance: 50.01

Ingredient key Pairwise distance distribution
ext_A Distance distribution ext_A

mogres and others added 14 commits October 10, 2023 11:46
@mogres
* use kwargs to overload voxelization function
d480893 
* use object instance instead of class to create voxelization
* formatting
@mogres
fix tests
9065f17
@mogres
removed afvi from get_partners which was erroring out
e9dc5b6
@mogres
remove afvi argument causing partner packing to break
4be3fed
@meganrm
add get methods for packed objects
ee214e3
@meganrm
Merge branch 'feature/results-in-memory' of https://github.com/mesosc…
6430603 
…ope/cellpack into feature/results-in-memory
@meganrm @mogres
move when we store compartments in packed_objects
1605221 
Co-authored-by: Saurabh Mogre <[email protected]>
@meganrm
change radius
4a887de
@meganrm
use ingredient stored for compartment
25d9839
@mogres
fix linting
fe89fb7
@meganrm
pack sphere trees successfully and plot in 2D
c975872
@meganrm
Merge branch 'feature/results-in-memory' of https://github.com/mesosc…
5d9ecae 
…ope/cellpack into feature/results-in-memory
@rugeli
fix/S3 uploads and new browser tabs handling (#203)
9105668 
* modify if statement to check upload_results config

* format
@meganrm
format fixed
d35738d
Base automatically changed from feature/run-inherited-objects to feature/run-recipes-from-firebase October 12, 2023 21:24
mogres and others added 12 commits October 13, 2023 14:36
@mogres
Fix gradient inversion (#196)
5da57e0 
* update weights instead of distances while inverting gradient

* add credential file to gitignore

* add info in default values

* Log location of pack

* fix linting

* add invert options based on weight or distance

* * invert distances after checking for outliers
* check for outliers in weights

* updating tests
@meganrm
Merge pull request #193 from mesoscope/feature/results-in-memory
aae7ed1 
Feature/results in memory
@rugeli
Merge branch 'main' of https://github.com/mesoscope/cellpack into fea…
7e2c633 
…ture/run-recipes-from-firebase
@rugeli
Merge branch 'feature/run-recipes-from-firebase' of https://github.co…
0d577fe 
…m/mesoscope/cellpack into feature/run-recipes-from-firebase
@rugeli
formatting
44c3cb2
@rugeli
testing and refactor
7c0aae3
@rugeli
initial draft
e6777dd
@rugeli
reorder methods in firebase handler
c14dfbc
@rugeli
avoid initializing app twice
ad7c7c4
@rugeli
modularize firebase handler
7f7f0f7
@rugeli
modularize AWSHandler & refactors
1c8ac68
@rugeli
Merge branch 'feature/save-metadata' of https://github.com/mesoscope/…
82f7f7f 
…cellpack into feature/save-metadata
@rugeli
Copy link
Collaborator Author

rugeli commented Oct 23, 2023

changes in this PR are now in PR #206

@rugeli rugeli closed this Oct 23, 2023
@rugeli rugeli deleted the feature/save-metadata branch April 9, 2024 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@meganrm meganrm Awaiting requested review from meganrm

@mogres mogres Awaiting requested review from mogres

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@rugeli @mogres @meganrm