Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metadata schema #23

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open

Metadata schema #23

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
9611ff4
Very first integration of next.jdbc/honeysql, for a small function in…
ianmcorvidae Oct 20, 2021
6b017c1
Use next.jdbc features better
ianmcorvidae Oct 20, 2021
a90bf99
move select-favorites-of-type to next.jdbc
ianmcorvidae Oct 21, 2021
40e7f7a
migrate insert-favorite
ianmcorvidae Oct 21, 2021
5eb6de1
Migrate delete-favorite
ianmcorvidae Oct 21, 2021
c3b0f17
Migrate delete-favorites-of-type
ianmcorvidae Oct 21, 2021
7eaa8b8
Remove korma stuff from metadata.persistence.favorites namespace
ianmcorvidae Oct 21, 2021
ac877f5
Start migrating tags to use next.jdbc/honeysql
ianmcorvidae Oct 25, 2021
ccc6a23
migrate tags-base-query and things using it
ianmcorvidae Oct 26, 2021
fcba766
Migrate more tags stuff
ianmcorvidae Oct 29, 2021
760ceac
Migrate some insert/update stuff in tags
ianmcorvidae Nov 9, 2021
07a71e5
finish up tags
ianmcorvidae Nov 9, 2021
3608ba3
partially complete updates to avu persistence
ianmcorvidae Nov 17, 2021
37534cb
migrate delete-target-avu
ianmcorvidae Nov 17, 2021
28fd95e
hopefully-final AVUs work
ianmcorvidae Nov 17, 2021
e156bc8
Migrate comments
ianmcorvidae Nov 19, 2021
971ea25
Migrate ontologies-table stuff
ianmcorvidae Nov 23, 2021
bdeeea4
Migrate ontologies namespace
ianmcorvidae Nov 24, 2021
13ea1ab
rename imports as in other persistence namespaces
ianmcorvidae Nov 25, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions project.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@
[cheshire "5.10.0"]
[org.clojure/data.csv "1.0.0"]

[com.github.seancorfield/next.jdbc "1.2.737"]
[org.postgresql/postgresql "42.2.10"]
[com.github.seancorfield/honeysql "2.1.818"]

;; Langohr version 5.0.0 removes langohr.basic/blocking-subscribe.
[com.novemberain/langohr "4.2.0"]

Expand Down
178 changes: 101 additions & 77 deletions src/metadata/persistence/avu.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,52 +1,92 @@
(ns metadata.persistence.avu
(:use [korma.core :exclude [update]]
[korma.db :only [transaction]]
[slingshot.slingshot :only [throw+]])
(:require [kameleon.db :as db]
[korma.core :as sql]))

(defn- target-where-clause
"Adds a where-clause to the given query for the given target."
[query target-type target-id]
(where query {:target_id target-id
:target_type (db/->enum-val target-type)}))
(:use [slingshot.slingshot :only [throw+]])
(:require [metadata.util.db :refer [ds t]]
[next.jdbc :as jdbc]
[next.jdbc.plan :as plan]
[next.jdbc.sql :as jsql]
[next.jdbc.types :as jtypes]
[honey.sql :as sql]
[honey.sql.helpers :as h]
[kameleon.db :as db]))

(def avu-columns [:id :attribute :value :unit :target_id :target_type :created_by :modified_by :created_on :modified_on])

(defn- avus-base-query
[cols]
(-> (apply h/select cols)
(h/from (t "avus"))))
Comment on lines +12 to +17
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been doing a pattern like these two definitions in a bunch of places. Basically, have a plain def with the most common setup of columns, plus a whatever-base-query function that takes a list of columns and selects them from the appropriate table. I'm not sure if it's super useful, but it does cut down a little bit at least. May be worth generalizing, idk


(defn filter-targets-by-attrs-values
"Finds the given targets that have any of the given attributes and values."
[target-types target-ids attributes values]
(select :avus
(modifier "DISTINCT")
(fields :target_id
:target_type)
(where {:target_id [in target-ids]
:target_type [in (map db/->enum-val target-types)]
:attribute [in attributes]
:value [in values]})))
(let [cols [:target_id :target_type]
q (-> (avus-base-query cols)
(dissoc :select) ;; change to select distinct
(h/select-distinct cols)
(h/where [:in :target_id target-ids]
[:in :target_type (map jtypes/as-other target-types)]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First common change: jtypes/as-other is how we make things with our enums work properly; we don't need our ->enum-val any more.

[:in :attribute attributes]
[:in :value values]))]
(plan/select! ds cols (sql/format q))))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line means:

  • turn the query into SQL + parameters
  • fetch results from the DB defined by ds
  • on each row, run select-keys cols, and return a collection of the results. Or, if the cols slot is not a vector of keywords, it will be applied as a function (which also means, passing a single keyword will extract just that column)


(defn get-avu-by-id
"Fetches an AVU by its ID."
[id]
(first (select :avus (where {:id id}))))
(let [q (-> (avus-base-query avu-columns)
(h/where [:= :id id]))]
(plan/select-one! ds avu-columns (sql/format q))))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is exactly like select! but only does it for the first result row, and doesn't return a collection as a result.

Or basically it's the same as running first on the output of the same call with select!. except probably more efficient I guess


(defn get-avus-by-ids
"Finds existing AVUs by a set of IDs."
[ids]
(select :avus (where {:id [in ids]})))
(let [q (-> (avus-base-query avu-columns)
(h/where [:in :id ids]))]
(plan/select! ds avu-columns (sql/format q))))


(defn get-avus-for-target
"Gets AVUs for the given target."
[target-type target-id]
(select :avus (target-where-clause target-type target-id)))
(let [q (-> (avus-base-query avu-columns)
(h/where [:= :target_id target-id]
[:= :target_type (jtypes/as-other target-type)]))]
(plan/select! ds avu-columns (sql/format q))))

(defn get-avus-by-attrs
"Finds all existing AVUs by the given targets and the given set of attributes."
[target-types target-ids attributes]
(select :avus
(where {:attribute [in attributes]
:target_id [in target-ids]
:target_type [in (map db/->enum-val target-types)]})))
(let [q (-> (avus-base-query avu-columns)
(h/where [:in :attribute attributes]
[:in :target_id target-ids]
[:in :target_type (map jtypes/as-other target-types)]))]
(plan/select! ds avu-columns (sql/format q))))

(defn add-avus
"Adds the given AVUs to the Metadata database."
[user-id avus]
(let [cols [:id :attribute :value :unit :target_type :target_id :created_by :modified_by]
fmt-avu (fn [avu]
(let [avu (-> avu
(update :target_type jtypes/as-other)
(assoc :created_by user-id
:modified_by user-id))]
(mapv #(get avu %) cols)))]
(jsql/insert-multi! ds
(t "avus")
cols
(map fmt-avu avus))))
Comment on lines +74 to +77
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The format for these bulk inserts is different than korma. For this, we pass a collection of columns, plus a collection-of-collections, where each entry should be a collection corresponding to the columns listed before.

i.e. (:id :attribute ...) and then [("4609ef48-...." "some-attribute" ...), ...]


(defn get-avu-for-target
(defn update-avu
"Updates the attribute, value, unit, modified_by, and modified_on fields of the given AVU."
[user-id avu]
(let [q (-> (h/update (t "avus"))
(h/set (-> (select-keys avu [:attribute :value :unit])
(assoc :modified_by user-id
:modified_on :%now)))
(h/where [:= :id (:id avu)]))]
(jdbc/execute-one! ds (sql/format q))))

(defn- get-avu-for-target
"Finds an AVU by ID, validating its target_type and target_id match what's fetched from the database.
Returns nil if an AVU with the given ID does not already exist in the database."
[{:keys [id] :as avu}]
Expand All @@ -58,26 +98,6 @@
:avu existing-avu}))
existing-avu))

(defn add-avus
"Adds the given AVUs to the Metadata database."
[user-id avus]
(let [fmt-avu (fn [avu]
(-> avu
(select-keys [:id :attribute :value :unit :target_type :target_id])
(update :target_type db/->enum-val)
(assoc :created_by user-id
:modified_by user-id)))]
(insert :avus (values (map fmt-avu avus)))))

(defn update-avu
"Updates the attribute, value, unit, modified_by, and modified_on fields of the given AVU."
[user-id avu]
(sql/update :avus
(set-fields (-> (select-keys avu [:attribute :value :unit])
(assoc :modified_by user-id
:modified_on (sqlfn now))))
(where (select-keys avu [:id]))))

(defn- update-valid-avu
"Updates an AVU, validating its given target_type and target_id match what's already in the database.
Returns nil if an AVU with the given ID does not already exist in the database."
Expand All @@ -91,9 +111,11 @@
[avu]
(let [required-keys [:attribute :value :unit :target_type :target_id]]
(when (every? (partial contains? avu) required-keys)
(first (select :avus
(where (-> (select-keys avu (conj required-keys :id))
(update :target_type db/->enum-val))))))))
(let [avu (update avu :target_type jtypes/as-other)
keys-used (if (contains? avu :id) (conj required-keys :id) required-keys)
q (-> (avus-base-query avu-columns)
(apply h/where (map #(vector := % (get avu %)) keys-used)))]
(plan/select-one! ds avu-columns (sql/format q))))))

(defn add-or-update-avu
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whole function could probably be a proper upsert but I wanted to not get too mired down (not that it really worked since I still didn't finish, but whatever)

[user-id avu]
Expand All @@ -103,36 +125,37 @@
existing-avu
(add-avus user-id [avu]))))

(defn- add-orphaned-ids-where-clause
[query avu-ids-to-keep]
(if (empty? avu-ids-to-keep)
query
(where query {:id [not-in avu-ids-to-keep]})))

(defn remove-orphaned-avus
"Removes AVUs for the given target-id that are not in the given set of avu-ids-to-keep."
[target-type target-id avu-ids-to-keep]
(let [avus-to-remove (-> (select* :avus)
(target-where-clause target-type target-id)
(add-orphaned-ids-where-clause avu-ids-to-keep)
select)]
(let [add-keep-clause (fn [query avu-ids-to-keep]
(if (empty? avu-ids-to-keep)
query
(h/where query [:not-in :id avu-ids-to-keep])))
q (-> (avus-base-query [:id])
(h/where [:= :target_id target-id]
[:= :target_type (jtypes/as-other target-type)])
(add-keep-clause avu-ids-to-keep))
avus-to-remove (plan/select! ds [:id] (sql/format q))]
;; Remove orphaned sub-AVUs of any AVUs to be removed by this request.
(doseq [{:keys [id]} avus-to-remove]
(remove-orphaned-avus "avu" id nil))
(-> (delete* :avus)
(target-where-clause target-type target-id)
(add-orphaned-ids-where-clause avu-ids-to-keep)
delete)))
(jdbc/execute-one! ds
(-> q
(dissoc :select)
(h/delete-from (t "avus"))
sql/format))))

(defn delete-target-avu
"Deletes the AVU with the given attribute, value, and unit from the given target."
[target-types target-id attribute value unit]
(delete :avus
(where {:target_id target-id
:target_type [in (map db/->enum-val target-types)]
:attribute attribute
:value value
:unit unit})))
(let [q (-> (h/delete-from (t "avus"))
(h/where [:= :target_id target-id]
[:in :target_type (map jtypes/as-other target-types)]
[:= :attribute attribute]
[:= :value value]
[:= :unit unit]))]
(jdbc/execute-one! ds (sql/format q))))

(defn format-avu
"Formats a Metadata AVU for JSON responses."
Expand All @@ -149,13 +172,14 @@
(defn- find-avus
"Searches for AVUs matching the given criteria."
[attributes target-types values units]
(let [add-criterion (fn [query f vs] (if (seq vs) (where query {f [in vs]}) query))]
(-> (select* :avus)
(add-criterion :attribute attributes)
(add-criterion :target_type (map db/->enum-val target-types))
(add-criterion :value values)
(add-criterion :unit units)
(select))))
(let [add-criterion (fn [query f vs] (if (seq vs) (h/where query [:in f vs]) query))]
(plan/select! ds avu-columns
(-> (avus-base-query avu-columns)
(add-criterion :attribute attributes)
(add-criterion :target_type (map jtypes/as-other target-types))
(add-criterion :value values)
(add-criterion :unit units)
(sql/format)))))

(defn avu-list
"Lists AVUs for the given target."
Expand Down
Loading