Vebt 352 - Automate Vsoc Spreadsheet on GIDS Dashboard (#1209)

* Automate GIDS Dashboard VSOC spreadsheet * Add API_SOURCE to Vsoc * modify curl parameters for vsoc * Add comments to describe Vsoc specific changes * Add space to README to trigger jenkins --------- Co-authored-by: nfstern02 <[email protected]>
department-of-veterans-affairs · Oct 1, 2024 · 6821204 · 6821204
1 parent 0c09ece
commit 6821204
Show file tree

Hide file tree

Showing 5 changed files with 13 additions and 5 deletions.
diff --git a/README.md b/README.md
@@ -9,7 +9,7 @@ Secondarily, institution information may be exported as a CSV for regulatory rep
 GIDS data is accessible via an API intended for use by the GI Bill Comparison Tool client (**GIBCT**), which is part of
 the `vets-api` and `vets-website` applications.
 
-### Data Modes and Versions
+### Data Modes and Versions 
 GIDS profile data is logically partitioned in two modes: **preview** mode and **production** mode. In preview mode the
 data retrieved via the API has not yet been completely processed and built out by the InstitutionBuilder model. In 
 contrast, production mode is the actual data pushed to **GIBCT** for public consumption.

diff --git a/app/controllers/dashboards_controller.rb b/app/controllers/dashboards_controller.rb
@@ -210,7 +210,7 @@ def download_csv(class_nm)
 
   def unzip_csv(class_nm)
     # Some downloads do are not a zip file, so skip and return true
-    return true if class_nm.eql?('Hcm') || class_nm.eql?('EightKey') || class_nm.eql?('Mou')
+    return true if class_nm.eql?('Hcm') || class_nm.eql?('EightKey') || class_nm.eql?('Mou') || class_nm.eql?('Vsoc')
 
     ZipFileUtils::Unzipper.new.unzip_the_file
   end

diff --git a/app/models/vsoc.rb b/app/models/vsoc.rb
@@ -10,5 +10,7 @@ class Vsoc < ImportableRecord
     'vetsuccess_email' => { column: :vetsuccess_email, converter: Converters::BaseConverter }
   }.freeze
 
+  API_SOURCE = 'https://vbaw.vba.va.gov/EDUCATION/job_aids/documents/'
+
   validates :facility_code, presence: true
 end
diff --git a/app/utilities/no_key_apis/no_key_api_downloader.rb b/app/utilities/no_key_apis/no_key_api_downloader.rb
@@ -12,9 +12,11 @@ class NoKeyApiDownloader
       'IpedsIcAy' => 'tmp/ic2022_ay.csv',
       'IpedsIcPy' => 'tmp/ic2022_py.csv',
       'IpedsIc' => 'tmp/ic2022.csv',
-      'Mou' => 'tmp/mou.xlsx'
+      'Mou' => 'tmp/mou.xlsx',
+      'Vsoc' => 'tmp/vsoc.csv'
     }.freeze
 
+    # Vsoc uses -k parameter to bypass SSL certificate errors
     API_NO_KEY_DOWNLOAD_SOURCES = {
       'Accreditation' => [' -X POST', 'https://ope.ed.gov/dapip/api/downloadFiles/accreditationDataFiles'],
       'AccreditationAction' => [' -X POST', 'https://ope.ed.gov/dapip/api/downloadFiles/accreditationDataFiles'],
@@ -26,7 +28,8 @@ class NoKeyApiDownloader
       'IpedsIc' => [' -X GET', 'https://nces.ed.gov/ipeds/datacenter/data/IC2022.zip'],
       'IpedsIcAy' => [' -X GET', 'https://nces.ed.gov/ipeds/datacenter/data/IC2022_AY.zip'],
       'IpedsIcPy' => [' -X GET', 'https://nces.ed.gov/ipeds/datacenter/data/IC2022_PY.zip'],
-      'Mou' => [' -X GET', "'https://www.dodmou.com/Home/DownloadS3File?s3bucket=dodmou-private-ah9xbf&s3Key=participatinginstitutionslist%2Fproduction%2FInstitutionsList.xlsx'"]
+      'Mou' => [' -X GET', "'https://www.dodmou.com/Home/DownloadS3File?s3bucket=dodmou-private-ah9xbf&s3Key=participatinginstitutionslist%2Fproduction%2FInstitutionsList.xlsx'"],
+      'Vsoc' => [' -k -X GET', "'https://vbaw.vba.va.gov/EDUCATION/job_aids/documents/Vsoc_08132024.csv'"]
     }.freeze
 
     attr_accessor :class_nm, :curl_command
@@ -46,7 +49,9 @@ def download_csv
     private
 
     def h_parm
+      # Vsoc uses the octet-stream header to pull down from source
       return '-H "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0"' if @class_nm.eql?('Hcm')
+      return '-H \'Content-Type: application/octet-stream\'' if @class_nm.eql?('Vsoc')
 
       '-H \'Content-Type: application/json\''
     end
@@ -56,6 +61,7 @@ def o_parm
       when 'Hcm' then '-o tmp/hcm.xlsx'
       when 'EightKey' then '-o tmp/eight_key.xls'
       when 'Mou' then '-o tmp/mou.xlsx'
+      when 'Vsoc' then '-o tmp/vsoc.csv'
       else '-o tmp/download.zip'
       end
     end

diff --git a/config/initializers/csv_types.rb b/config/initializers/csv_types.rb
@@ -19,7 +19,7 @@
     { klass: ScorecardDegreeProgram, required?: false, has_api?: true },
     { klass: Sec702, required?: true },
     { klass: Sva, required?: true },
-    { klass: Vsoc, required?: true },
+    { klass: Vsoc, required?: true, has_api?: true, no_api_key?: true },
     { klass: Weam, required?: true },
     { klass: CalculatorConstant, required?: false },
     { klass: IpedsCipCode, required?: true },