Update Patch install documentation

Change-Id: Id0682e6df28563a6c0718e1f08a1e62141050ad0

Documentation/InstallPatches.rst

@@ -4,21 +4,61 @@ Install FOIA Patches
 **--- These instructions are for Windows Caché environments only ---**
 
 
+Update VistA
+------------
+
+Clone the latest version of master from the VistA_ and VistA-M_ repositories.
+
+Create a local VistA branch, the name is not important.
+
+Setup and Configure
+~~~~~~~~~~~~~~~~~~~
+
+`Set up the testing environment`_ using the TEST_VISTA_FRESH and
+TEST_VISTA_SETUP options.
+
+Run the `automated VistA import script`_ and, if necessary, configure_ VistA.
+
+Save a copy of the configured Cache.dat so that a clean instance is available
+without re-running the import script.
+
 Download Patches
 ----------------
 
 `Download FOIA patches`_ from the OSEHRA FOIA VistA site. Before doing any
 processing, it is recommended to make a copy of the downloaded patches, so
 they don't need to be downloaded again if something goes wrong.
 
-Cleanup
-~~~~~~~
+Cleanup Line Endings
+~~~~~~~~~~~~~~~~~~~~
 
 Convert text files to LF-only newlines::
 
  find FOIA -regex '.*\.\(GBL\|GBLs\|KID\|KIDs\|KIDS\|kid\|kids\|TXT\|TXTs\|txt\|txts\)$' -print0 |
  xargs -0 fromdos
 
+If you don't do this, when running the next script, you may see errors like::
+
+  2017-12-20 07:28:25,391 WARNING Can not parse **INSTALL NAME**
+  Traceback (most recent call last):
+    File "Scripts/ConvertToExternalData.py", line 310, in <module>
+      main()
+    File "Scripts/ConvertToExternalData.py", line 296, in main
+      converter.convertCurrentDir(result.inputDir)
+    File "Scripts/ConvertToExternalData.py", line 230, in convertCurrentDir
+      self.convertKIDSBuildFile(absFileName)
+    File "Scripts/ConvertToExternalData.py", line 241, in convertKIDSBuildFile
+      kidsParser.parseKIDSBuild(kidsFile)
+    File "C:\Users\betsy.mcphail\development\VistA\Scripts\KIDSBuildParser.py", line 538, in parseKIDSBuild
+      kidsBuild=self._curKidsBuild)
+    File "C:\Users\betsy.mcphail\development\VistA\Scripts\KIDSBuildParser.py", line 236, in parseLines
+      handler(lines, kidsBuild)
+    File "C:\Users\betsy.mcphail\development\VistA\Scripts\KIDSBuildParser.py", line 266, in __handlePostInstallRoutine__
+      if kidsBuild.postInstallRoutine:
+  AttributeError: 'NoneType' object has no attribute 'postInstallRoutine'
+
+Convert Large Files
+~~~~~~~~~~~~~~~~~~~
 
 Convert large files to content links referencing external data::
 
@@ -47,31 +87,7 @@ CovertToExternalData script may fail with errors like::
         if kidsBuild.postInstallRoutine:
     AttributeError: 'NoneType' object has no attribute 'postInstallRoutine'
 
-Update VistA
-------------
-
-Clone the latest version of master from the VistA_ and VistA-M_ repositories.
-
-Create a local VistA branch, the name is not important.
-
-Setup and Configure
-~~~~~~~~~~~~~~~~~~~
-
-`Set up the testing environment`_ using the TEST_VISTA_FRESH and
-TEST_VISTA_SETUP options.
-
-Run the `automated VistA import script`_ and configure_ VistA.
-
-Save a copy of the configured Cache.dat so that a clean instance is available
-without re-running the import script.
-
-
-Populate Packages
-~~~~~~~~~~~~~~~~~
-
-Sort_ the new patches into the ``Packages/`` directory structure.
-
-Rename any ".ExternalData_SHA1_<SHA>" files to just the SHA, for example::
+Rename ".ExternalData_SHA1_<SHA>" files to just the SHA, for example::
 
   mv .ExternalData_SHA1_4d8ef9763be16099e4a10ea67ba8812245c227d3 4d8ef9763be16099e4a10ea67ba8812245c227d3
 
@@ -82,26 +98,30 @@ Alternatively, the script will also look in
 
 Do *not* commit the external data files to the repository.
 
+Populate Packages
+~~~~~~~~~~~~~~~~~
+
+Sort_ the new patches into the ``Packages/`` directory structure.
+
 Add new files (excludes external data files) and commit changes so far to a
 local branch.
 
 Update Spreadsheet
 ~~~~~~~~~~~~~~~~~~
 
 Follow the instructions in `Update Spreadsheet`_ to update the current year's
-patch spreadsheet to the latest available version. Submit the updated
-spreadsheet to Gerrit.
+patch spreadsheet to the latest available version.
 
 **Note:**: In January, the previous year's spreadsheet may also need to be
 updated.
 
-**Note:** The testing portion of `Update Spreadsheet`_ is also described below.
+Move .csv file to the ``Packages/Uncategorized/`` directory and commit to local branch.
 
 Patch Sequence
 ~~~~~~~~~~~~~~
 
 Run the `patch installer script`_ from the VistA Scripts directory to verify
-that all require patches are available. Choose the option to generate patch
+that all required patches are available. Choose the option to generate patch
 sequence order with actually installing any of the patches::
 
   cd Scripts
@@ -114,6 +134,8 @@ installer script completes with no errors.
 
 For help with other potential errors, refer to `Potential Issues`_.
 
+Commit any changes to the working VistA branch.
+
 Apply Patches
 ~~~~~~~~~~~~~
 
@@ -125,19 +147,22 @@ patches::
 
 Fix any errors and commit changes so far to the working VistA branch.
 
-Commit Changes
-~~~~~~~~~~~~~~
+Create PR For VistA Changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Create a Gerritt PR for the VistA changes as desribed in the
 `Contributor Git Instructions`_.
 
+If the spreadsheet needed to be modified, note the lines that were changed or
+removed in the commit message.
+
 Update VistA-M
 --------------
 
 `Set up the testing environment`_ but this time, do *not* use the
 TEST_VISTA_SETUP option.
 
-Run the `automated VistA import script`_ and configure_ VistA.
+Run the `automated VistA import script`_.
 
 Save a copy of the configured Cache.dat so that a clean instance is available
 without re-running the import script.
@@ -188,7 +213,7 @@ the VistA-M repo, this expected, the changes can be discarded.
 Uncategorized files
 ~~~~~~~~~~~~~~~~~~~
 
-Update Packages.csv to take into account the files that are in
+Update Packages.csv to take into account the files that are in VistA-M
 ``Packages/Uncategorized/``.
 
 Move the contents of the ``Packages/Uncategorized/Routines/`` and
@@ -204,19 +229,22 @@ Make a commit with the updated Packages.csv and any moved files.
 Update Packages.csv in the *VistA* directory to include any changes just made.
 Revise the topic and push it back to Gerrit.
 
-Commit Changes
-~~~~~~~~~~~~~~
+Create VistA-M Branch
+~~~~~~~~~~~~~~~~~~~~~
 
 Create a branch in the `OSEHRA VistA-M Sandbox`_ (e.g. OV_Nov_2017) and push
 all of the VistA-M commits there.
 
+**Note:** There will be several uncommitted files with just a date change in
+the VistA-M repository. This is expected, these changes can be discarded.
+
 Run Tests
 ~~~~~~~~~
 
 `Set up the testing environment`_ using the TEST_VISTA_FRESH and
 TEST_VISTA_SETUP options.
 
-Run the `automated VistA import script`_ and configure_ VistA.
+Run the `automated VistA import script`_.
 
 Save a copy of the configured Cache.dat so that a clean instance is available
 without re-running the import script.
@@ -267,6 +295,7 @@ If any of the GUIs need to be updated, new versions can be downloaded from:
 https://foia-vista.osehra.org/Patches_By_Application
 
 
+
 .. _VistA: https://github.com/OSEHRA/VistA
 .. _VistA-M: https://github.com/OSEHRA/VistA-M
 .. _`Set up the testing environment`: SetupTestingEnvironment.rst

Scripts/HowtoUpdateSpreadsheets.rst

@@ -2,77 +2,33 @@ Howto Update Spreadsheets
 =========================
 
 This document describes how to update the ``Packages/Uncategorized/*.csv`` FOIA
-patch spreadsheets using the latest ``.xls`` files from the OSEHRA FOIA VistA
+patch spreadsheets using the latest ``.xls`` file from the OSEHRA FOIA VistA
 site.
 
-Branch
-------
-
-Create a branch, say ``upstream-spreadsheets``, starting from the last commit
-that updated the ``.csv`` files from the FOIA site::
-
- base=$(git rev-list master -n 1 --author='va.gov' -- 'Packages/Uncategorized/*.csv') &&
- git checkout -b upstream-spreadsheets $base
-
-It is important to base the update branch on a version that has no other
-modifications since the last update from the FOIA site.  This way the branch
-represents only the upstream changes and can be merged with changes made in
-other branches instead of overwriting them.
-
 Download
 --------
 
-Download the ``.xls`` files from the OSEHRA FOIA VistA web site for some ``${year}``::
+Download the ``.xls`` file from the OSEHRA FOIA VistA web site for the current ``${year}``::
 
  wget --continue --input-file=- --user-agent='' \
   https://foia-vista.osehra.org/DBA_VistA_FOIA_System_Files/All_Listing_of_Released_VistA_Patches/${year}_Listing_of_Released_VistA_Patches.xls
 
 Convert
 -------
 
-Obtain the ``xlsx2csv`` tool and hack it to recognize the VA dates::
+First, use MS Excel to convert the ``.xls`` file to ``.xlsx`` by opening the
+file and saving as 'Excel Workbook'.
+
+Next, obtain the ``xlsx2csv`` tool and hack it to recognize the VA dates::
 
  git clone https://github.com/dilshod/xlsx2csv.git &&
  (cd xlsx2csv && git reset --hard fa44ba6f &&
   sed -i '/  '"'"'d-mmm-yyyy'"'"'/ {p;i\
   '"'"'[$-409]d-mmm-yyyy;@'"'"' : '"'"'date'"'"',
   d}' xlsx2csv.py)
 
-Run ``xlsx2csv`` to convert the ``.xls`` files to ``.csv`` format::
-
- mkdir -p Packages/Uncategorized &&
- for xls in *.xls; do
-   csv="Packages/Uncategorized/${xls/xls/csv}" &&
-   python xlsx2csv/xlsx2csv.py --ignoreempty --dateformat='%Y-%m-%d' "${xls}" "${csv}" &&
-   fromdos "${csv}"
- done &&
- rm *.xls &&
- rm -rf xlsx2csv
-
-Remove extra rows from end of files by hand if desired.
-
-If xlsx2csv complains about the format, use MS Excel to convert the
-.xls files to .xlsx and adjust the above command accordingly.
-
-Commit
-------
-
-Add the changes and commit them with the VA as author::
-
- git add Packages/Uncategorized/*.csv
- git commit --author='US DVA <va.gov>'
-
-Mention in the commit message this instructions file and the URL of the
-``.xls`` file downloaded.
-
-Merge
------
-
-Create a branch, say ``update-spreadsheets``, and merge the upstream changes::
-
- git checkout -b update-spreadsheets master
- git merge upstream-spreadsheets
+Run ``xlsx2csv`` to convert the ``.xlsx`` file to ``.csv`` format::
 
-If there are conflicts resolve them and commit with a message describing how.
+ python xlsx2csv/xlsx2csv.py --ignoreempty --dateformat='%Y-%m-%d' ${year}_Listing_of_Released_VistA_Patches.xlsx ${year}_Listing_of_Released_VistA_Patches.csv
 
-Finally, push this topic for review like any other change.
+Remove extra rows from end of the file by hand if desired.

Scripts/PatchSequenceApply.rst

@@ -298,25 +298,40 @@ Example
 +++++++++
 ::
 
-  File "c:\Users\joe.snyder\Work\OSEHRA\VistA\Scripts\PatchOrderGenerator.py", line 737, in visitNode
-    raise Exception("DAG is NOT acyclic")
-  Exception: DAG is NOT acyclic
-  2016-01-19 11:32:51,857 ERROR This is a cycle among these items:
-  'DG*5.3*904'
-  'DG*5.3*901'
-  'IVM*2.0*160'
-  'EAS*1.0*113'
+  2018-09-19 16:02:22,858 ERROR This is a cycle among these items:
+  'ECX*3.0*153'
+  'DG*5.3*895'
+  'PSN*4.0*434'
+  'PSN*4.0*433'
+  'PSN*4.0*432'
+  'PSN*4.0*365'
+  'PXRM*2.0*57'
+  'IB*2.0*538'
+  'SD*5.3*620'
+  'DVBA*2.7*191'
+  'PSJ*5.0*304'
+  'GMPL*2.0*46'
+  'DENT*1.2*67'
+  'XU*8.0*638'
+  'LR*5.2*438'
+  'VPR*1.0*4'
+  'SD*5.3*621'
+  'IBD*3.0*66'
+  'OR*3.0*398'
+  'XT*7.3*137'
+  2018-09-19 16:02:22,858 ERROR Failed to sort patches: DAG is NOT acyclic
+  2018-09-19 16:02:22,859 INFO Total patches are 0
+
 
 Resolution
 ++++++++++
 
 The above problem is a convergence of the extra dependencies causing a cycle of
 dependencies.
 
-Unfortunately, the solutions so far have been to remove available information
-from the parser's reach.  Tne solution for the above problem was  to remove the
-text files, thus removing number information about the builds and eliminating
-one avenue of dependency information.
+Unfortunately, the solution so far have been to remove available information
+from the parser's reach.  Tne solution for the above problem was to remove
+patches from the most recent spreadsheet.
 
 
 .. _some: https://github.com/OSEHRA/VistA/blob/master/Packages/Kernel/Patches/XU_8.0_599/XU_8.0_599.py