From 4b6aa44aaf6be4b6346fc2fa0d1f2fa6a00a726a Mon Sep 17 00:00:00 2001 From: Betsy McPhail Date: Wed, 19 Sep 2018 16:57:05 -0400 Subject: [PATCH] Update Patch install documentation Change-Id: Id0682e6df28563a6c0718e1f08a1e62141050ad0 --- Documentation/InstallPatches.rst | 105 ++++++++++++++++++---------- Scripts/HowtoUpdateSpreadsheets.rst | 62 +++------------- Scripts/PatchSequenceApply.rst | 39 +++++++---- 3 files changed, 103 insertions(+), 103 deletions(-) diff --git a/Documentation/InstallPatches.rst b/Documentation/InstallPatches.rst index 1c43a4661..529a818c0 100644 --- a/Documentation/InstallPatches.rst +++ b/Documentation/InstallPatches.rst @@ -4,6 +4,24 @@ Install FOIA Patches **--- These instructions are for Windows Caché environments only ---** +Update VistA +------------ + +Clone the latest version of master from the VistA_ and VistA-M_ repositories. + +Create a local VistA branch, the name is not important. + +Setup and Configure +~~~~~~~~~~~~~~~~~~~ + +`Set up the testing environment`_ using the TEST_VISTA_FRESH and +TEST_VISTA_SETUP options. + +Run the `automated VistA import script`_ and, if necessary, configure_ VistA. + +Save a copy of the configured Cache.dat so that a clean instance is available +without re-running the import script. + Download Patches ---------------- @@ -11,14 +29,36 @@ Download Patches processing, it is recommended to make a copy of the downloaded patches, so they don't need to be downloaded again if something goes wrong. -Cleanup -~~~~~~~ +Cleanup Line Endings +~~~~~~~~~~~~~~~~~~~~ Convert text files to LF-only newlines:: find FOIA -regex '.*\.\(GBL\|GBLs\|KID\|KIDs\|KIDS\|kid\|kids\|TXT\|TXTs\|txt\|txts\)$' -print0 | xargs -0 fromdos +If you don't do this, when running the next script, you may see errors like:: + + 2017-12-20 07:28:25,391 WARNING Can not parse **INSTALL NAME** + Traceback (most recent call last): + File "Scripts/ConvertToExternalData.py", line 310, in + main() + File "Scripts/ConvertToExternalData.py", line 296, in main + converter.convertCurrentDir(result.inputDir) + File "Scripts/ConvertToExternalData.py", line 230, in convertCurrentDir + self.convertKIDSBuildFile(absFileName) + File "Scripts/ConvertToExternalData.py", line 241, in convertKIDSBuildFile + kidsParser.parseKIDSBuild(kidsFile) + File "C:\Users\betsy.mcphail\development\VistA\Scripts\KIDSBuildParser.py", line 538, in parseKIDSBuild + kidsBuild=self._curKidsBuild) + File "C:\Users\betsy.mcphail\development\VistA\Scripts\KIDSBuildParser.py", line 236, in parseLines + handler(lines, kidsBuild) + File "C:\Users\betsy.mcphail\development\VistA\Scripts\KIDSBuildParser.py", line 266, in __handlePostInstallRoutine__ + if kidsBuild.postInstallRoutine: + AttributeError: 'NoneType' object has no attribute 'postInstallRoutine' + +Convert Large Files +~~~~~~~~~~~~~~~~~~~ Convert large files to content links referencing external data:: @@ -47,31 +87,7 @@ CovertToExternalData script may fail with errors like:: if kidsBuild.postInstallRoutine: AttributeError: 'NoneType' object has no attribute 'postInstallRoutine' -Update VistA ------------- - -Clone the latest version of master from the VistA_ and VistA-M_ repositories. - -Create a local VistA branch, the name is not important. - -Setup and Configure -~~~~~~~~~~~~~~~~~~~ - -`Set up the testing environment`_ using the TEST_VISTA_FRESH and -TEST_VISTA_SETUP options. - -Run the `automated VistA import script`_ and configure_ VistA. - -Save a copy of the configured Cache.dat so that a clean instance is available -without re-running the import script. - - -Populate Packages -~~~~~~~~~~~~~~~~~ - -Sort_ the new patches into the ``Packages/`` directory structure. - -Rename any ".ExternalData_SHA1_" files to just the SHA, for example:: +Rename ".ExternalData_SHA1_" files to just the SHA, for example:: mv .ExternalData_SHA1_4d8ef9763be16099e4a10ea67ba8812245c227d3 4d8ef9763be16099e4a10ea67ba8812245c227d3 @@ -82,6 +98,11 @@ Alternatively, the script will also look in Do *not* commit the external data files to the repository. +Populate Packages +~~~~~~~~~~~~~~~~~ + +Sort_ the new patches into the ``Packages/`` directory structure. + Add new files (excludes external data files) and commit changes so far to a local branch. @@ -89,19 +110,18 @@ Update Spreadsheet ~~~~~~~~~~~~~~~~~~ Follow the instructions in `Update Spreadsheet`_ to update the current year's -patch spreadsheet to the latest available version. Submit the updated -spreadsheet to Gerrit. +patch spreadsheet to the latest available version. **Note:**: In January, the previous year's spreadsheet may also need to be updated. -**Note:** The testing portion of `Update Spreadsheet`_ is also described below. +Move .csv file to the ``Packages/Uncategorized/`` directory and commit to local branch. Patch Sequence ~~~~~~~~~~~~~~ Run the `patch installer script`_ from the VistA Scripts directory to verify -that all require patches are available. Choose the option to generate patch +that all required patches are available. Choose the option to generate patch sequence order with actually installing any of the patches:: cd Scripts @@ -114,6 +134,8 @@ installer script completes with no errors. For help with other potential errors, refer to `Potential Issues`_. +Commit any changes to the working VistA branch. + Apply Patches ~~~~~~~~~~~~~ @@ -125,19 +147,22 @@ patches:: Fix any errors and commit changes so far to the working VistA branch. -Commit Changes -~~~~~~~~~~~~~~ +Create PR For VistA Changes +~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create a Gerritt PR for the VistA changes as desribed in the `Contributor Git Instructions`_. +If the spreadsheet needed to be modified, note the lines that were changed or +removed in the commit message. + Update VistA-M -------------- `Set up the testing environment`_ but this time, do *not* use the TEST_VISTA_SETUP option. -Run the `automated VistA import script`_ and configure_ VistA. +Run the `automated VistA import script`_. Save a copy of the configured Cache.dat so that a clean instance is available without re-running the import script. @@ -188,7 +213,7 @@ the VistA-M repo, this expected, the changes can be discarded. Uncategorized files ~~~~~~~~~~~~~~~~~~~ -Update Packages.csv to take into account the files that are in +Update Packages.csv to take into account the files that are in VistA-M ``Packages/Uncategorized/``. Move the contents of the ``Packages/Uncategorized/Routines/`` and @@ -204,19 +229,22 @@ Make a commit with the updated Packages.csv and any moved files. Update Packages.csv in the *VistA* directory to include any changes just made. Revise the topic and push it back to Gerrit. -Commit Changes -~~~~~~~~~~~~~~ +Create VistA-M Branch +~~~~~~~~~~~~~~~~~~~~~ Create a branch in the `OSEHRA VistA-M Sandbox`_ (e.g. OV_Nov_2017) and push all of the VistA-M commits there. +**Note:** There will be several uncommitted files with just a date change in +the VistA-M repository. This is expected, these changes can be discarded. + Run Tests ~~~~~~~~~ `Set up the testing environment`_ using the TEST_VISTA_FRESH and TEST_VISTA_SETUP options. -Run the `automated VistA import script`_ and configure_ VistA. +Run the `automated VistA import script`_. Save a copy of the configured Cache.dat so that a clean instance is available without re-running the import script. @@ -267,6 +295,7 @@ If any of the GUIs need to be updated, new versions can be downloaded from: https://foia-vista.osehra.org/Patches_By_Application + .. _VistA: https://github.com/OSEHRA/VistA .. _VistA-M: https://github.com/OSEHRA/VistA-M .. _`Set up the testing environment`: SetupTestingEnvironment.rst diff --git a/Scripts/HowtoUpdateSpreadsheets.rst b/Scripts/HowtoUpdateSpreadsheets.rst index 2cade2154..2cc9e554d 100644 --- a/Scripts/HowtoUpdateSpreadsheets.rst +++ b/Scripts/HowtoUpdateSpreadsheets.rst @@ -2,27 +2,13 @@ Howto Update Spreadsheets ========================= This document describes how to update the ``Packages/Uncategorized/*.csv`` FOIA -patch spreadsheets using the latest ``.xls`` files from the OSEHRA FOIA VistA +patch spreadsheets using the latest ``.xls`` file from the OSEHRA FOIA VistA site. -Branch ------- - -Create a branch, say ``upstream-spreadsheets``, starting from the last commit -that updated the ``.csv`` files from the FOIA site:: - - base=$(git rev-list master -n 1 --author='va.gov' -- 'Packages/Uncategorized/*.csv') && - git checkout -b upstream-spreadsheets $base - -It is important to base the update branch on a version that has no other -modifications since the last update from the FOIA site. This way the branch -represents only the upstream changes and can be merged with changes made in -other branches instead of overwriting them. - Download -------- -Download the ``.xls`` files from the OSEHRA FOIA VistA web site for some ``${year}``:: +Download the ``.xls`` file from the OSEHRA FOIA VistA web site for the current ``${year}``:: wget --continue --input-file=- --user-agent='' \ https://foia-vista.osehra.org/DBA_VistA_FOIA_System_Files/All_Listing_of_Released_VistA_Patches/${year}_Listing_of_Released_VistA_Patches.xls @@ -30,7 +16,10 @@ Download the ``.xls`` files from the OSEHRA FOIA VistA web site for some ``${yea Convert ------- -Obtain the ``xlsx2csv`` tool and hack it to recognize the VA dates:: +First, use MS Excel to convert the ``.xls`` file to ``.xlsx`` by opening the +file and saving as 'Excel Workbook'. + +Next, obtain the ``xlsx2csv`` tool and hack it to recognize the VA dates:: git clone https://github.com/dilshod/xlsx2csv.git && (cd xlsx2csv && git reset --hard fa44ba6f && @@ -38,41 +27,8 @@ Obtain the ``xlsx2csv`` tool and hack it to recognize the VA dates:: '"'"'[$-409]d-mmm-yyyy;@'"'"' : '"'"'date'"'"', d}' xlsx2csv.py) -Run ``xlsx2csv`` to convert the ``.xls`` files to ``.csv`` format:: - - mkdir -p Packages/Uncategorized && - for xls in *.xls; do - csv="Packages/Uncategorized/${xls/xls/csv}" && - python xlsx2csv/xlsx2csv.py --ignoreempty --dateformat='%Y-%m-%d' "${xls}" "${csv}" && - fromdos "${csv}" - done && - rm *.xls && - rm -rf xlsx2csv - -Remove extra rows from end of files by hand if desired. - -If xlsx2csv complains about the format, use MS Excel to convert the -.xls files to .xlsx and adjust the above command accordingly. - -Commit ------- - -Add the changes and commit them with the VA as author:: - - git add Packages/Uncategorized/*.csv - git commit --author='US DVA ' - -Mention in the commit message this instructions file and the URL of the -``.xls`` file downloaded. - -Merge ------ - -Create a branch, say ``update-spreadsheets``, and merge the upstream changes:: - - git checkout -b update-spreadsheets master - git merge upstream-spreadsheets +Run ``xlsx2csv`` to convert the ``.xlsx`` file to ``.csv`` format:: -If there are conflicts resolve them and commit with a message describing how. + python xlsx2csv/xlsx2csv.py --ignoreempty --dateformat='%Y-%m-%d' ${year}_Listing_of_Released_VistA_Patches.xlsx ${year}_Listing_of_Released_VistA_Patches.csv -Finally, push this topic for review like any other change. +Remove extra rows from end of the file by hand if desired. diff --git a/Scripts/PatchSequenceApply.rst b/Scripts/PatchSequenceApply.rst index b98f55bbb..e5a4003b3 100644 --- a/Scripts/PatchSequenceApply.rst +++ b/Scripts/PatchSequenceApply.rst @@ -298,14 +298,30 @@ Example +++++++++ :: - File "c:\Users\joe.snyder\Work\OSEHRA\VistA\Scripts\PatchOrderGenerator.py", line 737, in visitNode - raise Exception("DAG is NOT acyclic") - Exception: DAG is NOT acyclic - 2016-01-19 11:32:51,857 ERROR This is a cycle among these items: - 'DG*5.3*904' - 'DG*5.3*901' - 'IVM*2.0*160' - 'EAS*1.0*113' + 2018-09-19 16:02:22,858 ERROR This is a cycle among these items: + 'ECX*3.0*153' + 'DG*5.3*895' + 'PSN*4.0*434' + 'PSN*4.0*433' + 'PSN*4.0*432' + 'PSN*4.0*365' + 'PXRM*2.0*57' + 'IB*2.0*538' + 'SD*5.3*620' + 'DVBA*2.7*191' + 'PSJ*5.0*304' + 'GMPL*2.0*46' + 'DENT*1.2*67' + 'XU*8.0*638' + 'LR*5.2*438' + 'VPR*1.0*4' + 'SD*5.3*621' + 'IBD*3.0*66' + 'OR*3.0*398' + 'XT*7.3*137' + 2018-09-19 16:02:22,858 ERROR Failed to sort patches: DAG is NOT acyclic + 2018-09-19 16:02:22,859 INFO Total patches are 0 + Resolution ++++++++++ @@ -313,10 +329,9 @@ Resolution The above problem is a convergence of the extra dependencies causing a cycle of dependencies. -Unfortunately, the solutions so far have been to remove available information -from the parser's reach. Tne solution for the above problem was to remove the -text files, thus removing number information about the builds and eliminating -one avenue of dependency information. +Unfortunately, the solution so far have been to remove available information +from the parser's reach. Tne solution for the above problem was to remove +patches from the most recent spreadsheet. .. _some: https://github.com/OSEHRA/VistA/blob/master/Packages/Kernel/Patches/XU_8.0_599/XU_8.0_599.py