WIP: ADAT - Improvements

[bijayraj]

Issues and bug fixes related to #388

Bug Fixes:

• Alternate scoring - make optional with hierarchy
• Alternate scoring - graphs missing
• Hierarchy info missing from Images folder

New Features:

• Load command line arguments from json/ini file with --file option
• Compare two files - new script to generate diff options
• GUI for saving/editing configs, invoking visualizations

Notes

• New python module "csvdiff" is required. Can be installed using pip install csvdiff.

• GUI uses pyqt4.

• The diff is visible in "Image Result" table in generated HTML.

• The code is more modular with main functions moved to "adat_classification.py" file.

• "generate-visualization.py" is responsible for parsing command-line arguments and invoking html generate function from "adat_classification.py"

• "adat_classification.py" can now be imported to external python scripts.

[lgtm-com]

This pull request introduces 3 alerts and fixes 1 when merging aa22136 into e302042 - view on LGTM.com

new alerts:

• 1 for Unused local variable
• 1 for Unused import
• 1 for Variable defined multiple times

fixed alerts:

• 1 for Variable defined multiple times

[lgtm-com]

This pull request fixes 3 alerts when merging 29b6279 into e302042 - view on LGTM.com

fixed alerts:

• 1 for Testing equality to None
• 1 for Implicit string concatenation in a list
• 1 for Variable defined multiple times

[kiritigowda]

@bijayraj missing 'import argparse`

 python toolkit/amd_data_analysis_toolkit/classification/generate-visualization.py 
2020-11-09 17:53:36,540 [MainThread  ] [DEBUG]  AMD Data Analysis Toolkit - Classification
Traceback (most recent call last):
  File "toolkit/amd_data_analysis_toolkit/classification/generate-visualization.py", line 70, in <module>
    main()
  File "toolkit/amd_data_analysis_toolkit/classification/generate-visualization.py", line 37, in main
    parser = argparse.ArgumentParser()

[kiritigowda]

@bijayraj running `python toolkit/amd_data_analysis_toolkit/classification/generate-visualization.py' script should open the GUI if no arguments are passed. Check out this app - https://github.com/kiritigowda/mivisionx-inference-analyzer for reference.

python toolkit/amd_data_analysis_toolkit/classification/generate-visualization.py
2020-11-09 17:58:05,712 [MainThread  ] [DEBUG]  AMD Data Analysis Toolkit - Classification
Traceback (most recent call last):
  File "toolkit/amd_data_analysis_toolkit/classification/generate-visualization.py", line 71, in <module>
    main()
  File "toolkit/amd_data_analysis_toolkit/classification/generate-visualization.py", line 67, in main
    generateAnalysisOutput(argsDict)
  File "/Users/kiriti/work/adat/MIVisionX/toolkit/amd_data_analysis_toolkit/classification/adat_classification.py", line 1058, in generateAnalysisOutput
    validateConfig(argsDict)
  File "/Users/kiriti/work/adat/MIVisionX/toolkit/amd_data_analysis_toolkit/classification/adat_classification.py", line 1037, in validateConfig
    raise Exception(errorMessage)
Exception: Following required arguments were missing: 
--inference_results
--label
--output_dir
--output_name

[kiritigowda]

@bijayraj any updates on these?

[lgtm-com]

This pull request fixes 3 alerts when merging b032ff6 into 607f6ea - view on LGTM.com

fixed alerts:

• 1 for Testing equality to None
• 1 for Implicit string concatenation in a list
• 1 for Variable defined multiple times

[bijayraj]

@bijayraj any updates on these?

@kiritigowda recent commits were made to fix the issues.

[lgtm-com]

This pull request fixes 3 alerts when merging 0a9e53d into 607f6ea - view on LGTM.com

fixed alerts:

• 1 for Testing equality to None
• 1 for Implicit string concatenation in a list
• 1 for Variable defined multiple times

[kiritigowda]

@bijayraj - the following error when the new feature compare is used

python generate-visualization.py 
Traceback (most recent call last):
  File "/home/kiriti/adat/MIVisionX/toolkit/amd_data_analysis_toolkit/classification/adat_gui.py", line 161, in runGenerate
    ac.generateAnalysisOutput(self.configDict)
  File "/home/kiriti/adat/MIVisionX/toolkit/amd_data_analysis_toolkit/classification/adat_classification.py", line 1086, in generateAnalysisOutput
    diffPatch = getDifferencePatch(inputCSVFile, compareFile, ['FileName'])
  File "/home/kiriti/adat/MIVisionX/toolkit/amd_data_analysis_toolkit/classification/adat_classification.py", line 1146, in getDifferencePatch
    patch = csvdiff.diff_files(csv1, csv2, id_field)
  File "/home/kiriti/.local/lib/python2.7/site-packages/csvdiff/__init__.py", line 44, in diff_files
    ignore_columns=ignored_columns)
  File "/home/kiriti/.local/lib/python2.7/site-packages/csvdiff/patch.py", line 205, in create
    to_indexed = records.index(to_records, index_columns)
  File "/home/kiriti/.local/lib/python2.7/site-packages/csvdiff/records.py", line 58, in index
    raise InvalidKeyError('invalid column name {k} as key'.format(k=k))
csvdiff.records.InvalidKeyError: invalid column name 'FileName' as key

[lgtm-com]

This pull request fixes 3 alerts when merging 9724d63 into 607f6ea - view on LGTM.com

fixed alerts:

• 1 for Testing equality to None
• 1 for Implicit string concatenation in a list
• 1 for Variable defined multiple times

[bijayraj]

@kiritigowda New commit has fixed the problem. The problem was with incompatible compare file chosen. Extra file format checking has been added to avoid the error.

[lgtm-com]

This pull request fixes 3 alerts when merging 9724d63 into 2920638 - view on LGTM.com

fixed alerts:

• 1 for Testing equality to None
• 1 for Implicit string concatenation in a list
• 1 for Variable defined multiple times

[lgtm-com]

This pull request fixes 3 alerts when merging 9724d63 into e58c8b6 - view on LGTM.com

fixed alerts:

• 1 for Testing equality to None
• 1 for Implicit string concatenation in a list
• 1 for Variable defined multiple times

[lgtm-com]

This pull request fixes 3 alerts when merging 9724d63 into 63aec3e - view on LGTM.com

fixed alerts:

• 1 for Testing equality to None
• 1 for Implicit string concatenation in a list
• 1 for Variable defined multiple times

[lgtm-com]

This pull request fixes 3 alerts when merging 9724d63 into 82f264f - view on LGTM.com

fixed alerts:

• 1 for Testing equality to None
• 1 for Implicit string concatenation in a list
• 1 for Variable defined multiple times