Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(tools): Updated get.py with ability to verify extracted files and skip if ok #8720

Merged
merged 20 commits into from
Jun 3, 2024
Merged

feat(tools): Updated get.py with ability to verify extracted files and skip if ok #8720

merged 20 commits into from
Jun 3, 2024

Conversation

me-no-dev
Copy link
Member

Redo of: #8496

New features:

  • Detects corrupted archives (for example not completely downloaded) and automatically re-downloads them (the corrupted archive is renamed with a suffix ".corrupted)
  • Detects incomplete extraction (checks only missing filenames - intentional corruption such as changing contents of files is not expected and not detected) - incomplete extraction triggers new and complete extraction
  • Updated download report to show download speed and elapsed time
  • New parameters:
    • '-v' Verbose - displays more info
    • '-d' Force download all archives (skipping checks)
    • '-e' Force extraction (skipping checks)
    • '-f' Force both download and extraction (skipping checks)

Test scenarios (only on Linux)

  • Fresh - nothing is downloaded, nothing is extracted
    • Time to complete: 2m51.901s
  • Downloaded - only full extraction
    • Time to complete: 1m22.337s
  • Already installed - only performing checks - nothing is missing
    • Time to complete: 0m45.402s
  • Multiple corruption - 1 missing archive, 1 archive not downloaded properly, 1 archive not extracted at all, 1 file missing
    • Handled without error or crashing - all tools installed

@me-no-dev me-no-dev mentioned this pull request Oct 5, 2023
Closed
@VojtechBartoska VojtechBartoska added this to the 3.0.0-Aplha3 milestone Nov 28, 2023
@VojtechBartoska
Copy link
Contributor

@me-no-dev ready for review?

@me-no-dev
Copy link
Member Author

@VojtechBartoska because of how the libs work now and for ease of development, we will probably close this.

@VojtechBartoska VojtechBartoska added the Status: Awaiting triage Issue is waiting for triage label Nov 29, 2023
@me-no-dev me-no-dev closed this Nov 29, 2023
@me-no-dev me-no-dev deleted the feature/get_py_verify branch November 29, 2023 13:50
@VojtechBartoska VojtechBartoska removed the Status: Awaiting triage Issue is waiting for triage label Nov 30, 2023
@lucasssvaz lucasssvaz restored the feature/get_py_verify branch May 22, 2024 16:12
@lucasssvaz lucasssvaz reopened this May 22, 2024
@lucasssvaz
Copy link
Collaborator

Reopened as this will be required to optimize the CI.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 22, 2024 •
edited
Loading

Warnings
⚠️

Some issues found for the commit messages in this PR:

  • the commit message "Fix formatting and add checksum comparison":
    • summary looks empty
    • type/action looks empty
  • the commit message "Make the linter happy":
    • summary looks empty
    • type/action looks empty
  • the commit message "Minor fixes":
    • summary looks empty
    • type/action looks empty
  • the commit message "Push binary to tools":
    • summary looks empty
    • type/action looks empty
  • the commit message "Push binary to tools":
    • summary looks empty
    • type/action looks empty
  • the commit message "Push binary to tools":
    • summary looks empty
    • type/action looks empty
  • the commit message "Push binary to tools":
    • summary looks empty
    • type/action looks empty
  • the commit message "Push binary to tools":
    • summary looks empty
    • type/action looks empty
  • the commit message "Push binary to tools":
    • summary looks empty
    • type/action looks empty
  • the commit message "Skip checksum for libs":
    • summary looks empty
    • type/action looks empty
  • the commit message "Updated get.py with ability to verify extracted files and skip if ok":
    • summary looks empty
    • type/action looks empty
  • the commit message "Updates":
    • summary looks empty
    • type/action looks empty
  • the commit message "fix(get.py): Fix header comment":
    • summary looks too short
  • the commit message "refactor(get.py): Small improvements":
    • summary looks too short

Please fix these commit messages - here are some basic tips:

  • follow Conventional Commits style
  • correct format of commit message should be: <type/action>(<scope/component>): <summary>, for example fix(esp32): Fixed startup timeout issue
  • allowed types are: change,ci,docs,feat,fix,refactor,remove,revert,test
  • sufficiently descriptive message summary should be between 20 to 72 characters and start with upper case letter
  • avoid Jira references in commit messages (unavailable/irrelevant for our customers)

TIP: Install pre-commit hooks and run this check when committing (uses the Conventional Precommit Linter).
⚠️ Please consider squashing your 19 commits (simplifying branch history).

👋 Hello me-no-dev, we appreciate your contribution to this project!

Click to see more instructions ...


This automated output is generated by the PR linter DangerJS, which checks if your Pull Request meets the project's requirements and helps you fix potential issues.

DangerJS is triggered with each push event to a Pull Request and modify the contents of this comment.

Please consider the following:
- Danger mainly focuses on the PR structure and formatting and can't understand the meaning behind your code or changes.
- Danger is not a substitute for human code reviews; it's still important to request a code review from your colleagues.
- Resolve all warnings (⚠️ ) before requesting a review from human reviewers - they will appreciate it.
- To manually retry these Danger checks, please navigate to the Actions tab and re-run last Danger workflow.

Review and merge process you can expect ...


We do welcome contributions in the form of bug reports, feature requests and pull requests.

1. An internal issue has been created for the PR, we assign it to the relevant engineer.
2. They review the PR and either approve it or ask you for changes or clarifications.
3. Once the GitHub PR is approved we do the final review, collect approvals from core owners and make sure all the automated tests are passing.
- At this point we may do some adjustments to the proposed change, or extend it by adding tests or documentation.
4. If the change is approved and passes the tests it is merged into the default branch.

Generated by 🚫 dangerJS against 7d8ca2f

@lucasssvaz lucasssvaz self-assigned this May 22, 2024
@lucasssvaz lucasssvaz removed this from the 3.0.0-Aplha3 milestone May 22, 2024
@lucasssvaz lucasssvaz force-pushed the feature/get_py_verify branch from 37bb47c to 40d16ca Compare May 22, 2024 16:37
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 22, 2024 •
edited
Loading

Unit Test Results

0 tests   0 ✔️  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit 2df340f.

♻️ This comment has been updated with latest results.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 22, 2024 •
edited
Loading

Memory usage test (comparing PR against master branch)

The table below shows the summary of memory usage change (decrease - increase) in bytes and percentage for each target.

MemoryFLASH [bytes]FLASH [%]RAM [bytes]RAM [%]
TargetDECINCDECINCDECINCDECINC
ESP32S30⚠️ +200.000.00000.000.00
ESP32S2💚 -12⚠️ +200.000.00000.000.00
ESP32C30⚠️ +380.000.00000.000.00
ESP32C60⚠️ +380.000.00000.000.00
ESP32H2000.000.00000.000.00
ESP320⚠️ +200.000.00000.000.00
Click to expand the detailed deltas report [usage change in BYTES]
TargetESP32S3ESP32S2ESP32C3ESP32C6ESP32H2ESP32
ExampleFLASHRAMFLASHRAMFLASHRAMFLASHRAMFLASHRAMFLASHRAM
ArduinoOTA/examples/BasicOTA⚠️ +12000⚠️ +380⚠️ +380--00
AsyncUDP/examples/AsyncUDPClient00000000--00
AsyncUDP/examples/AsyncUDPMulticastServer00000000--00
AsyncUDP/examples/AsyncUDPServer00000000--00
BLE/examples/BLE5_extended_scan00--000000--
BLE/examples/BLE5_multi_advertising00--000000--
BLE/examples/BLE5_periodic_advertising00--000000--
BLE/examples/BLE5_periodic_sync00--000000--
BLE/examples/Beacon_Scanner00--00000000
BLE/examples/Client00--00000000
BLE/examples/EddystoneTLM_Beacon00--0000--00
BLE/examples/EddystoneURL_Beacon00--0000--00
BLE/examples/Notify00--00000000
BLE/examples/Scan00--00000000
BLE/examples/Server00--00000000
BLE/examples/Server_multiconnect00--00000000
BLE/examples/UART00--00000000
BLE/examples/Write00--00000000
BLE/examples/iBeacon00--00000000
DNSServer/examples/CaptivePortal00000000--00
EEPROM/examples/eeprom_class000000000000
EEPROM/examples/eeprom_extra000000000000
EEPROM/examples/eeprom_write000000000000
ESP32/examples/AnalogOut/LEDCFade000000000000
ESP32/examples/AnalogOut/LEDCSoftwareFade000000000000
ESP32/examples/AnalogOut/SigmaDelta000000000000
ESP32/examples/AnalogOut/ledcFrequency000000000000
ESP32/examples/AnalogOut/ledcWrite_RGB000000000000
ESP32/examples/AnalogRead000000000000
ESP32/examples/AnalogReadContinuous000000000000
ESP32/examples/ArduinoStackSize000000000000
ESP32/examples/CI/CIBoardsTest000000000000
ESP32/examples/Camera/CameraWebServer0000------00
ESP32/examples/ChipID/GetChipID000000000000
ESP32/examples/DeepSleep/ExternalWakeUp0000------00
ESP32/examples/DeepSleep/TimerWakeUp00000000--00
ESP32/examples/DeepSleep/TouchWakeUp0000------00
ESP32/examples/FreeRTOS/BasicMultiThreading000000000000
ESP32/examples/FreeRTOS/Mutex000000000000
ESP32/examples/FreeRTOS/Queue000000000000
ESP32/examples/FreeRTOS/Semaphore000000000000
ESP32/examples/GPIO/BlinkRGB000000000000
ESP32/examples/GPIO/FunctionalInterrupt000000000000
ESP32/examples/GPIO/FunctionalInterruptStruct000000000000
ESP32/examples/GPIO/GPIOInterrupt000000000000
ESP32/examples/HWCDC_Events00--000000--
ESP32/examples/MacAddress/GetMacAddress000000000000
ESP32/examples/RMT/RMTCallback000000000000
ESP32/examples/RMT/RMTLoopback000000000000
ESP32/examples/RMT/RMTReadXJT000000000000
ESP32/examples/RMT/RMTWriteNeoPixel000000000000
ESP32/examples/RMT/RMT_CPUFreq_Test000000000000
ESP32/examples/RMT/RMT_EndOfTransmissionState000000000000
ESP32/examples/RMT/RMT_LED_Blink000000000000
ESP32/examples/ResetReason/ResetReason000000000000
ESP32/examples/ResetReason/ResetReason2000000000000
ESP32/examples/Serial/BaudRateDetect_Demo000000000000
ESP32/examples/Serial/OnReceiveError_BREAK_Demo000000000000
ESP32/examples/Serial/OnReceive_Demo000000000000
ESP32/examples/Serial/RS485_Echo_Demo000000000000
ESP32/examples/Serial/RxFIFOFull_Demo000000000000
ESP32/examples/Serial/RxTimeout_Demo000000000000
ESP32/examples/Serial/Serial_All_CPU_Freqs000000000000
ESP32/examples/Serial/Serial_STD_Func_OnReceive000000000000
ESP32/examples/Serial/onReceiveExample000000000000
ESP32/examples/TWAI/TWAIreceive000000000000
ESP32/examples/TWAI/TWAItransmit000000000000
ESP32/examples/Template/ExampleTemplate000000000000
ESP32/examples/Time/SimpleTime00000000--00
ESP32/examples/Timer/RepeatTimer000000000000
ESP32/examples/Timer/WatchdogTimer000000000000
ESP32/examples/Touch/TouchButtonV20000--------
ESP32/examples/Touch/TouchInterrupt0000------00
ESP32/examples/Touch/TouchRead0000------00
ESP32/examples/Utilities/HEXBuilder000000000000
ESP32/examples/Utilities/MD5Builder000000000000
ESP32/examples/Utilities/SHA1Builder000000000000
ESP_I2S/examples/ES8388_loopback000000000000
ESP_I2S/examples/Record_to_WAV00--------00
ESP_I2S/examples/Simple_tone000000000000
ESP_NOW/examples/ESP_NOW_Broadcast_Master00000000--00
ESP_NOW/examples/ESP_NOW_Broadcast_Slave00000000--00
ESP_NOW/examples/ESP_NOW_Network00000000--00
ESP_NOW/examples/ESP_NOW_Serial00000000--00
ESP_SR/examples/Basic00----------
ESPmDNS/examples/mDNS-SD_Extended00000000--00
ESPmDNS/examples/mDNS_Web_Server00000000--00
Ethernet/examples/ETH_W5500_Arduino_SPI000000000000
Ethernet/examples/ETH_W5500_IDF_SPI000000000000
FFat/examples/FFat_Test000000000000
FFat/examples/FFat_time00000000--00
HTTPClient/examples/Authorization00000000--00
HTTPClient/examples/BasicHttpClient00000000--00
HTTPClient/examples/BasicHttpsClient00000000--00
HTTPClient/examples/HTTPClientEnterprise00000000--00
HTTPClient/examples/ReuseConnection00000000--00
HTTPClient/examples/StreamHttpClient00000000--00
HTTPUpdate/examples/httpUpdate00000000--00
HTTPUpdate/examples/httpUpdateSPIFFS00000000--00
HTTPUpdate/examples/httpUpdateSecure00000000--00
HTTPUpdateServer/examples/WebUpdater00000000--00
Insights/examples/DiagnosticsSmokeTest000000----00
Insights/examples/MinimalDiagnostics000000----00
LittleFS/examples/LITTLEFS_test000000000000
LittleFS/examples/LITTLEFS_time00000000--00
NetBIOS/examples/ESP_NBNST00000000--00
NetworkClientSecure/examples/WiFiClientInsecure00000000--00
NetworkClientSecure/examples/WiFiClientPSK00000000--00
NetworkClientSecure/examples/WiFiClientSecure00000000--00
NetworkClientSecure/examples/WiFiClientSecureEnterprise00000000--00
NetworkClientSecure/examples/WiFiClientSecureProtocolUpgrade00000000--00
NetworkClientSecure/examples/WiFiClientShowPeerCredentials00000000--00
NetworkClientSecure/examples/WiFiClientTrustOnFirstUse00000000--00
PPP/examples/PPP_Basic000000000000
Preferences/examples/Prefs2Struct000000000000
Preferences/examples/StartCounter000000000000
RainMaker/examples/RMakerCustom000000----00
RainMaker/examples/RMakerCustomAirCooler000000----00
RainMaker/examples/RMakerSonoffDualR3000000----00
RainMaker/examples/RMakerSwitch000000----00
SD/examples/SD_Test000000000000
SD/examples/SD_time00000000--00
SD_MMC/examples/SDMMC_Test00--------00
SD_MMC/examples/SDMMC_time00--------00
SPI/examples/SPI_Multiple_Buses0000------00
SPIFFS/examples/SPIFFS_Test000000000000
SPIFFS/examples/SPIFFS_time00000000--00
SimpleBLE/examples/SimpleBleDevice00----000000
TFLiteMicro/examples/hello_world000000000000
Ticker/examples/Blinker000000000000
Ticker/examples/TickerBasic000000000000
Ticker/examples/TickerParameter000000000000
USB/examples/CompositeDevice0000--------
USB/examples/ConsumerControl0000--------
USB/examples/CustomHIDDevice0000--------
USB/examples/FirmwareMSC0000--------
USB/examples/Gamepad0000--------
USB/examples/HIDVendor0000--------
USB/examples/Keyboard/KeyboardLogout0000--------
USB/examples/Keyboard/KeyboardMessage0000--------
USB/examples/Keyboard/KeyboardReprogram0000--------
USB/examples/Keyboard/KeyboardSerial0000--------
USB/examples/KeyboardAndMouseControl0000--------
USB/examples/MIDI/MidiController0000--------
USB/examples/MIDI/MidiInterface0000--------
USB/examples/MIDI/MidiMusicBox0000--------
USB/examples/MIDI/ReceiveMidi0000--------
USB/examples/Mouse/ButtonMouseControl0000--------
USB/examples/SystemControl0000--------
USB/examples/USBMSC0000--------
USB/examples/USBSerial0000--------
USB/examples/USBVendor0000--------
Update/examples/AWS_S3_OTA_Update00000000--00
Update/examples/HTTPS_OTA_Update00000000--00
Update/examples/HTTP_Client_AES_OTA_Update00000000--00
Update/examples/HTTP_Server_AES_OTA_Update00000000--00
Update/examples/OTAWebUpdater00000000--00
Update/examples/SD_Update000000000000
WebServer/examples/AdvancedWebServer00000000--00
WebServer/examples/FSBrowser00000000--00
WebServer/examples/HelloServer00000000--00
WebServer/examples/HttpAdvancedAuth00⚠️ +120⚠️ +380⚠️ +380--⚠️ +40
WebServer/examples/HttpAuthCallback⚠️ +120⚠️ +120⚠️ +380⚠️ +380--⚠️ +40
WebServer/examples/HttpAuthCallbackInline00⚠️ +40⚠️ +380⚠️ +380--⚠️ +40
WebServer/examples/HttpBasicAuth00💚 -120⚠️ +380⚠️ +380--⚠️ +40
WebServer/examples/HttpBasicAuthSHA1⚠️ +120⚠️ +120⚠️ +380⚠️ +380--00
WebServer/examples/HttpBasicAuthSHA1orBearerToken⚠️ +40⚠️ +120⚠️ +380⚠️ +380--00
WebServer/examples/MultiHomedServers00000000--00
WebServer/examples/PathArgServer00000000--00
WebServer/examples/SDWebServer00000000--00
WebServer/examples/SimpleAuthentification00000000--00
WebServer/examples/UploadHugeFile00000000--00
WebServer/examples/WebServer00000000--00
WebServer/examples/WebUpdate00000000--00
WiFi/examples/FTM/FTM_Initiator00000000--00
WiFi/examples/FTM/FTM_Responder00000000--00
WiFi/examples/SimpleWiFiServer00000000--00
WiFi/examples/WPS00000000--00
WiFi/examples/WiFiAccessPoint00000000--00
WiFi/examples/WiFiBlueToothSwitch00--0000--00
WiFi/examples/WiFiClient00000000--00
WiFi/examples/WiFiClientBasic00000000--00
WiFi/examples/WiFiClientConnect00000000--00
WiFi/examples/WiFiClientEnterprise00000000--00
WiFi/examples/WiFiClientEvents00000000--00
WiFi/examples/WiFiClientStaticIP⚠️ +200⚠️ +200⚠️ +160⚠️ +160--⚠️ +200
WiFi/examples/WiFiIPv6⚠️ +12000⚠️ +380⚠️ +380--00
WiFi/examples/WiFiMulti00000000--00
WiFi/examples/WiFiMultiAdvanced00000000--00
WiFi/examples/WiFiScan00000000--00
WiFi/examples/WiFiScanAsync00000000--00
WiFi/examples/WiFiScanDualAntenna00000000--00
WiFi/examples/WiFiSmartConfig00000000--00
WiFi/examples/WiFiTelnetToSerial00000000--00
WiFi/examples/WiFiUDPClient00⚠️ +160⚠️ +380⚠️ +380--⚠️ +40
WiFiProv/examples/WiFiProv00000000--00
Wire/examples/WireMaster000000000000
Wire/examples/WireScan000000000000
Wire/examples/WireSlave000000000000
BluetoothSerial/examples/DiscoverConnect----------00
BluetoothSerial/examples/GetLocalMAC----------00
BluetoothSerial/examples/SerialToSerialBT----------00
BluetoothSerial/examples/SerialToSerialBTM----------00
BluetoothSerial/examples/SerialToSerialBT_Legacy----------00
BluetoothSerial/examples/SerialToSerialBT_SSP----------00
BluetoothSerial/examples/bt_classic_device_discovery----------00
BluetoothSerial/examples/bt_remove_paired_devices----------00
ESP32/examples/DeepSleep/SmoothBlink_ULP_Code----------00
ESP32/examples/Touch/TouchButton----------00
Ethernet/examples/ETH_LAN8720----------00
Ethernet/examples/ETH_TLK110----------00

@lucasssvaz lucasssvaz force-pushed the feature/get_py_verify branch from dee03c6 to ba98f0c Compare May 22, 2024 20:35
@lucasssvaz lucasssvaz removed their request for review May 23, 2024 22:08
@lucasssvaz lucasssvaz requested review from P-R-O-C-H-Y and removed request for P-R-O-C-H-Y May 23, 2024 22:08
@VojtechBartoska VojtechBartoska changed the title Updated get.py with ability to verify extracted files and skip if ok feat(tools): Updated get.py with ability to verify extracted files and skip if ok May 27, 2024
@lucasssvaz lucasssvaz mentioned this pull request May 27, 2024
Merged
@me-no-dev
Copy link
Member Author

@lucasssvaz the changes look good to me

@me-no-dev me-no-dev added the Status: Pending Merge Pull Request is ready to be merged label May 30, 2024
@lucasssvaz lucasssvaz force-pushed the feature/get_py_verify branch from adcd0f3 to 95ad38d Compare June 2, 2024 16:28
@me-no-dev
Copy link
Member Author

@lucasssvaz we need to find out why "Push binary to tools" does not trigger CI

@me-no-dev me-no-dev merged commit 00fb799 into master Jun 3, 2024
1 check passed
@me-no-dev me-no-dev deleted the feature/get_py_verify branch June 3, 2024 07:04
@lucasssvaz
Copy link
Collaborator

@lucasssvaz we need to find out why "Push binary to tools" does not trigger CI

It's due to the default behavior of GITHUB_TOKEN. In the other CI PR I fixed it (I think at least, it worked in my fork)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@P-R-O-C-H-Y P-R-O-C-H-Y P-R-O-C-H-Y approved these changes

@lucasssvaz lucasssvaz lucasssvaz approved these changes

Assignees

@lucasssvaz lucasssvaz

Labels
Status: Pending Merge Pull Request is ready to be merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@me-no-dev @VojtechBartoska @lucasssvaz @P-R-O-C-H-Y @PilnyTomas