Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

qsv 0.138.0 #196802

Merged
merged 2 commits into from
Nov 6, 2024
Merged

qsv 0.138.0 #196802

merged 2 commits into from
Nov 6, 2024

Conversation

BrewTestBot
Copy link
Member

Created by brew bump

Created with brew bump-formula-pr.

release notes 
## Highlights:
* __:star: New `template` command for rendering templates with CSV data.__  
This should allow users to generate very complex documents (Form letters, JSON/XML files, etc.) with the powerful [MiniJinja template engine](https://docs.rs/minijinja/latest/minijinja/) ([Example template](https://github.com/jqnatividad/qsv/blob/master/scripts/template.tpl)).   

    

  • :star: New lookup module for fetching reference data from remote and local files.
    
In addition to the typical http/https schemes for remote files, qsv adds two additional schemes - CKAN:// and datHere://, fetching lookup data from a CKAN site or datHere maintained reference data respectively. The lookup module has simple file-based caching as well to minimize repeated fetching of typically static reference data (default cache age: 600 seconds).
    
The lookup module is now being used by the luau (for its qsv_register_lookup helper) and validate (for its dynamicEnum custom JSON Schema keyword) commands. More commands will take advantage of this module over time (e.g. apply, geocode, template, sqlp, etc.) to do extended lookups (e.g. lookup Census information given spatiotemporal data - like demographic info of a Census tract).
    • 

  • :sparkles: Enhanced fetchpost with MiniJinja templating for payload construction.
    
Previously, fetchpost was limited to posting url-encoded HTML Form data. Now with the --payload-tpl and --content-type options, users can render and post request bodies using MiniJinja using other content types as well (typically application/json, text/plain, multipart/form-data).
    • 

  • :sparkles: Improved Polars integration with automatic schema detection
    
The joinp and sqlp commands now use qsv's stats cache to automatically determine column data types, rather than having Polars scan a sample of rows. This provides two key benefits:

      

    1. Faster execution by skipping Polars' schema inference step
      2. 

    2. More accurate data type detection since the stats cache analyzes the entire dataset, not just a sample
      3. 

    

    • 

  • :running: fast-float2 crate for faster float parsing
    
Casting string/bytes to float is now much faster (2 to 8x faster than Rust's standard library) with fast-float2.
    • 

  • :muscle: Major dependency updates including Polars 0.44.2, Luau 0.650, mlua 0.10.0 and jsonschema 0.26.1
    
These core crates underpin much of qsv's functionality. Using the latest version of these crates allow qsv to stay true to its goal of being the fastest and most comprehensive data-wrangling toolkit.
    • 





Added



Changed



Fixed



Removed


    

  • removed need to set RAYON_NUM_THREADS env var and just call the Rayon API directly dathere/qsv@aa6ef89
    • 

  • removed unneeded create_dir_all_threadsafe helper now that std::create_dir_all is threadsafe dathere/qsv@d0af83b
    • 



Full Changelog: dathere/qsv@0.137.0...0.138.0

@github-actions github-actions bot added rust Rust use is a significant feature of the PR or issue bump-formula-pr PR was created using `brew bump-formula-pr` labels Nov 6, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024

🤖 An automated task has requested bottles to be published to this PR.

@github-actions github-actions bot added the CI-published-bottle-commits The commits for the built bottles have been pushed to the PR branch. label Nov 6, 2024
@BrewTestBot BrewTestBot enabled auto-merge November 6, 2024 04:14
@BrewTestBot BrewTestBot added this pull request to the merge queue Nov 6, 2024
Merged via the queue into master with commit 3bb6132 Nov 6, 2024
15 checks passed
@BrewTestBot BrewTestBot deleted the bump-qsv-0.138.0 branch November 6, 2024 04:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] approved these changes

@branchvincent branchvincent branchvincent approved these changes

Assignees
No one assigned
Labels
bump-formula-pr PR was created using `brew bump-formula-pr` CI-published-bottle-commits The commits for the built bottles have been pushed to the PR branch. rust Rust use is a significant feature of the PR or issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@BrewTestBot @branchvincent