Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FIX: Support UTF-8 encoding for JSON files #1357

Merged
merged 7 commits into from
Jan 1, 2025

Conversation

scott-huberty
Copy link
Contributor

@scott-huberty scott-huberty commented Dec 19, 2024

fixes #1356

_write_json is used a lot in the MNE-BIDS codebase... I want to get your feedback on whether adding a new parameter ensure_ascii=False to _write_tsv is the right way to go?

Because I'm not sure if there are cases in MNE-BIDS where we do want to enforce ASCII encoding during JSON writing.. If so, we could change the default to ensure_ascii=True, and pass False explicitly for the cases where we want to support UTF-8?

cc @sappelhoff @hoechenberger @larsoner

TODO

  • add test

@scott-huberty scott-huberty marked this pull request as draft December 19, 2024 21:27
@scott-huberty scott-huberty changed the title WIP: add ensure_ascii parameter to _write_json WIP: Support UTF-8 encoding for JSON files Dec 19, 2024
@hoechenberger
Copy link
Member

hoechenberger commented Dec 19, 2024

I don't think we want to convert to ASCII for any JSON we're writing to disk. Hence, we don't need a parameter to control this behavior :)

TIL: That json.loads will always convert unicode. So to test that unicode was properly encoded while writing to disk, I had to had to just read the text on disk without the json module
@scott-huberty scott-huberty marked this pull request as ready for review December 20, 2024 18:15
Copy link
Member

@drammock drammock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, nice adaptation of the existing test. One suggested improvement, see below

mne_bids/tests/test_write.py Outdated Show resolved Hide resolved
Instead of closing and re-opening the file, rewind the "playhead" to the start of the open file, then use fid.read() as usual

Co-authored-by: Daniel McCloy <[email protected]>
@scott-huberty scott-huberty changed the title WIP: Support UTF-8 encoding for JSON files FIX: Support UTF-8 encoding for JSON files Dec 20, 2024
@sappelhoff sappelhoff enabled auto-merge (squash) January 1, 2025 15:53
Copy link

codecov bot commented Jan 1, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.42%. Comparing base (3492fa0) to head (d96b16a).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1357   +/-   ##
=======================================
  Coverage   97.42%   97.42%           
=======================================
  Files          40       40           
  Lines        8918     8921    +3     
=======================================
+ Hits         8688     8691    +3     
  Misses        230      230           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sappelhoff sappelhoff merged commit 3f59b0e into mne-tools:main Jan 1, 2025
23 checks passed
@sappelhoff
Copy link
Member

Thanks @scott-huberty

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

UTF-8 Encoding is not respected by make_dataset_description
4 participants