localtime_r: deduplicate timezone name allocation #4069

shamb0 · 2024-12-03T03:40:57Z

This PR improves timezone string handling in localtime_r() by utilizing the newly introduced allocate_bytes() function from rust-lang/rust#133861. This change provides a more efficient approach to string deduplication and memory management.

Changes implemented:

Integration of allocate_bytes() in localtime_r():
- Replaces existing string buffer allocation logic
- Provides proper handling for null-terminated C strings
- Ensures efficient memory deduplication for timezone strings

This is part 2 of the planned improvements for timezone handling:

Part 1 (Add allocate_bytes and refactor allocate_str in InterpCx for raw byte… rust#133861): Added allocate_bytes() to InterpCx
Part 2 (this PR): Implementing timezone string handling using allocate_bytes()

Related: rust-lang/rust#133861

shamb0 · 2024-12-03T03:41:54Z

@rustbot ready

RalfJung · 2024-12-03T07:38:55Z

Thanks for the PR! However, I am afraid this is the wrong approach. I have added a comment in #4025 laying out the right way to do deduplication here.

Resolves: #4025

Please write comments like this in a way that github recognizes so that issues get automatically closed. For instance: "Fixes #4025".

shamb0 · 2024-12-03T09:13:00Z

Thanks for the PR! However, I am afraid this is the wrong approach. I have added a comment in #4025 laying out the right way to do deduplication here.

Resolves: #4025

Please write comments like this in a way that github recognizes so that issues get automatically closed. For instance: "Fixes #4025".

Thank you, @RalfJung, for providing detailed implementation guidelines.

I referred to your comment here: #4025 (comment).

In the current implementation, I assumed that all shims system calls are emulated in a single-thread context. Your observation is indeed correct—the implemented function is not thread-safe when multiple Miri interpreter instances are used in the same process.

I’ll begin working on the suggested guidelines, as they provide a solid starting point for refactoring the code.

RalfJung · 2024-12-03T10:52:56Z

In the current implementation, I assumed that all shims system calls are emulated in a single-thread context. Your observation is indeed correct—the implemented function is not thread-safe when multiple Miri interpreter instances are used in the same process.

Concurrency is not even needed to cause problems. One could run the interpreter to completion once, and then afterwards start a second interpreter and run that, and your cache would try to use old pointers from the first interpreter.

Generally, static should be avoided whenever possible. :)

shamb0 · 2024-12-04T16:38:09Z

Hi @RalfJung,

Completed implementing the changes requested in #4025.

Current Status:

RUSTC PR (Add allocate_bytes and refactor allocate_str in InterpCx for raw byte… rust#133861):
- Implemented allocate_bytes()
- Refactored allocate_str() to use the new function
MIRI PR (this PR):
- Implemented timezone logic using allocate_bytes()
- Verified changes locally in rust-lang/rust/src/tools/miri

Question: Since this PR depends on the RUSTC changes and isn't currently buildable independently, would it make sense to:

Close this PR
Move the MIRI-specific changes to the RUSTC PR

Looking forward to your guidance on how to proceed.

RalfJung · 2024-12-04T16:39:50Z

Question: Since this PR depends on the RUSTC changes and isn't currently buildable independently, would it make sense to:

I'd say we land this in two stages -- rustc PR first, then this one. So don't worry about CI here for now. Just make sure that the rustc PR is ready and passes CI. :)

shamb0 · 2024-12-05T02:32:08Z

Question: Since this PR depends on the RUSTC changes and isn't currently buildable independently, would it make sense to:

I'd say we land this in two stages -- rustc PR first, then this one. So don't worry about CI here for now. Just make sure that the rustc PR is ready and passes CI. :)

Hi @RalfJung,

Thank you for the guidance on the two-stage implementation approach. I've prepared the RUSTC PR with the following implementation details:

Key Implementation Choices:

For allocate_str():
- Maintained existing metadata and layout configuration for consistency
- Preserved current behavior while using the new allocate_bytes() function
For allocate_bytes():
- Using self.tcx.types.u8 for layout
- Set MemPlaceMeta::None as the type is sized

Please let me know if any of these choices need adjustment to better align with the requirements. I'm happy to make any necessary refinements.

Looking forward to your review of the RUSTC PR before proceeding with the MIRI changes.

RalfJung · 2024-12-05T07:00:47Z

Let's discuss that in the rustc PR. :) I added it to my review queue.

(And btw, it's rustc, not RUSTC. And we usually write Miri, not MIRI. Not a big deal, just FYI :)

shamb0 · 2024-12-05T07:12:04Z

Let's discuss that in the rustc PR. :) I added it to my review queue.

(And btw, it's rustc, not RUSTC. And we usually write Miri, not MIRI. Not a big deal, just FYI :)

Thankyou :)

shamb0 · 2024-12-05T12:16:53Z

@rustbot ready

…str, r=RalfJung Add allocate_bytes and refactor allocate_str in InterpCx for raw byte… Fixes rust-lang/miri#4025 This PR introduces a new `allocate_bytes` function in InterpCx and refactors `allocate_str` to use it internally. This change improves memory allocation handling in the interpreter by: 1. Adding `allocate_bytes`: - Direct byte slice allocation support - Handles both mutable and immutable allocations - Maintains proper memory alignment and deduplication 2. Refactoring `allocate_str`: - Now uses `allocate_bytes` internally - Adds string-specific metadata handling - Preserves existing string allocation behavior This is part 1 of the planned changes to improve timezone string handling in Miri. A follow-up PR will update Miri's timezone logic to use this new allocation mechanism. Related: rust-lang/miri#4069

Rollup merge of rust-lang#133861 - shamb0:refactor_InterpCx_allocate_str, r=RalfJung Add allocate_bytes and refactor allocate_str in InterpCx for raw byte… Fixes rust-lang/miri#4025 This PR introduces a new `allocate_bytes` function in InterpCx and refactors `allocate_str` to use it internally. This change improves memory allocation handling in the interpreter by: 1. Adding `allocate_bytes`: - Direct byte slice allocation support - Handles both mutable and immutable allocations - Maintains proper memory alignment and deduplication 2. Refactoring `allocate_str`: - Now uses `allocate_bytes` internally - Adds string-specific metadata handling - Preserves existing string allocation behavior This is part 1 of the planned changes to improve timezone string handling in Miri. A follow-up PR will update Miri's timezone logic to use this new allocation mechanism. Related: rust-lang/miri#4069

RalfJung · 2024-12-09T08:48:39Z

@shamb0 I have synced Miri with the latest rustc. So please rebase this PR over the latest Miri master branch, run ./miri toolchain to sync your local toolchain, and then you should be able to use the new method you added. :)

shamb0 · 2024-12-09T09:54:20Z

@shamb0 I have synced Miri with the latest rustc. So please rebase this PR over the latest Miri master branch, run ./miri toolchain to sync your local toolchain, and then you should be able to use the new method you added. :)

Hi @RalfJung,

Thank you for the guidance! The local tests look good, and I believe the CI build should pass now. Once confirmed, we can proceed with the review and intake.

Let me know if any further enhancements are needed. :)

shamb0 · 2024-12-09T10:37:37Z

@rustbot ready

tests/pass-dep/libc/libc-time.rs

RalfJung

Thanks! Some minor comments for the tests.

The new tests have nothing to do with the deduplication change, right? That confused me for a bit.^^

Would it be worth adding a test that checks the deduplication? That should be fairly easy, just call localtime_r again and check that the tm_zone pointer stays the same.

tests/pass-dep/libc/libc-time.rs

shamb0 · 2024-12-11T13:42:36Z

Hi @RalfJung,

Thankyou! for the valuable suggestion regarding the test scenarios! It indeed helped uncover a deduplication failure during the process.

From the trace at fn allocate_bytes_dedup(), it seems like the salt value is varying:

    error: actual output differed from expected  
    Execute `./miri test --bless` to update `tests/pass-dep/libc/libc-time.stdout` to the actual output  
    --- tests/pass-dep/libc/libc-time.stdout  
    +++ <stdout output>  
    +Allocating deduplicated bytes: [  
    +    45,  
    +    48,  
    +    55,  
    +    0,  
    +], salt: 15, allocation id: a1391  
    +Allocating deduplicated bytes: [  
    +    45,  
    +    48,  
    +    55,  
    +    0,  
    +], salt: 5, allocation id: a1393  
    +Allocating deduplicated bytes: [  
    +    45,  
    +    48,  
    +    55,  
    +    0,  
    +], salt: 29, allocation id: a1395  
    +tz res1 :: 0x0000000000029b92  
    +tz res2 :: 0x0000000000029bb3  
    +tz res3 :: 0x0000000000029bd0

I hope this isn't too basic a question, but could you clarify why we need to consider the salt here? Is there any possibility to replace it with a fixed constant?

Here's the relevant snippet for reference:

    /// Allocates a sequence of bytes in the interpreter's memory with alignment 1.  
    /// This is allocated in immutable global memory and deduplicated.  
    pub fn allocate_bytes_dedup(  
        &mut self,  
        bytes: &[u8],  
    ) -> InterpResult<'tcx, Pointer<M::Provenance>> {  
        let salt = M::get_global_alloc_salt(self, None);  
        let id = self.tcx.allocate_bytes_dedup(bytes, salt);  
    }

Looking forward to your insights!

RalfJung · 2024-12-11T13:46:37Z

From the trace at fn allocate_bytes_dedup(), it seems like the salt value is varying:

Ah, right... so what the test would have to do is call localtime_r 50 times and then check that the number of different addresses we saw is at most 49 (and at least 2).

shamb0 · 2024-12-11T14:23:49Z

Hi @RalfJung,

I’ve implemented the test scenario fn test_localtime_r_multiple_calls_deduplication() based on your suggestion. Below are the captured traces and my observations:

* From the traces, I noticed approximately 50% repetition in the salt values.
* Salt values like 0, 12, 13, 22, and others appear multiple times.
* The unique salt count indicates 24 unique values across a larger number of allocations.

--- tests/pass-dep/libc/libc-time.stdout
+++ <stdout output>
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 24, allocation id: a2863
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 31, allocation id: a3683
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 21, allocation id: a4120
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 10, allocation id: a4557
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 22, allocation id: a6400
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 19, allocation id: a6837
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 28, allocation id: a7274
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 6, allocation id: a7711
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 7, allocation id: a10898
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 1, allocation id: a11334
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 28, allocation id: a7274
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 6, allocation id: a7711
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 9, allocation id: a12517
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 31, allocation id: a3683
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 26, allocation id: a13328
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 2, allocation id: a13765
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 22, allocation id: a6400
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 13, allocation id: a14691
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 20, allocation id: a15127
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 22, allocation id: a6400
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 20, allocation id: a15127
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 9, allocation id: a12517
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 17, allocation id: a21807
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 21, allocation id: a4120
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 22, allocation id: a6400
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 3, allocation id: a22991
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 5, allocation id: a23428
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 13, allocation id: a14691
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 12, allocation id: a24238
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 31, allocation id: a3683
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 17, allocation id: a21807
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 18, allocation id: a25420
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 26, allocation id: a13328
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 15, allocation id: a26231
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 15, allocation id: a26231
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 0, allocation id: a27272
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 0, allocation id: a27272
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 24, allocation id: a2863
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 12, allocation id: a24238
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 3, allocation id: a22991
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 12, allocation id: a24238
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 5, allocation id: a23428
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 14, allocation id: a30178
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 22, allocation id: a6400
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 0, allocation id: a27272
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 23, allocation id: a31476
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 13, allocation id: a14691
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 13, allocation id: a14691
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 0, allocation id: a27272
+Allocating deduplicated bytes: [45, 48, 55, 0], salt: 26, allocation id: a13328
+Number of unique tm_zone pointers: 24

RalfJung

I don't understand the point of test_localtime_r_verify_string_deduplication... that's a very complicated test but what does it test?

test_localtime_r_multiple_calls_deduplication looks great! However, please make the entire test cfg(any(...)); there's no point in even running this test on targets that don't have the tm_zone field.

tests/pass-dep/libc/libc-time.rs

shamb0 · 2024-12-12T03:58:04Z

Hi @RalfJung,

Thank you for the feedback!

The function test_localtime_r_verify_string_deduplication() was an intermediate minimal test scenario meant to demonstrate unexpected behavior related to string deduplication. However, it has since been removed to streamline the test suite.

Currently, all tests in libc-time.rs are verified using the fn allocate_bytes() function. If needed, I can update the tests to validate against the new, cleaned-up API pub fn allocate_bytes_dedup(). Please let me know if you'd like me to take this up!

Thanks again for your suggestions.

tests/pass-dep/libc/libc-time.rs

RalfJung · 2024-12-12T06:33:02Z

Currently, all tests in libc-time.rs are verified using the fn allocate_bytes() function. If needed, I can update the tests to validate against the new, cleaned-up API pub fn allocate_bytes_dedup(). Please let me know if you'd like me to take this up!

allocate_bytes_dedup doesn't exist yet in the version of rustc that Miri uses. So just go ahead with allocate_bytes for now. If a rustup PR lands before this PR lands, you'll have to update, otherwise I will do the update as part of the rustup.

shamb0 · 2024-12-12T06:58:15Z

Currently, all tests in libc-time.rs are verified using the fn allocate_bytes() function. If needed, I can update the tests to validate against the new, cleaned-up API pub fn allocate_bytes_dedup(). Please let me know if you'd like me to take this up!

allocate_bytes_dedup doesn't exist yet in the version of rustc that Miri uses. So just go ahead with allocate_bytes for now. If a rustup PR lands before this PR lands, you'll have to update, otherwise I will do the update as part of the rustup.

Thankyou :), I hope PR is ready for intake review 👍🏾

RalfJung · 2024-12-12T07:03:16Z

Remember to write this when the PR is ready :)
@rustbot ready

RalfJung · 2024-12-12T07:05:41Z

tests/pass-dep/libc/libc-time.rs

-    const TIME_SINCE_EPOCH: libc::time_t = 1712475836;
-    let custom_time_ptr = &TIME_SINCE_EPOCH;
-    let mut tm = libc::tm {
+// Helper function to create an empty tm struct.


Suggested change

// Helper function to create an empty tm struct.

/// Helper function to create an empty tm struct.

/// indicates a doc comment that documents the following function.

Please also do that with the other similar comments in this file.

Note that I wrote this:

Please also do that with the other similar comments in this file.

Updated doc comments for all newly introduced function headers.

RalfJung · 2024-12-12T07:07:52Z

tests/pass-dep/libc/libc-time.rs

+
+    let mut unique_pointers = std::collections::HashSet::new();
+
+    unsafe {


Please make this unsafe block smaller. Only the call to localtime_r needs to be inside of it, I think?

Good One, Thankyou :)

This one is not resolved yet, either.

I modified the code, but I'm wondering why it hasn't been committed :)

for i in 0..NUM_CALLS { let timestamp = TIME_SINCE_EPOCH_BASE + (i as libc::time_t * 3600); // Increment by 1 hour for each call let mut tm: libc::tm = create_empty_tm(); let tm_ptr = unsafe { libc::localtime_r(&timestamp, &mut tm) }; assert!(!tm_ptr.is_null(), "localtime_r failed for timestamp {timestamp}"); unique_pointers.insert(tm.tm_zone); }

I'm afraid I can't help you with that -- you have to commit and push your changes as usual.

RalfJung · 2024-12-12T07:40:48Z

@rustbot author
Please post @rustbot ready when the PR is ready for the next round of review.

shamb0 · 2024-12-12T07:43:28Z

@rustbot ready

RalfJung · 2024-12-12T07:49:06Z

No this is not ready yet, some of my comments from the previous round have not been resolved.
@rustbot author

Signed-off-by: shamb0 <[email protected]>

shamb0 · 2024-12-12T10:19:47Z

@rustbot ready

RalfJung · 2024-12-12T10:48:48Z

Looks great, thanks. :)

rustbot added the S-waiting-on-review Status: Waiting for a review to complete label Dec 3, 2024

shamb0 mentioned this pull request Dec 4, 2024

Add allocate_bytes and refactor allocate_str in InterpCx for raw byte… rust-lang/rust#133861

Merged

shamb0 force-pushed the string_deduplication_for_localtime_r branch from 4745d81 to 7d874af Compare December 9, 2024 09:50

saethlin reviewed Dec 9, 2024

View reviewed changes

tests/pass-dep/libc/libc-time.rs Show resolved Hide resolved

RalfJung reviewed Dec 11, 2024

View reviewed changes

tests/pass-dep/libc/libc-time.rs Outdated Show resolved Hide resolved

tests/pass-dep/libc/libc-time.rs Outdated Show resolved Hide resolved

shamb0 force-pushed the string_deduplication_for_localtime_r branch from 3e121f0 to 8e71c74 Compare December 11, 2024 13:41

shamb0 force-pushed the string_deduplication_for_localtime_r branch from 8e71c74 to df4d483 Compare December 11, 2024 14:22

RalfJung reviewed Dec 11, 2024

View reviewed changes

tests/pass-dep/libc/libc-time.rs Outdated Show resolved Hide resolved

shamb0 force-pushed the string_deduplication_for_localtime_r branch 2 times, most recently from cfd6df9 to 332dc52 Compare December 12, 2024 03:53

RalfJung reviewed Dec 12, 2024

View reviewed changes

tests/pass-dep/libc/libc-time.rs Outdated Show resolved Hide resolved

RalfJung reviewed Dec 12, 2024

View reviewed changes

tests/pass-dep/libc/libc-time.rs Outdated Show resolved Hide resolved

shamb0 force-pushed the string_deduplication_for_localtime_r branch from 332dc52 to 8c5bc25 Compare December 12, 2024 06:53

RalfJung changed the title ~~localtime_r, Ensure string deduplication with a single buffer~~ localtime_r: deduplicate timezone name allocation Dec 12, 2024

RalfJung reviewed Dec 12, 2024

View reviewed changes

rustbot added S-waiting-on-author Status: Waiting for the PR author to address review comments and removed S-waiting-on-review Status: Waiting for a review to complete labels Dec 12, 2024

rustbot added S-waiting-on-review Status: Waiting for a review to complete and removed S-waiting-on-author Status: Waiting for the PR author to address review comments labels Dec 12, 2024

rustbot added S-waiting-on-author Status: Waiting for the PR author to address review comments and removed S-waiting-on-review Status: Waiting for a review to complete labels Dec 12, 2024

Improve timezone handling in 'localtime_r()' using 'allocate_bytes()'

ab15d13

Signed-off-by: shamb0 <[email protected]>

shamb0 force-pushed the string_deduplication_for_localtime_r branch from b2153b6 to ab15d13 Compare December 12, 2024 10:18

rustbot added S-waiting-on-review Status: Waiting for a review to complete and removed S-waiting-on-author Status: Waiting for the PR author to address review comments labels Dec 12, 2024

RalfJung added this pull request to the merge queue Dec 12, 2024

Merged via the queue into rust-lang:master with commit ebaf27c Dec 12, 2024
7 checks passed

	// Helper function to create an empty tm struct.
	/// Helper function to create an empty tm struct.


		let mut unique_pointers = std::collections::HashSet::new();

		unsafe {

localtime_r: deduplicate timezone name allocation #4069

localtime_r: deduplicate timezone name allocation #4069

Conversation

shamb0 commented Dec 3, 2024 • edited Loading

shamb0 commented Dec 3, 2024 • edited Loading

RalfJung commented Dec 3, 2024 • edited Loading

shamb0 commented Dec 3, 2024

RalfJung commented Dec 3, 2024

shamb0 commented Dec 4, 2024

RalfJung commented Dec 4, 2024

shamb0 commented Dec 5, 2024

RalfJung commented Dec 5, 2024

shamb0 commented Dec 5, 2024

shamb0 commented Dec 5, 2024

RalfJung commented Dec 9, 2024

shamb0 commented Dec 9, 2024

shamb0 commented Dec 9, 2024

RalfJung left a comment

Choose a reason for hiding this comment

shamb0 commented Dec 11, 2024

RalfJung commented Dec 11, 2024

shamb0 commented Dec 11, 2024

RalfJung left a comment

Choose a reason for hiding this comment

shamb0 commented Dec 12, 2024

RalfJung commented Dec 12, 2024

shamb0 commented Dec 12, 2024

RalfJung commented Dec 12, 2024

RalfJung Dec 12, 2024

Choose a reason for hiding this comment

RalfJung Dec 12, 2024

Choose a reason for hiding this comment

shamb0 Dec 12, 2024

Choose a reason for hiding this comment

RalfJung Dec 12, 2024 • edited Loading

Choose a reason for hiding this comment

shamb0 Dec 12, 2024

Choose a reason for hiding this comment

RalfJung Dec 12, 2024

Choose a reason for hiding this comment

shamb0 Dec 12, 2024

Choose a reason for hiding this comment

RalfJung Dec 12, 2024

Choose a reason for hiding this comment

RalfJung commented Dec 12, 2024

shamb0 commented Dec 12, 2024

RalfJung commented Dec 12, 2024

shamb0 commented Dec 12, 2024

RalfJung commented Dec 12, 2024

shamb0 commented Dec 3, 2024 •

edited

Loading

shamb0 commented Dec 3, 2024 •

edited

Loading

RalfJung commented Dec 3, 2024 •

edited

Loading

RalfJung Dec 12, 2024 •

edited

Loading