[V4] DateTime handling adjustments for inconsistencies between local and UTC times.

[boblodgett]

Description

This PR addresses the inconsistencies of using LocalTime and UniversalTime within the SDK to consistently use UniversalTime (UTC) whenever possible as a first choice and only using LocalTime when appropriate to do so.

DevConfig commit: 8884a55
Manual changes are in these commits only: cb4f74d, 5f88660, 3f93a6d, and c072d6d
Generated changes commit: 9611cba

Note for reviewers:

• Please pay special attention to things like expiration times, data sent to and from DynamoDb, and credentials.
• DynamoDB (TestContext test [.\sdk\test\NetStandard\IntegrationTests\IntegrationTests\DynamoDB\DataModelTests.cs] gave me some problems and unspecified times can still be sent to and from DynamoDb. Testing may need to expand here).
• Pay attention to the impact of the breaking change list and how that could impact developers.
• Cosmetic changes exist in this PR. See all notes below.
• One TODO needs some discussion for SimpleDB. (Decided to leave as is for now using localtime)

Motivation and Context

Breaking changes: Changes where the behavior is now different and may or may not produce a compile time error.

• Removed obsolete properties such EndTime then changed EndTimeUtc to EndTime. This could lead to offset times if anyone was still using the marked obsolete original EndTime for example. A compile time error will occur for anyone using EndTimeUtc. Removing BackwardCompatibility properties and code in the generator is part of this change. Also id this to the manual S3 models.
• Response unmarshallers for TimeStamps and list TimeStamps for formats TimestampFormat.ISO8601 || TimestampFormat.RFC822 datetimes were being parsed into local times. Adjusted DateTime parsing to return UTC times. (BaseResponseUnmarshaller.tt, MultiValueHeaderParser.cs)
• Fixed the DateTimeUnmarshaller which was parsing datetime strings into and returning them as local time which in some cases were still getting converted back to UTC on a prior bug fix but not always. DateTime strings unmarshalled are assumed to be UTC time and will be specified and unmarshalled as UTC. (SimpleTypeUnmarshaller.cs)
• ConvertFromUnixEpochSeconds/ConvertFromUnixEpochMilliseconds incorrectly returning the Unix Epoch time as localtime instead of by definition is a UTC time. This changes the behavior where these methods were used. These were assigned to headers typically: https://github.com/search?q=repo%3Aaws%2Faws-sdk-net+ConvertFromUnixEpoch&type=code (AWSSDKUtils.cs)
• DynamoDB RetrieveDateTimeInUtc has been switched to true as the default.

Cosmetic changes: These changes were mainly so that I could keep track and ensure I looked at and ensured all code using DateTime.Now / .ToUniversalTime, .ToLocalTime, new DateTime, etc, was for a correct and valid reason. Updating to UTC will assist with validation errors at a later time.

• Timing loops such as: var elapsed = DateTime.UtcNow - StartTime;
• Ticks usage: DateTime.Now.Ticks and DateTime.UtcNow.Ticks produces the same long.
• ToFileTime usage DateTime.Now.ToFileTime => DateTime.UtcNow.ToFileTime
• Removal of extra .ToUniversalTime() calls. Example: string epochSeconds = AWSSDKUtils.ConvertToUnixEpochSecondsString(expiresOn.ToUniversalTime()); -- AWSSDKUtils.ConvertToUnixEpochSecondsString already ensures the DateTime is UTC.
• In generator changed IsDateTime to IsTimeStamp. This is because the shape is actually modeled as a TimeStamp not a C# language specific DateTime.
• Switch service specific DateTime utility methods over to runtime internal methods in StringUtils. Example: Amazon.EC2.Internal.CustomMarshallTransformations.ConvertDateTimeISOWithoutMillisecondsUtc => Amazon.Runtime.Internal.Util.StringUtils.FromDateTimeToISO8601NoMs. Deleted the service specific DateTime utility methods.
• Removed #pragma warning disable CS0612, CS0618 // Type or member is obsolete around AWSSDKUtils.CorrectedUtcNow because the property isn't marked obsolete.
• Internal Cache class internally uses UTC time. (SdkCache.cs)

Minor bug fixes: These are not breaking changes but fix slight issues in the SDK.

• Fixed Epoch date to UTC per definition where the epoch date was created in local time. (TokenBucket.cs)
• Stopped using expiry times as localtime. Changed to UTC. (InstanceProfileAWSCredentials.cs, ProcessAWSCredentials.cs, RefreshingAWSCredentials.cs)
• Ensured DateTime.Max and DateTime.Min are marked with a DateTime.Kind DateTimeKind.Utc for proper calculations.
• Instad of assuming SAML credentails are localtime then converting to UTC assume that the time given is UTC to workproperly with credential expiration being in UTC time for other credential providers. (SAMLImutableCredentials.cs)
• Console logger outputs timestamps as a UTC date incase output is sent off the local machine and for easier comparison with other UTC dates. We could add a preference flag here though. (Logger.Console.cs)
• RetryPolicies return UTC server time instead of a UTC time converted to local time. (RetryPolicy.cs)
• AWSPublicIpAddressRanges mixing UTC and local time. (AWSPublicIpAddressRanges.cs)
• GetFormattedTimestampISO8601 incorrectly creating a DateTime object as local time even though it is passed in as UTC. Then formatting it as a UTC string. Although incorrect the DaeTimeKind.Local flag didn't actually harm anything and the behavior is the same by not using it.

Testing

DryRun: Passed (https://tiny.amazon.com/b78l9mvc/IsenLink)

Manual Testing: In progress -- Example (PresignedURLs)

• S3 Presigned Urls: SigV4 and SigV2 --OK

Adding more tests where appropriate: Pending

Screenshots (if appropriate)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• My code follows the code style of this project
• My change requires a change to the documentation
• I have updated the documentation accordingly
• I have read the README document
• I have added tests to cover my changes
• All new and existing tests passed

License

• I confirm that this pull request can be released under the Apache 2 license

[normj]

Excluding my minor comments I already added I think the change looks good. I have gone through the credential expiration a couple times and everything looks like it is doing comparisons by UTC. I have done some interop testing with V3 and this branch saving and retrieving data from DynamoDB. I tested storing dates as both strings and epochs. I can confirm the data in the table was persisted the same no matter if I used V3 or this branch. The DateTime objects coming back were also the except for the expected change that V4 has the Kind set to Utc and V3 has the Kind set to Local.

We should look Roslyn/CodeAnalyzers rules that we can use to enforce the correct date behavior going forward but that can be a separate task.

[boblodgett]

Adjustments applied and a new commit has been pushed up. I will do another dry-run but nothing changed that should impact that.

[boblodgett]

TestFromJsonCanHandleAllDataTypes is the only integ test that failed. Investigating.