Skip to content

fix(parameters): handle invalid UTF-8 transform errors - #8609

Open
Golden007-prog wants to merge 2 commits into
aws-powertools:developfrom
Golden007-prog:fix/parameter-transform-invalid-utf8
Open

Golden007-prog wants to merge 2 commits into
aws-powertools:developfrom
Golden007-prog:fix/parameter-transform-invalid-utf8

Conversation

@Golden007-prog

Copy link
Copy Markdown

Issue number: closes #8600

Summary

Changes

Resolve the transform from the parameter name before decoding bytes. Identity transforms preserve the original byte value; UTF-8 decoding for JSON/base64 transforms runs inside the existing error handler. This wraps decoding failures in TransformParameterError, or returns None when transformation errors are suppressed.

User experience

An invalid UTF-8 SecretBinary now follows the documented transformation-error policy. With transform="auto" and an unmatched suffix, both valid and invalid UTF-8 binary secrets remain bytes instead of being decoded or raising an unexpected exception.

Validation

  • Nine new regression cases failed on the original implementation and pass with the fix, including SecretsProvider calls with botocore Stubber; no AWS access required.
  • Parameter functional suite: 147 passed.
  • Ruff formatting/lint and mypy for the changed source pass.
  • Broader Windows unit/functional run: 3,594 passed, 72 skipped, 11 failed. Ten failures reproduced with the original parameter implementation (Windows paths/newlines and bytecode-loader behavior); the Datadog logging failure passed when rerun in isolation. These are disclosed rather than reported as a green full suite.
  • git diff --check passes.

The full make pr pipeline, live AWS tests, and Valkey native extra were not run. The Valkey extra could not build in this Windows environment. Implementation and verification assisted by OpenAI Codex.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@Golden007-prog
Golden007-prog requested a review from a team as a code owner October 18, 2026 12:43
@Golden007-prog
Golden007-prog requested a review from svozza October 18, 2026 12:43
@boring-cyborg

boring-cyborg Bot commented Oct 18, 2026

Copy link
Copy Markdown

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@boring-cyborg boring-cyborg Bot added the tests label Oct 18, 2026
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 18, 2026
@leandrodamascena leandrodamascena added this to the 3.39.0 milestone Oct 19, 2026
Merge upstream develop while retaining the invalid UTF-8 transform error policy and raw-byte identity behavior. Preserve the adjacent upstream AppConfig environment/application regression.

Signed-off-by: Oikantik Basu <140976895+Golden007-prog@users.noreply.github.com>
@sonarqubecloud

Copy link
Copy Markdown

@leandrodamascena leandrodamascena left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Golden007-prog, this is exactly the fix I had in mind when I opened the issue. I ran it against real Secrets Manager with invalid and valid binary secrets across every transform, and the error policy now holds everywhere: TransformParameterError when asked to raise, None in a batch when not, and raw bytes when auto finds no suffix to act on.

One thing I want to be explicit about: with transform="auto" and no matching suffix, a valid UTF-8 binary secret used to come back as str and now comes back as bytes, same as transform=None. I think that's the right behavior, the old str was a side effect of decoding too early, but I'll call it out in the release notes.

I'm approving now. This goes out in 3.39.0, so it will sit here for a while before I merge it. No rebase needed in the meantime, and you'll be credited in the release notes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: non-UTF-8 SecretBinary with transform='binary' or 'auto' raises UnicodeDecodeError instead of TransformParameterError

2 participants