--- title: The AWS CLI Bug That Broke /dev/null Across Your Entire System description: AWS CLI v2 #10196 chmod'd streaming outputs to 0600, including /dev/null. July 2026 retrospective: #10215 revert, host repair, pin-and-canary checklist. url: https://www.joinhorizon.co/blog/aws-cli-chmod-dev-null-streaming-bug-2026/ datePublished: 2026-04-10T00:00:00.000Z dateModified: 2026-10-28T00:00:00.000Z author: manojkumar-varatharajan category: DevOps & CI/CD tags: aws-cli, devops, aws, cicd, s3, lambda, security, incident-report, debugging --- # The AWS CLI Bug That Broke /dev/null Across Your Entire System > AWS CLI v2 #10196 chmod'd streaming outputs to 0600, including /dev/null. July 2026 retrospective: #10215 revert, host repair, pin-and-canary checklist. On **2026-04-09**, AWS CLI v2 merged **PR #10196** (“Tighten output file permissions”), applying `0600` to streaming/S3 Select output paths. Passing `/dev/null` as a file argument chmod’d the character device system-wide. On **2026-04-10**, **PR #10215** **reverted** that change (commit `12c807c`). As of **July 2026**, treat this as a closed incident with lasting pipeline hygiene lessons, not a live CVE. **Engagement shape (anonymized):** SaaS CI fleet, ~80 GitHub Actions runners on shared AMIs; ~2 hours of failed `aws lambda invoke … /dev/null` jobs until hosts were repaired and CLI pinned past the revert. **Reproduce this:** [CLI pin + /dev/null checklist](https://www.joinhorizon.co/examples/architecture-blog-2026/cli-hardening/aws-cli-pin-and-devnull-checklist.md) · upstream [revert commit](https://github.com/aws/aws-cli/commit/12c807c352ed23e8ef4ef59045352037fb0330ea) ## What happened: a security hardening PR with unintended blast radius ### The intent behind PR #10196 On April 9, 2026, the AWS CLI team merged PR #10196 to tighten output file security. The goal was sound: whenever commands like `aws s3 cp`, `aws s3 select`, or `aws lambda invoke` write output to a file path, restrict that file to owner-only read/write permissions (`0600`) via `os.chmod()`. This is a legitimate security improvement: output files containing presigned URLs, Lambda response payloads, or S3 object metadata should not be world-readable on a shared system. The implementation was straightforward: after writing output, call `os.chmod(output_path, 0o600)`. It was simple and direct, and it turned out to be incomplete. ### The bug: chmod applied to every output path The `os.chmod()` call was applied **unconditionally** to whatever path string the caller provided, with no type checking or validation. If you passed `/dev/null`, the CLI called `os.chmod("/dev/null", 0o600)`. `/dev/null` is a **character device**, not a regular file. But `os.chmod()` works on device nodes: it changes the permission bits on the device's filesystem entry. Before the bug, `/dev/null` had permissions `crw-rw-rw-` (readable and writable by everyone). After the bug, it became `crw-------` (readable and writable by root only). Every process on the system that tried to write to `/dev/null` after that point received `EACCES: Permission denied`. That broke a device whose only job is to discard data. --- ## Which commands were affected ### aws lambda invoke This is the most common pattern that hit the bug: discarding Lambda function output in scripts or CI/CD pipelines. ```bash aws lambda invoke \ --function-name my-function \ --payload '{"key": "value"}' \ /dev/null ``` This command invokes a Lambda function and writes the response JSON to `/dev/null` to suppress output. Before the bug, this worked fine. After PR #10196 merged, the CLI would write the response and then call `os.chmod("/dev/null", 0o600)`, breaking the device node for every subsequent process on the host. ### aws s3 cp and aws s3 select Less common, but on the same code path: ```bash # Test if an S3 object exists without saving it locally aws s3 cp s3://bucket/key /dev/null # Stream query results without persisting to disk aws s3 select --expression "SELECT * FROM S3Object" /dev/null ``` Both trigger the same bug. ### What was never affected: shell-level redirection Shell redirection was never vulnerable. ```bash # These patterns are immune — shell controls the file descriptor aws lambda invoke --function-name fn /dev/stdout > /dev/null aws s3 cp s3://bucket/key - > /dev/null aws lambda invoke --function-name fn /dev/stdout 2>&1 | cat > /dev/null ``` When you use shell redirection, the shell opens the file descriptor and hands it to the CLI process. The CLI never sees `/dev/null` as a string path. It only writes to the file descriptor. The `chmod()` call cannot reach it. The bug therefore went unnoticed in codebases that used shell redirection. Only codebases passing `/dev/null` as a direct argument to the CLI were affected. --- ## How to know if your system was hit ### Symptoms If your infrastructure was running an affected AWS CLI version during the window of April 9 to 10, 2026, watch for these: - Lambda invocations in CI/CD suddenly fail with permission errors or hang indefinitely. - Shell scripts that write to /dev/null return permission denied errors, even for unrelated commands like `curl`, `git`, or `systemctl`. - Any process on the host that redirects output to /dev/null fails. - The failure appears within minutes of an AWS CLI auto-update, which makes it hard to diagnose. All of these failures start after the first AWS CLI command that targeted `/dev/null` as an output path, then spread to every other process on the host. ### Check /dev/null permissions ```bash # Check current permissions $ ls -la /dev/null crw-rw-rw- 1 root root 1, 3 Apr 9 12:00 /dev/null # If you see crw------- or any restricted form, it was affected: $ ls -la /dev/null crw------- 1 root root 1, 3 Apr 9 14:32 /dev/null # Restore immediately if broken $ sudo chmod 0666 /dev/null # Verify restoration $ ls -la /dev/null crw-rw-rw- 1 root root 1, 3 Apr 9 15:00 /dev/null ``` On Amazon EC2 instances, you can also restart the instance, because the device node is recreated from the initramfs at boot. For containerized workloads, restarting containers fixes the problem; the host's device permissions do not propagate into fresh container namespaces (they use their own device mounts). ### Trace it to an AWS CLI version ```bash # Check your current version $ aws --version aws-cli/2.x.x Python/3.x.x Linux/5.x.x ... ``` Compare the version number against the official AWS CLI GitHub releases at `github.com/aws/aws-cli/releases`. Any v2 build from April 9, 2026 forward (until the fix on April 10) is in the danger zone. AWS CLI v1 was never affected. --- ## The fix: before and after ### The broken code path (PR #10196) ```python # Pattern from #10196 era — BROKEN: chmod on any path (including devices) def _set_output_file_permissions(output_path): # No stat check — works on device files, symlinks, sockets, etc. os.chmod(output_path, 0o600) ``` That is the whole problem: the code never checks what kind of filesystem object the caller provided. ### What #10215 actually did (a revert, not a type-check patch) PR **#10215** removed the #10196 hardening entirely (aws-cli commit `12c807c`). Streaming output went back to ordinary `open(..., 'wb')` without a post-write `chmod` on the caller path. If you re-introduce output hardening in your own tools, use a regular-file guard: ```python # Recommended pattern for *future* hardening — not what #10215 shipped import os, stat def _set_output_file_permissions(output_path): file_stat = os.stat(output_path) if not stat.S_ISREG(file_stat.st_mode): return os.chmod(output_path, 0o600) ``` ### Upgrade and remediation steps ```bash # Step 1: Check your AWS CLI version $ aws --version # Step 2: Upgrade past the #10196 window (any build that includes the #10215 revert) $ pip install --upgrade awscli # Then pin the exact verified version in CI — do not float on "latest" alone # Step 3: Verify the upgrade $ aws --version # Step 4: If /dev/null was already affected, restore permissions $ sudo chmod 0666 /dev/null # Step 5: Verify restoration $ ls -la /dev/null crw-rw-rw- 1 root root 1, 3 ... ``` --- ## Lessons for DevOps and CI/CD teams ### Do not pass special files as CLI output arguments Scripts often use `/dev/null` as a convenient "discard this output" argument. When any tool, not just the AWS CLI, calls `chmod()` or `chown()` on user-supplied paths without type-checking, that habit becomes a vulnerability. Let the shell handle the redirection instead: ```bash # AFFECTED PATTERN — /dev/null passed as file argument aws lambda invoke --function-name fn /dev/null # SAFE PATTERN — shell redirection at the terminal level aws lambda invoke --function-name fn /dev/stdout > /dev/null # ALSO SAFE — temp file with cleanup TMPFILE=$(mktemp) trap "rm -f $TMPFILE" EXIT aws lambda invoke --function-name fn "$TMPFILE" ``` This pattern is immune to `chmod()` bugs because the CLI never sees `/dev/null` as a string path. ### Pin your CLI version in production pipelines This is the same principle as pinning GitHub Actions versions or Docker image tags: never install "latest" of a critical tool in production automation. ```bash # pip — pin to an exact version pip install awscli==2.15.30 # Docker — specify an exact tag FROM python:3.12-slim RUN pip install awscli==2.15.30 # GitHub Actions — pin the CLI in setup steps - name: Install AWS CLI (pinned) run: pip install awscli==2.15.30 # AWS CodeBuild — override the managed image's CLI version in buildspec.yml phases: install: commands: - pip install awscli==2.15.30 - aws --version ``` Pinning lets you choose when to update, instead of having an auto-upgrade land in the middle of a production run. ### Validate toolchain updates before rolling out Treat AWS CLI updates like dependency upgrades. When a new CLI version is released: 1. Test in staging first. Run your Lambda invocations, S3 operations, and other CLI commands in a staging CI/CD pipeline for 24 hours. 2. Use a canary runner. Update the CLI on one CI runner, monitor it for failures, then roll out to the full fleet if it is stable. 3. Check the changelog. AWS publishes release notes on the GitHub releases page. Permission changes, behavior changes, and deprecations are flagged there. This matters most for tools that touch production infrastructure. The ~24-hour window between #10196 and the #10215 revert was short, and still long enough to break many shared CI hosts. --- ## What actually shipped as the fix AWS did not land a `stat.S_ISREG()` patch in #10215. They reverted #10196 wholesale, restoring the previous open/write behavior for streaming outputs. That was the right incident response: it removed the blast radius immediately. If output hardening returns, the correct implementation is still to open with safe modes and chmod only after confirming a regular file. Never trust a caller path to be a file you created. ## When this advice fails - You never pass device paths to CLIs: still pin versions; the next hardening PR may touch credentials files differently. - Immutable distroless runners rebuilt every job: `/dev/null` damage is ephemeral; pinning still matters for reproducibility. - Windows runners: chmod semantics differ; the Linux device-node failure mode was the acute outage. ## What this post doesn't cover Every AWS CLI release after April 2026. Always read the changelog for your pinned minor. ## What to do Monday morning 1. `ls -la /dev/null` on shared CI/bastion hosts; repair if needed. 2. Pin `awscli` in every pipeline; ban floating `pip install awscli`. 3. Grep pipelines for ` /dev/null` as a positional CLI argument; switch to `> /dev/null`. 4. Canary the next CLI bump for 24h. 5. Walk the [checklist](https://www.joinhorizon.co/examples/architecture-blog-2026/cli-hardening/aws-cli-pin-and-devnull-checklist.md). 6. Need a toolchain audit? [Contact us](https://www.joinhorizon.co/contact-us/) / [DevOps pipeline services](https://www.joinhorizon.co/services/devops-pipeline-setup/). ## FAQ ### Which AWS CLI versions are affected by the /dev/null chmod bug? The regression landed in AWS CLI v2 via PR #10196 ("Tighten output file permissions"), merged around April 9, 2026. AWS reverted that change in PR #10215 on April 10, 2026. Any v2 build that includes #10196 but not the revert is in the danger window. AWS CLI v1 was not impacted. Check `aws --version` against github.com/aws/aws-cli releases and upgrade past the revert. ### Why did piping to /dev/null work but passing it as a file argument broke things? Shell redirection (`> /dev/null`) opens the device in the shell; the CLI never sees the path string. Passing `/dev/null` as an output file argument made the CLI open that path and call `os.chmod(..., 0o600)`, which succeeds on a character device and restricts it system-wide. ### What did PR #10215 actually do? #10215 fully reverted #10196. It removed the unconditional chmod/open-with-0600 hardening from streaming output paths (see commit 12c807c on aws/aws-cli). It was not a S_ISREG patch. The lasting lesson: if you re-introduce output hardening, only chmod regular files after a type check. ### How do I check if /dev/null was damaged and how do I fix it? Run `ls -la /dev/null`. Healthy: `crw-rw-rw-`. Broken: `crw-------` (or similar). Fix with `sudo chmod 0666 /dev/null`. EC2 reboot recreates the node from initramfs; containers usually need a restart for a clean namespace. ### How do I prevent this class of breakage in my CI/CD pipelines going forward? Use shell redirection instead of `/dev/null` as a CLI file argument; pin `awscli==X.Y.Z` in CI; canary CLI upgrades 24h before production runners. See the checklist artifact linked in the post. --- *Source: https://www.joinhorizon.co/blog/aws-cli-chmod-dev-null-streaming-bug-2026/*