Skip to content

fix(scripts): reject whitespace-only plans - #5044

Open
Uddoo wants to merge 2 commits into
github:mainfrom
Uddoo:fix/5035-reject-whitespace-plans
Open

Uddoo wants to merge 2 commits into
github:mainfrom
Uddoo:fix/5035-reject-whitespace-plans

Conversation

@Uddoo

@Uddoo Uddoo commented Oct 21, 2026 •

Copy link
Copy Markdown

Description

Whitespace-only plan templates currently produce unusable plans, and a whitespace-only plan.md passes prerequisite checks. Treat that content as empty in the Bash, PowerShell and Python scripts, using the existing diagnostics. Setup still preserves an existing plan; the prerequisite check asks the contributor to populate it or remove it and rerun planning.

Use Unicode White_Space consistently for UTF-8 content, independent of the caller's locale. Ignore one leading UTF-8 BOM only during validation, and tolerate decoding errors when Python checks existing non-UTF-8 plans. The two shared predicates add no runtime dependency or content conversion.

Extend the existing parity tests for blank templates and overrides, retrying with a valid template, existing-plan preservation, Unicode whitespace, BOMs and non-empty UTF-16/cp1252 plans. Valid content retains each port's existing write semantics.

Fixes #5035.

Testing

  • Before the fix, the focused regressions reported 20 failed, 28 passed, 24 skipped on both Windows and Linux. The Windows run used native Windows PowerShell 5.1.26100.9549 and Python 3.11.9; Linux used Bash and Python 3.12.3.
  • Review regressions against the published 3f6daf1 production bytes: 18 failed, 114 passed, 66 skipped on native Windows PowerShell 5.1/Python; 36 failed, 96 passed, 66 skipped on Linux Bash/Python. Python cases overlap across the two runs.
  • Current complete setup-plan/prerequisite parity modules: 192 passed, 117 skipped on native Windows PowerShell 5.1/Python; 210 passed, 99 skipped on Linux Bash/Python, including Bash under LC_ALL=C.
  • Full native Linux suite: .venv/bin/python -m pytest tests -q -- 12,608 passed, 464 skipped, 62 warnings, 0 failed.
  • Ruff 0.15.0 (check src tests scripts/python), all four repository pre-commit checks under Linux, ShellCheck 0.11.0 at the CI's --severity=error, and git diff --check passed. No dependency or security-snapshot files changed.
  • Built a wheel and initialized isolated Codex sample projects from it. All eight packaged setup/prerequisite/common scripts match source byte-for-byte. The current three-port harness completed 48 expected command checks per port (144 total, including initialization) covering ASCII/NBSP/BOM blank-template rejection, a valid retry, blank and non-empty UTF-16/cp1252/BOM plans, downstream prerequisite flags, and paths-only lookup.

Windows full-suite validation is not claimed. The run was interrupted after 19 adapter tests failed because this host cannot create symlinks (WinError 1314); all 19 failures were reproduced on unchanged base 505781d7. An initial long temporary path also caused WinError 206, so later Windows checks used a short task-owned temporary directory. This review round's Windows pre-commit environment setup also hit WinError 206; the same four configured hooks passed under isolated Linux. No system setting was changed, and no assertions were relaxed to accommodate these environment failures.

An earlier WSL run inherited Windows executable paths and found 28 PowerShell interop failures; all 28 reproduced on the same unchanged base. The final full Linux run used a native Linux PATH. PowerShell validation ran natively on Windows 5.1, rather than through WSL interop. Skipped tests are not counted as passes.

Test selection reasoning

Changed files Affects Tests Reason
scripts/bash/setup-plan.sh, scripts/powershell/setup-plan.ps1, scripts/python/setup_plan.py $speckit-plan T1-T3 Reject blank templates before creating a plan; retain normal setup and planning.
The three check-prerequisites ports $speckit-analyze, $speckit-implement, $speckit-converge, $speckit-checklist, $speckit-taskstoissues T4-T9 Check an existing plan before processing downstream artifacts.
The --paths-only / -PathsOnly path $speckit-clarify T2 Verify that resolution remains available without validating plan content.

Manual test results

Agent: OpenAI Codex, autonomous, in the current session. OS/Shell: Windows / Windows PowerShell 5.1 for the generated-skill walkthrough; the separate script-entry matrix also exercised Python and Linux Bash.

The original submission at 3f6daf1 followed the generated skills in an isolated documentation-only sample initialized from its wheel; the table below records that original walkthrough. In the current review round, the agent re-read the generated plan/tasks skills and rebuilt the wheel, then reran the three-port script-entry matrix described above. These are agent-driven checks, not human or independent validation. No separate coding-agent process or paid model call was launched.

Test / command Original walkthrough result
T1: $speckit-specify Authored the local-note specification and evaluated its built-in quality checklist: 16/16 criteria satisfied by the same agent.
T2: $speckit-clarify Paths-only lookup succeeded; the bounded sample needed no clarification questions.
T3: $speckit-plan A blank template was rejected without creating a plan. Retrying with the valid template succeeded; the agent produced the plan, research, data model and quickstart.
T4: $speckit-tasks The real setup-tasks script resolved its template; the agent generated four dependency-ordered tasks with phases, paths and requirement references.
T5: $speckit-analyze Required-artifact lookup succeeded; the agent found no blocking inconsistency across the sample specification, plan and tasks.
T6: $speckit-implement The agent completed the four documentation-only tasks and checked the resulting note against its specification. The built-in checklist remained complete.
T7: $speckit-converge No remaining sample work was found; zero tasks appended and tasks.md remained byte-for-byte identical.
T8: $speckit-checklist Template resolution succeeded; six traceable requirements-quality questions were generated and left unchecked for their reviewer.
T9: $speckit-taskstoissues The prerequisite gate succeeded, then the documented non-GitHub-remote guard stopped conversion. No GitHub issue read or write was attempted.

All five downstream command flag combinations reject a whitespace-only plan and accept the valid plan in all three script ports. The deprecated tasks-to-issues command's real GitHub publication path was not exercised. The sample had no extension hooks.

AI Disclosure

  • I did not use AI assistance for this contribution
  • I did use AI assistance (described below)

AI disclosure: OpenAI Codex acted autonomously on behalf of @Uddoo for investigation, implementation, regression tests, local validation, the generated-skill sample walkthrough, self-review, review remediation and submission. The runtime reports routing group group/smart-split with reasoning effort xhigh; its underlying model ID is not exposed, so the model is reported as unknown. No human line-by-line review or independent review is claimed. Each commit carries its own Assisted-by: disclosure.

Treat whitespace-only plan templates and existing plans as empty in the
Bash, PowerShell and Python scripts, using the existing diagnostics.
Extend the parity regressions for blank content, overrides, valid retries
and preservation of existing plans and padded template content.

Assisted-by: OpenAI Codex (model: unknown; routing: group/smart-split; effort: xhigh; autonomous)
@Uddoo
Uddoo requested a review from mnriem as a code owner October 21, 2026 02:20
@mnriem mnriem added the triage-must-have Verdict: high-value, important work for Spec Kit — do first label Oct 21, 2026
@mnriem
mnriem requested a max review from Copilot October 21, 2026 13:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved blank-content validation gaps and a Python decoding regression affect setup and downstream prerequisite checks.

3 open issues
What changed in this PR

Addresses #5035 by treating whitespace-only templates and plans as empty across Specify’s Bash, PowerShell, and Python scripts.

Changes:

  • Add whitespace-aware setup and prerequisite validation.
  • Extend parity tests for blank overrides, retries, existing-plan preservation, and valid padded content.

AI-generated review for the requester: GitHub Copilot (model unknown, automated mode); source inspection and isolated checks.

File Description
tests/​test_setup_plan_python_parity.py Cover blank templates, retries, and content preservation.
tests/​test_check_prerequisites_python_parity.py Cover blank plans and padded valid content.
scripts/​python/​setup_plan.py Reject whitespace-only templates.
scripts/​python/​check_prerequisites.py Check plan content instead of size.
scripts/​powershell/​setup-plan.ps1 Reject whitespace-only templates.
scripts/​powershell/​check-prerequisites.ps1 Validate plan content.
scripts/​bash/​setup-plan.sh Require non-whitespace template content.
scripts/​bash/​check-prerequisites.sh Require non-whitespace plan content.

Review effort: Max

Comment thread scripts/bash/setup-plan.sh Outdated
trap 'rm -f "$template_file"' EXIT
if resolve_template_content "plan-template" "$REPO_ROOT" > "$template_file"; then
if [[ ! -s "$template_file" ]]; then
if ! grep -q '[^[:space:]]' "$template_file"; then
Comment thread scripts/python/check_prerequisites.py Outdated
Comment thread scripts/python/setup_plan.py Outdated
return 1
if template_content is not None:
if template_content == "":
if not template_content.strip():
@mnriem

mnriem commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Please address Copilot feedback. If not applicable, please explain why

Use Unicode White_Space consistently across Bash, Python and PowerShell,
independent of the caller's locale. Ignore one leading UTF-8 BOM only
while checking content, and keep Python prerequisite checks tolerant of
non-UTF-8 plans that the previous size check accepted.

Extend parity regressions for Unicode blanks, BOM-prefixed templates and
plans, non-empty UTF-16/cp1252 plans, and content preservation.

Assisted-by: OpenAI Codex (model: unknown; routing: group/smart-split; effort: xhigh; autonomous)
@Uddoo

Uddoo commented Oct 21, 2026

Copy link
Copy Markdown
Author

Addressed all three findings in d35ae94.

Bash and Python now use the same Unicode White_Space set as native PowerShell,
including under LC_ALL=C. Validation ignores exactly one file-leading UTF-8
BOM, and Python's prerequisite read tolerates decoding errors in existing
non-empty UTF-16/cp1252 plans. The checks do not rewrite existing plans; setup
keeps each port's existing BOM/write semantics.

The new regressions fail against the previous published production bytes:
36 failures on Linux and 18 on native Windows (overlapping Python cases).
Current complete parity modules pass with 210 passed/99 skipped on Linux and
192 passed/117 skipped on native Windows PowerShell 5.1/Python. The full native
Linux suite reports 12,608 passed, 464 skipped, 62 warnings, zero failures.
Ruff, all four configured pre-commit checks under Linux, error-level ShellCheck
and diff checks pass. Eight packaged scripts match source, and fresh wheel
samples complete 48 expected script-entry checks per port across Bash, Python
and native PowerShell 5.1.

The PR body records the current evidence and retains the Windows validation
limits. No full Windows-suite pass is claimed. Conversations remain with their
reviewers.

AI-generated author response on behalf of @Uddoo: OpenAI Codex, model unknown
(verified runtime route group/smart-split, reasoning effort xhigh),
autonomous mode. The agent performed reproduction, code/test changes,
validation, its own code review, publication and this response. No human
line-by-line review or independent review is claimed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unreadable nonempty plans are reported as empty, with removal advice that could cause existing work to be deleted.

2 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Report unreadable plans instead of treating them as empty

scripts/​bash/​check-prerequisites.sh:151

A nonempty plan.md without read permission reaches this empty-file branch: sed fails, and grep receives no content and returns 1. I reproduced this with a chmod 000 plan under an unprivileged account. The resulting advice to remove the "empty" file could discard a real plan. Check readability before this validation and report a permission error instead; add a regression covering an unreadable nonempty plan.

AI disclosure: GitHub Copilot (model unknown, autonomous) prepared this review and targeted verification for the requester.

1 resolved since last review

Review effort: Max

@mnriem

mnriem commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Please address Copilot feedback, please include the previously missed one too. If not applicable, please explain why

@mnriem mnriem added the author-awaiting Waiting on author response label Oct 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author-awaiting Waiting on author response triage-must-have Verdict: high-value, important work for Spec Kit — do first

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Plan setup and prerequisite checks accept a whitespace-only plan template or plan.md

3 participants