How well each of the 69 activities in NABS Confidence 0.14.9 is tested, and how confident we are in releasing it. The levels come from a written rule, below, not from a feeling.
High 0 Medium 51 Low 18
Release readiness
Release 0.13.0 (adds Release Confidence to the 0.11.0 code under the NABS Confidence name) to the team as a preview. Do not call it fully verified.
The everyday flow is well tested by machine: create a retro, gather feedback from several people, edit and lock it, choose objectives, write action items and read insights from Git. Each of these has unit tests and integration tests, and most were also run by hand on 0.7.1 with real Git, GitHub and Copilot.
Nothing earns High yet, because nobody has run the manual script on the version being released. The newer features (the dashboard, objective focus, archiving, Home) have never been run by hand. The automated tests (unit, and integration in a real VS Code under a virtual display) do now run in GitHub Actions on every push to the main branch, so a broken build shows up there.
The weakest areas are the parts that talk to the outside world. Archiving to a GitHub Discussion has only been tested against a fake service. The Copilot call (model, consent, answer) and the @retro chat participant have never been run by an automated test. The real GitHub client has no test of its own.
Before the next release
Check that the latest CI run on the main branch is green (compile, lint, unit tests, coverage check, boundary check, integration tests), or run npm test yourself.
Run sections 13 to 17 of the real-world script on the release version, in particular section 15 (archive to a real Discussion in a test repository).
Run sections 11 and 12 (offline and no-Copilot, panels keeping input) that were left over from 0.7.x.
Re-run sections 5 to 10 on the release version, because the Copilot prompts changed in 0.9.0.
Then add a run record to catalogue/runs (for example manual-0.12.0.yml) with the version, date and sections run, update this file, re-render with python scripts/user-guide/render.py, and run npm run check:coverage. Many Medium activities will then become High.
The main gaps
Archive to a Discussion (post, resume, remove files): fake GitHub only, never run by hand.
Copilot summaries, objective assessment and @retro: the Copilot call has no automated test and was last run by hand on 0.7.1, before the prompts changed.
The real GitHub client (insights, issues, action items, templates, objectives): fake GitHub only in the automated tests.
Real GitHub and Copilot are never used by the automated tests, in CI or on a developer machine; only the manual script does that.
Commands with no test of their own: Open Action Item, Show Log, Customise Template, Refresh Feedback Templates (each is covered only indirectly, if at all).
Not covered at all: multi-root workspaces, non-Git folders, very large repositories, Windows paths with unusual characters, and remote or web VS Code.
How confidence is decided
Each activity gets one level. The levels mean:
Level
Meaning
High
Safe to release without further checking. The activity has unit tests and integration tests, and a manual run of the real-world script has passed on the version being released. Nothing it needs is known to be untested.
Medium
Likely to work. The activity has automated tests, and either both kinds (unit and integration) or a manual run on some version. Look at the gaps before relying on it.
Low
Not verified. The activity has no automated test, or depends on GitHub or Copilot and has never been run by hand, or has only one kind of automated test and no manual run. Treat it as unproven.
The rule, in order:
No automated test at all: Low.
Needs GitHub or Copilot and has no manual run: Low.
Only unit tests, or only integration tests, and no manual run: Low.
Unit tests and integration tests and a manual run on the version being released: High.
Everything else: Medium.
A level may be set lower than the rule gives, with a written reason. It may never be set higher.
Screenshot runs are shown as supporting evidence only. They do not change a level.
What counts as evidence
Unit tests check small pieces of logic (parsing, writing files, building prompts, deciding what to show) in plain Node, with no VS Code.
Integration tests run inside a real VS Code instance with a real Git repository, and drive the extension's commands, forms and panels. They use fakes for GitHub (issues, Discussions, shared templates and objective issues), and they switch Copilot off. So they show that the extension does the right thing with the answers it is given, not that GitHub or Copilot gives them.
The manual script is a checklist run by a person against a real test repository, with real GitHub and real Copilot. It was last run on 0.7.1 (2026-10-03). The maintainer ran sections 0 to 10 by hand on 0.7.1 against a real test repository (real Git, real GitHub, real Copilot) and reported no failing steps. The tick boxes in the tracking issue were not all set, so this is the maintainer's statement, not a signed-off checklist. Sections 11 to 17 (offline and no-Copilot, panels keeping input, the Retro Dashboard, objective focus, archiving to a Discussion, Home, Report a Problem) have not been run by hand.
Screenshot runs show a feature working in a real VS Code window on a fictional repository. They help a reader see what is there, but they do not change a level.
A level can only be as high as its evidence allows. Where a person judged it should be lower than the rule gives, the page says why. These levels describe the tests that exist. The automated tests run on every push to the main branch in GitHub Actions, and on a developer machine with npm test.
The manual script
The script has 20 numbered sections. Section 0 is preparation.
Section
Topic
Run by hand
0
Before you start
Yes, on 0.7.1
1
Setup
Yes, on 0.7.1
2
Objectives
Yes, on 0.7.1
3
New Retro
Yes, on 0.7.1
4
Feedback from two people
Yes, on 0.7.1
5
Refresh Insights (real Git and GitHub data)
Yes, on 0.7.1
6
Copilot: summarise and assess
Yes, on 0.7.1
7
Team verdict with an override
Yes, on 0.7.1
8
Action items to real GitHub issues (test repository only)
Yes, on 0.7.1
9
Update Action Items after closing one
Yes, on 0.7.1
10
The @retro chat participant
Yes, on 0.7.1
11
Offline and no-Copilot fallbacks
Not yet
12
Panels keep unsaved input (#37)
Not yet
13
The Retro Dashboard (0.8.0, #41)
Not yet
14
A retro focuses on its chosen objectives (0.9.0, #53)
Not yet
15
Archive a retro to a GitHub Discussion (0.10.0, #57)
Not yet
16
Home, the landing page (0.11.0, #61)
Not yet
17
Report a Problem and wrap-up
Not yet
18
Release Confidence (0.13.0, #91, #93) and the Release Report (0.14.0, #94, #95)
Not yet
19
The GitHub client against the real GitHub (0.14.9, #112, #113, #114)
Not yet
Screenshot runs
Home (0.11.0). Screenshots of Home in light and dark themes, with and without objectives and retros, taken by the screenshot harness on a fictional repository.
Retro Dashboard (0.8.0, Gather again at 0.11.0). Screenshots of every dashboard stage as facilitator and participant, on a fictional repository.
Gather (0.11.0). A scripted run of the Gather stage, adding feedback as several people and marking them done, with 56 screenshots.
Objectives (0.9.0 to 0.11.0). Screenshots of the Objective form, focus and assessment panels.
Archive (0.11.0). A scripted run of the Archive panel against a fake Discussions service, with 50 screenshots.
The matrix
One table per group, then the detail for each activity: the exact tests, the manual steps and what is not covered.
Manual script steps 5.1, 5.2: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.
Tests: 12 unit, 2 integration
Unit tests
src/test/unit/insights.test.ts
parses commits, numstat lines, binary files and renames
normalises rename notation
summarises authors, churn and hot files
handles an empty log
src/test/unit/gitHistory.test.ts
a very large history: thousands of commits and a vendor import of tens of thousands of files are read and summarised, only the period counts
output over the buffer is a plain sentence about the period, not Node's buffer message
a shallow clone says so and leaves out its oldest commit, which Git shows as adding every file
merges: merge commits are not counted, the commits they bring in are, once
unusual renames and names: moved to the root, into a new folder, spaces, non-ASCII, and a tab; each file is counted under its new path
git's quoted paths and rename notation are read back to the real path
src/test/unit/document.test.ts
reads and replaces only the block content
returns undefined when markers are missing
Integration tests
src/test/suite/extension.test.ts
creates a first retro via the New Retro command (form values as argument), without AI
every command opens its form or picker, whatever VS Code passes from its menus (#22)
Add GitHub issues, pull requests and discussions to the Insights Medium
What it does · Impact if it breaks: medium · Needs GitHub
Not covered: The real GitHub client (sign-in, REST and GraphQL calls, paging, rate limits, private repositories) has no automated test, not even against a fake. Only the pure helpers are tested. It was checked by hand on 0.7.1.
Manual script steps 5.1, 5.2: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.
Tests: 4 unit, 0 integration
Unit tests
src/test/unit/insights.test.ts
parses common remote URL formats
builds search queries scoped to the repo and period
classifies issues, PRs and discussions by what happened in the period
Not covered: The modal that protects hand-edited sections is not exercised by an automated test.
Manual script steps 11.1: not run by hand yet.
Tests: 2 unit, 2 integration
Unit tests
src/test/unit/feedback.test.ts
renders grouped lists and counts by configured type
src/test/unit/release040.test.ts
summaries and prompts include the trend
Integration tests
src/test/suite/extension.test.ts
feedback can be added and summarised into the retro
every command opens its form or picker, whatever VS Code passes from its menus (#22)
Summarise feedback with Copilot Low
What it does · Impact if it breaks: medium · Needs Copilot
Why this level: The Copilot call itself (model choice, consent, the request and the streamed answer) is never run by an automated test: the integration tests switch Copilot off. Only the prompts and the parsing of answers are unit tested. The last real run was on 0.7.1, before the prompts changed in 0.9.0.
Manual script steps 6.1, 14.7: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.
Tests: 5 unit, 0 integration
Unit tests
src/test/unit/insights.test.ts
feedback prompt includes the feedback and the markers
parses a well-formed model answer and rejects a malformed one
src/test/unit/release090.test.ts
the summary prompts focus on the chosen objectives, with an "Outside focus" line
without objectives the prompts are as before
src/test/unit/release040.test.ts
summaries and prompts include the trend
Integration tests: none.
Keep working without Copilot or offline Low
What it does · Impact if it breaks: medium · Needs Copilot
Why this level: Every integration test runs with Copilot off, so the no-Copilot path is well covered, but it was never run by hand on a current version (the offline and no-Copilot section was still to do after the 0.7.1 run) and the Copilot-on path is untested.
Not covered: Offline behaviour for GitHub calls is only covered for template caching; the manual offline section has not been run.
Manual script steps 11.1, 11.2, 11.3: not run by hand yet.
Tests: 0 unit, 2 integration
Unit tests: none.
Integration tests
src/test/suite/extension.test.ts
feedback can be added and summarised into the retro
feedback templates from the repo and a shared repository (fake GitHub), with caching and offline fallback (#14)
See release confidence in the retro's Insights Medium
Not covered: Only reports somebody saved and committed are read (it is not live). The section is not part of the Copilot summary prompt in the chat participant.
No step in the manual script covers this.
Tests: 3 unit, 1 integration
Unit tests
src/test/unit/releaseActivityLinks.test.ts
"Release confidence since the last retro": start and end, the trend and the activities that moved, falls first
the section says so when the period has no report, one report, or old reports without levels
Retros adds the sections other modules provide to the Insights, and a section that fails is only left out
Integration tests
src/test/suite/extension.test.ts
0.14.4: an activity's level is an objective measure, the retro's Insights show release confidence since the last retro, and #act:<id> shows on the dashboard (#107)
Manual script steps 2.1, 2.2: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.
Screenshot runs: Objectives (0.9.0 to 0.11.0).
Tests: 13 unit, 2 integration
Unit tests
src/test/unit/objectiveForm.test.ts
validates the fields and criteria inline
a new objective can't reuse another objective's id
new criteria get the next free id; existing ids are kept
a new objective file round-trips through the parser
editing keeps other front matter keys and sections, and empties optional fields
the file New Objective used to write still parses after editing
a verdict must be chosen
an override of the AI's verdict needs a reason
agreeing with the AI, or no AI assessment, needs no reason
evidence must not be empty
src/test/unit/objectives.test.ts
parses an objective file: front matter, why, criteria with ids, done and automatic measures
reports problems instead of failing: bad ids, missing titles, unknown metrics, bad dates
slugs and new objective files
Integration tests
src/test/suite/extension.test.ts
the Objective form (#34) creates and edits an objective, with its acceptance criteria
every command opens its form or picker, whatever VS Code passes from its menus (#22)
Use GitHub issues as objectives Medium
What it does · Impact if it breaks: medium · Needs GitHub
Not covered: Issues are read through a fake in the integration test and a fake GitHub (paging, pull requests left out) in the unit tests; the real read was done by hand on 0.7.1 only.
Manual script steps 2.3: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.
Tests: 3 unit, 1 integration
Unit tests
src/test/unit/objectives.test.ts
parses an objective issue: header lines, ids for this and other repositories, state
merges sources in order (the first wins) and validates nabsConfidence.objectiveSources
src/test/unit/githubClient.test.ts
objective issues: labelled issues are listed over pages without pull requests, with their state and assignees
Integration tests
src/test/suite/extension.test.ts
objectives (#20): files and GitHub issues (fake), the chosen objectives in each new retro (0.9.0), linking, evidence, assessment and verdicts
Not covered: An integration test walks section 14's steps 14.1-14.5 and 14.9 through the real New Retro form, dashboard and Retro panel code, but nobody has run section 14 by hand in VS Code, so what the pages do in the browser (ticking boxes, the confirmation dialogs) and the Copilot steps 14.6-14.8 are unchecked.
Manual script steps 14.1, 14.2, 14.3, 14.4, 14.5, 14.9: not run by hand yet.
Screenshot runs: Retro Dashboard (0.8.0, Gather again at 0.11.0).
Tests: 9 unit, 3 integration
Unit tests
src/test/unit/release090.test.ts
New Retro: the open objectives are ticked by default; achieved and dropped ones are listed unticked
an existing retro: its own objectives are ticked, ones no longer in the sources are kept
readSelection and cleanIds read ticked boxes and well-formed ids only
changeSelection keeps existing regions verbatim, adds new ones and reports removals
removing an objective with evidence, an assessment or a verdict needs confirmation
New Retro form values: objectives are optional and cleaned; zero ticked gives a soft warning
Dashboard: Prepare asks for at least one objective (a soft gate)
src/test/unit/objectives.test.ts
the layout: the section is dropped when nothing is open, and added to older retros
regions: set, missing markers, meta escaping
Integration tests
src/test/suite/extension.test.ts
objectives (#20): files and GitHub issues (fake), the chosen objectives in each new retro (0.9.0), linking, evidence, assessment and verdicts
every command opens its form or picker, whatever VS Code passes from its menus (#22)
manual script section 14 (0.9.0, #53), steps 14.1 to 14.5 and 14.9 in order, as a person does them: New Retro, the dashboard, the Retro panel (#111)
Link feedback and action items to objectives Medium
Manual script steps 6.3, 12.5: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.
Screenshot runs: Objectives (0.9.0 to 0.11.0).
Tests: 3 unit, 2 integration
Unit tests
src/test/unit/objectiveForm.test.ts
evidence must not be empty
src/test/unit/objectives.test.ts
computes automatic measures from the numbers we already collect
renders the snapshot and reads it back (criteria, measures, evidence, links)
Integration tests
src/test/suite/extension.test.ts
objectives (#20): files and GitHub issues (fake), the chosen objectives in each new retro (0.9.0), linking, evidence, assessment and verdicts
every command opens its form or picker, whatever VS Code passes from its menus (#22)
Assess objectives with Copilot Low
What it does · Impact if it breaks: medium · Needs Copilot
Why this level: The Copilot call itself (model choice, consent, the request and the streamed answer) is never run by an automated test: the integration tests switch Copilot off. Only the prompts and the parsing of answers are unit tested. The last real run was on 0.7.1, before the prompts changed in 0.9.0.
Manual script steps 6.2: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.
Tests: 2 unit, 2 integration
Unit tests
src/test/unit/objectives.test.ts
the assessment prompt and its answer
records the AI assessment and the team verdict, with the override and its reason
Integration tests
src/test/suite/extension.test.ts
every command opens its form or picker, whatever VS Code passes from its menus (#22)
objectives (#20): files and GitHub issues (fake), the chosen objectives in each new retro (0.9.0), linking, evidence, assessment and verdicts
Not covered: Only reports somebody saved and committed are read (it is not live). A report saved before 0.14.4 has no levels in its front matter; they are read from its table. The activity measures are one per activity, so the objective form lists them all (by title). Nobody has run steps 18.12 and 18.13.
Manual script steps 18.12, 18.13: not run by hand yet.
Tests: 13 unit, 2 integration
Unit tests
src/test/unit/releaseRetroLink.test.ts
the newest score in the period, in plain words with the release and commit
the trend compares the newest score with the report before it
releases in a row at 80 or more: the evidence for "80 or more for three releases"
the Release module provides the measures through the core contract, reading the repository root it is given
a measure with a dot in its id is accepted in a criterion; other names still are not
the retro line for a release measure names the provider, the value and the period; one nobody provides says so
the snapshot and the refresh fill in the release measures; the measures Retros has are unchanged
Retros reads the providers it is given: every id gets a reading, a failing provider is only unavailable
the objective form accepts a release measure and the file keeps it as (auto: <id>)
src/test/unit/releaseActivityLinks.test.ts
a saved report lists every activity with its score and level, and it reads back
a report saved before 0.14.4 has the levels in its table only, and they are read from there
release.activity.<id>.level: the level (1 Low to 3 High) in the newest report of the period, with how it moved
the activities of the catalogue are on offer, one measure each, and the family answers any of them
Integration tests
src/test/suite/extension.test.ts
0.14.1: an objective can use release confidence as its automatic measure, read from the saved release reports (#95)
0.14.4: an activity's level is an objective measure, the retro's Insights show release confidence since the last retro, and #act:<id> shows on the dashboard (#107)
Manual script steps 3.1, 8.1: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.
Tests: 4 unit, 1 integration
Unit tests
src/test/unit/document.test.ts
parses open and done task list items
renders a follow-up section
src/test/unit/release040.test.ts
parses mentions, due dates and issue links; ignores the placeholder
the follow-up section shows the follow-through rate
Integration tests
src/test/suite/extension.test.ts
a second retro covers the gap since the first and carries its action items forward
Create GitHub issues for action items Medium
What it does · Impact if it breaks: high · Needs GitHub
Not covered: The real client's requests are only tested against a fake GitHub written from GitHub's documentation (labels, assignees, the rate limit, 401/404/410 and outage failures, resuming); the fake can be wrong about what GitHub does. Real issue creation was checked by hand on 0.7.1 only, and `npm run verify:github` (section 19) has not been run.
Manual script steps 8.1, 8.2, 19.1, 19.2, 19.4: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.
Tests: 11 unit, 4 integration
Unit tests
src/test/unit/release040.test.ts
issue titles and bodies are clean
links and ticks are written into the right line, keeping the rest
src/test/unit/githubClient.test.ts
issues: the label is checked then created, the issue request has the documented shape, and the answer is read from its real shape
issues: an assignee who cannot be assigned gives an unassigned issue, a label made meanwhile is fine, and other refusals say what GitHub said
issues: the failures GitHub really answers with: no access, issues turned off, a revoked sign-in, the rate limit, an outage and a dropped connection
not signed in: no request is made, and the message says to run the command again and sign in
action item states: up to 50 issues per request, closed and open read, an issue that does not exist is left out
explainFailure leaves other messages alone and only mentions the rate limit for the codes GitHub uses for it
it reports a failure when GitHub disagrees, and skips the Discussion checks when there is no category
read-only: it reads, never writes, and skips the writing checks
with --write: it creates a label, two issues (closed again) and a Discussion with comments, and every check passes
Integration tests
src/test/suite/extension.test.ts
action items become GitHub issues, are linked in the retro, and tick themselves off when closed
the Retro panel (#33): details, feedback done, and the Create GitHub Issues confirmation
every command opens its form or picker, whatever VS Code passes from its menus (#22)
action items become real GitHub requests (a fake GitHub that checks the documented rules): labels, assignees that cannot be assigned, the rate limit midway, resuming, then the ticking off (#112)
Update action items from GitHub Medium
What it does · Impact if it breaks: medium · Needs GitHub
Not covered: The state of real issues is read through a fake in the automated test.
Manual script steps 9.1: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.
Tests: 1 unit, 2 integration
Unit tests
src/test/unit/release040.test.ts
links and ticks are written into the right line, keeping the rest
Integration tests
src/test/suite/extension.test.ts
action items become GitHub issues, are linked in the retro, and tick themselves off when closed
every command opens its form or picker, whatever VS Code passes from its menus (#22)
Open an action item Low
What it does · Impact if it breaks: low · Needs GitHub
Not covered: No automated test and no manual step covers this command.
No step in the manual script covers this.
Tests: 0 unit, 0 integration
Unit tests: none.
Integration tests: none.
Tag feedback and action items with a release activity Medium
Not covered: The ids are not checked against the catalogue when you type them (the table says when there is no level for one). The tag is not offered in the Add Feedback form.
No step in the manual script covers this.
Tests: 1 unit, 1 integration
Unit tests
src/test/unit/releaseActivityLinks.test.ts
#act:<id> links a feedback entry or an action item to an activity; the issue gets an activity:<id> label and a clean title
Integration tests
src/test/suite/extension.test.ts
0.14.4: an activity's level is an objective measure, the retro's Insights show release confidence since the last retro, and #act:<id> shows on the dashboard (#107)
Close and archive
Check where the archive will go Low
What it does · Impact if it breaks: medium · Needs GitHub
Not covered: GitHub is a fake in the automated test; the real checks (Discussions turned off, a missing category) have never been run. Section 15 of the manual script has never been run by hand.
Manual script steps 15.1, 15.2, 15.3: not run by hand yet.
Tests: 0 unit, 1 integration
Unit tests: none.
Integration tests
src/test/suite/extension.test.ts
0.10.0: archive a retro to a GitHub Discussion, resume without duplicates, then remove its files safely (#57-#60)
Post the retro to a GitHub Discussion Low
What it does · Impact if it breaks: high · Needs GitHub
Not covered: Posting is tested through the real client against a fake GitHub written from GitHub's documentation (the rate limit, a revoked sign-in, paging, GitHub's 65,536-character limit, resuming, finding the Discussion by its marker), not against GitHub: the fake can be wrong about what GitHub does (GitHub's real refusal messages, how soon a new Discussion can be found by search, permission errors on organisation repositories). `npm run verify:github` and manual steps 15.5-15.7, 15.13 and 19.3 have not been run.
Manual script steps 15.4, 15.5, 15.6, 15.7, 15.13, 19.1, 19.3, 19.4: not run by hand yet.
Screenshot runs: Archive (0.11.0).
Tests: 17 unit, 2 integration
Unit tests
src/test/unit/release0100.test.ts
the parts follow the stages: body, one comment per person, insights, objectives, decide, close
the body says which objectives it addresses, the details, and who is done with counts
a retro with no objectives says so
each person's comment groups their feedback by type, with the objective
insights, objectives (assessment, verdict, override reason), decide and close
long parts are split under the limit, by paragraph, each with its own marker and a "continued" line
splitMarkdown: paragraphs, then lines, then a hard cut; nothing is lost
markers find what is already posted, for this retro only
pendingParts skips the posted ones, in order
progress lives in flat front matter keys
helpers: title, person keys, table cells, placeholders and the upwards section
the Archive record comment links the last commit and says how to restore
src/test/unit/githubClient.test.ts
Discussions: the settings are read, and a repository without Discussions or without the category is told apart
Discussions: creating one sends exactly what GitHub documents, and refusals (no category, too long, Discussions off) are explained
Discussions: comments are added and read back over pages, a part is found by its marker over pages, and a deleted Discussion is undefined
Discussions: the rate limit (as GraphQL reports it, with HTTP 200), a revoked sign-in and an outage stop the call with a message that says what to do
with --write: it creates a label, two issues (closed again) and a Discussion with comments, and every check passes
Integration tests
src/test/suite/extension.test.ts
archive and remove through the real GitHub client (a fake GitHub that checks the documented rules): the rate limit midway, resuming, a 70,000-character entry, a failed record comment, a failed git rm, a detached HEAD, and the restore command (#113, #114)
0.10.0: archive a retro to a GitHub Discussion, resume without duplicates, then remove its files safely (#57-#60)
Remove the archived files safely Low
What it does · Impact if it breaks: high · Needs GitHub
Not covered: Git is real in the integration tests (a local bare repository: not committed, no upstream, local changes, not pushed, a detached HEAD, a locked index, running the restore command), but the Discussion is a fake GitHub written from GitHub's documentation, and nobody has run the removal by hand. The record comment failing, git rm failing after it, and retrying are tested; `npm run verify:github` and manual steps 15.8, 15.9, 15.11 and 15.12 have not been run.
Manual script steps 15.8, 15.9, 15.11, 15.12, 19.3: not run by hand yet.
Screenshot runs: Archive (0.11.0).
Tests: 1 unit, 2 integration
Unit tests
src/test/unit/release0100.test.ts
the Archive record comment links the last commit and says how to restore
Integration tests
src/test/suite/extension.test.ts
archive and remove through the real GitHub client (a fake GitHub that checks the documented rules): the rate limit midway, resuming, a 70,000-character entry, a failed record comment, a failed git rm, a detached HEAD, and the restore command (#113, #114)
0.10.0: archive a retro to a GitHub Discussion, resume without duplicates, then remove its files safely (#57-#60)
Find archived retros Low
What it does · Impact if it breaks: medium · Needs GitHub
Not covered: Not run by hand.
Manual script steps 15.10: not run by hand yet.
Screenshot runs: Archive (0.11.0).
Tests: 3 unit, 1 integration
Unit tests
src/test/unit/release0100.test.ts
round-trips, newest first, one record per date
bad JSON or bad entries are reported, not thrown
@retro learns which retros were archived, and where
Integration tests
src/test/suite/extension.test.ts
0.10.0: archive a retro to a GitHub Discussion, resume without duplicates, then remove its files safely (#57-#60)
@retro chat and language model tools
Ask @retro in Copilot Chat Low
What it does · Impact if it breaks: medium · Needs Copilot
Why this level: No automated test registers or calls the chat participant, and no test reaches the tool loop; only some of the prompt text is unit tested. The Copilot call itself (model choice, consent, the request and the streamed answer) is never run by an automated test: the integration tests switch Copilot off. Only the prompts and the parsing of answers are unit tested. The last real run was on 0.7.1, before the prompts changed in 0.9.0.
Not covered: The participant, its three commands and the tool loop have no automated test. /objectives (0.9.0) has never been run by hand.
Manual script steps 10.1, 10.2, 14.8: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.
Tests: 2 unit, 0 integration
Unit tests
src/test/unit/release090.test.ts
@retro /objectives: the prompt has only the chosen objectives, their evidence and verdicts
src/test/unit/release0100.test.ts
@retro learns which retros were archived, and where
Integration tests: none.
Let an agent read retros (#retros tools) Medium
What it does · Impact if it breaks: low · Needs Copilot
Not covered: The tools are called directly by the test, not by Copilot, so the way Copilot chooses and calls them is not tested.
Manual script steps 10.3: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.
Tests: 0 unit, 2 integration
Unit tests: none.
Integration tests
src/test/suite/extension.test.ts
contributes its language model tools
the language model tools return data from the workspace
Let an agent read GitHub activity Low
What it does · Impact if it breaks: low · Needs GitHub and Copilot
Why this level: Only the registration of this tool is tested; it is never called by a test, and the real GitHub client has no automated test.
Not covered: Never called by an automated test.
Manual script steps 10.3: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.
Tests: 2 unit, 1 integration
Unit tests
src/test/unit/insights.test.ts
builds search queries scoped to the repo and period
classifies issues, PRs and discussions by what happened in the period
Not covered: The CI results are kept as an artifact; the dashboard does not download them yet, so a local score has no integration test results until you save the CI file under .nabs/results.
No step in the manual script covers this.
Tests: 15 unit, 0 integration
Unit tests
src/test/unit/releaseResults.test.ts
Mocha JSON becomes a result file: passed, failed and skipped per test, with the source file of each
output that is not Mocha JSON, or has no tests, is reported in plain words
a result file written out reads back the same, and a damaged one says what is wrong
results count only for the same commit with a clean tree; another commit counts as not run
the evidence for a list of tests counts passed ones and names failed ones; skipped and missing ones earn nothing
a test with the same title in another file does not count
stale results give no credit and say why
every failed test of a fresh result is listed, whether or not an activity lists it
a run that lists sections counts every step in them as passed; other steps are not covered
a run that lists steps counts only those, with the result it gives
the newest run that covers a step decides it, by date
how fresh a run is: from its commit when it has one, else from its version
is valid YAML that passes the run schema, whatever is in the text
lists a section as run only when every step of it was run
says what is wrong with the input instead of writing a bad record
Not covered: The Run the Tests button runs only the fixed npm script (a workspace setting must not be able to run any command), so a repository without it cannot run tests from the dashboard. The dashboard does not download the CI results yet. The report that can be shared comes later (#94).
Manual script steps 18.1, 18.2, 18.3, 18.4, 18.5, 18.7, 18.8: not run by hand yet.
Tests: 8 unit, 1 integration
Unit tests
src/test/unit/releaseDashboard.test.ts
the dashboard says the verdict and the score first, with the reason in words
every activity is listed with its score, its level, where the score comes from and what would raise it
a gate is shown as the reason the release is held
the changed-code summary names the unowned files, and says so when there is no release tag to compare with
anything that comes from the catalogue or a run is escaped
no catalogue, or a broken one, says what is wrong and what to do instead of showing a score
the tree and Home show one line: the verdict and the score
the Release activities of the module are the activities of the catalogue group "release", and the settings and commands match package.json
Integration tests
src/test/suite/extension.test.ts
0.13.0: Release Confidence: no catalogue, then the dashboard, Home card, tree node, Run tests and Record a real-world run (#91, #93)
Not covered: The form lists the script's steps by title only; open the script to read what to do. It cannot edit or delete a recorded run (edit the file).
Manual script steps 18.6: not run by hand yet.
Tests: 4 unit, 1 integration
Unit tests
src/test/unit/releaseDashboard.test.ts
the manual script's sections and steps are read from its headings and check boxes
both step formats are read, and a line that is not a step is ignored
the form has the run details, a choice for each section and one for each step, and escapes the script
a step takes its own answer, else its section's; steps that were not run are left out
Integration tests
src/test/suite/extension.test.ts
0.13.0: Release Confidence: no catalogue, then the dashboard, Home card, tree node, Run tests and Record a real-world run (#91, #93)
Share the release report Low
What it does · Impact if it breaks: medium · Needs GitHub
Not covered: Posting a real Discussion is only tested against a fake GitHub (as for archiving); step 18.11 is the real test, and nobody has run it. The badge is a shields.io image address with the score written into it, not a live badge.
Manual script steps 18.9, 18.10, 18.11: not run by hand yet.
Tests: 9 unit, 1 integration
Unit tests
src/test/unit/releaseReport.test.ts
it states the verdict, the score, the commit and the tool, and explains the score in plain words
every activity is in a table with its score, level, impact and why, and the inputs are listed
a gate is the first thing said about why the release is held
text from the catalogue cannot break the table or add HTML
the badge is one line of text made now, with the score, the verdict, the date and the commit
the file name says the version, the day and the commit; the body ends with a marker so it is never posted twice
a body too long for a Discussion is shortened to the first thing that would raise each activity
the front matter is the history of the score: it reads back, and anything else is not a report
the saved reports are read oldest first, and a file that is not a report is skipped
Integration tests
src/test/suite/extension.test.ts
0.14.0: Release Report: Markdown, copy, save in the repository, and post to a Discussion only after the form is confirmed (#94)
Check, score and list gaps of any repository from the command line Low
Not covered: The first pilot is Number Our Days (#96). The package is built and packed locally but has never been published or installed from a registry, and there is no importable library, only the command. There is no import-graph plugin for languages other than TypeScript (affects derive counts as all), and the Release Report and the dashboard are still VS Code and GitHub only.
No step in the manual script covers this.
Tests: 14 unit, 0 integration
Unit tests
src/test/unit/standaloneCatalogue.test.ts
the XML reader handles entities, CDATA, attributes and self-closing elements, and refuses malformed files
C# test names are the methods under [Fact] and [Theory], not the helpers
a test class is found by its namespace and name, also nested, or by its simple name when only one file has it
TRX results become test results by source file and method name; skipped, failed and unknown classes are handled
JUnit XML results use the file attribute when there is one, else the class; failure and skipped are read
method names drop arguments and namespaces; formats are told apart by extension and first element
the version comes from package.json, a JSON field, or a pattern, and is ? when it is not there
the bundled schemas are the files in catalogue/schema (run npm run gen:schemas after changing one)
a repository with no npm, no schemas, no readiness and no scoring section can be checked, scored and recorded
src/test/unit/catalogueInit.test.ts
init scaffolds a catalogue that passes check on a repository that has none, named after its package
the scaffold can be scored straight away, and --product and --code decide the name and the files the first activity owns
init never overwrites: when any file it would write exists, it writes nothing and says which
gaps lists each written gap with its stable id, for one group or as JSON, and says so when there is no catalogue
the npm package is the one bundle with a bin entry and no dependencies, and npm pack would publish exactly its files
Integration tests: none.
Tag issues by activity and turn catalogue gaps into issues Low
Not covered: Closed issues of a gap that is still there, and open issues of a gap that has gone, are only reported, not changed. The issue numbers are not kept in the catalogue.
No step in the manual script covers this.
Tests: 10 unit, 0 integration
Unit tests
src/test/unit/catalogueGitHub.test.ts
a tag is a marker comment or the Activities field of an issue form, and nothing else counts
an activity gives the label of its group, a group gives its own, and an unknown id is reported and never labelled
labels are planned from the tags: untagged items are never touched, labelled ones need nothing, typos are listed
the labels command is a dry run unless --apply: it writes nothing, then labels issues and Discussions that have a tag and no others
a gap has a stable id, and its issue carries the activity tag and the gap id
a gap is matched to its issue by the marker, never the title; a closed gap is only reported
gap-issues is a dry run unless --apply, refuses to open more than five without --limit, and never opens the same gap twice
gap-issues opens the highest-impact gaps first, and --only chooses gaps by id and refuses ids it does not know
an action item tagged #act:<id> makes an issue that carries the tag, so the labels script finds it
GitHub is reached with a token or through the GitHub CLI; no token is read by the tool itself