NABS Confidence

Test coverage and confidence

How well each of the 69 activities in NABS Confidence 0.14.9 is tested, and how confident we are in releasing it. The levels come from a written rule, below, not from a feeling.

High 0 Medium 51 Low 18

Release readiness

Release 0.13.0 (adds Release Confidence to the 0.11.0 code under the NABS Confidence name) to the team as a preview. Do not call it fully verified.

The everyday flow is well tested by machine: create a retro, gather feedback from several people, edit and lock it, choose objectives, write action items and read insights from Git. Each of these has unit tests and integration tests, and most were also run by hand on 0.7.1 with real Git, GitHub and Copilot.

Nothing earns High yet, because nobody has run the manual script on the version being released. The newer features (the dashboard, objective focus, archiving, Home) have never been run by hand. The automated tests (unit, and integration in a real VS Code under a virtual display) do now run in GitHub Actions on every push to the main branch, so a broken build shows up there.

The weakest areas are the parts that talk to the outside world. Archiving to a GitHub Discussion has only been tested against a fake service. The Copilot call (model, consent, answer) and the @retro chat participant have never been run by an automated test. The real GitHub client has no test of its own.

Before the next release

  1. Check that the latest CI run on the main branch is green (compile, lint, unit tests, coverage check, boundary check, integration tests), or run npm test yourself.
  2. Run sections 13 to 17 of the real-world script on the release version, in particular section 15 (archive to a real Discussion in a test repository).
  3. Run sections 11 and 12 (offline and no-Copilot, panels keeping input) that were left over from 0.7.x.
  4. Re-run sections 5 to 10 on the release version, because the Copilot prompts changed in 0.9.0.
  5. Then add a run record to catalogue/runs (for example manual-0.12.0.yml) with the version, date and sections run, update this file, re-render with python scripts/user-guide/render.py, and run npm run check:coverage. Many Medium activities will then become High.

The main gaps

How confidence is decided

Each activity gets one level. The levels mean:

LevelMeaning
HighSafe to release without further checking. The activity has unit tests and integration tests, and a manual run of the real-world script has passed on the version being released. Nothing it needs is known to be untested.
MediumLikely to work. The activity has automated tests, and either both kinds (unit and integration) or a manual run on some version. Look at the gaps before relying on it.
LowNot verified. The activity has no automated test, or depends on GitHub or Copilot and has never been run by hand, or has only one kind of automated test and no manual run. Treat it as unproven.

The rule, in order:

  1. No automated test at all: Low.
  2. Needs GitHub or Copilot and has no manual run: Low.
  3. Only unit tests, or only integration tests, and no manual run: Low.
  4. Unit tests and integration tests and a manual run on the version being released: High.
  5. Everything else: Medium.
  6. A level may be set lower than the rule gives, with a written reason. It may never be set higher.
  7. Screenshot runs are shown as supporting evidence only. They do not change a level.

What counts as evidence

A level can only be as high as its evidence allows. Where a person judged it should be lower than the rule gives, the page says why. These levels describe the tests that exist. The automated tests run on every push to the main branch in GitHub Actions, and on a developer machine with npm test.

The manual script

The script has 20 numbered sections. Section 0 is preparation.

SectionTopicRun by hand
0Before you startYes, on 0.7.1
1SetupYes, on 0.7.1
2ObjectivesYes, on 0.7.1
3New RetroYes, on 0.7.1
4Feedback from two peopleYes, on 0.7.1
5Refresh Insights (real Git and GitHub data)Yes, on 0.7.1
6Copilot: summarise and assessYes, on 0.7.1
7Team verdict with an overrideYes, on 0.7.1
8Action items to real GitHub issues (test repository only)Yes, on 0.7.1
9Update Action Items after closing oneYes, on 0.7.1
10The @retro chat participantYes, on 0.7.1
11Offline and no-Copilot fallbacksNot yet
12Panels keep unsaved input (#37)Not yet
13The Retro Dashboard (0.8.0, #41)Not yet
14A retro focuses on its chosen objectives (0.9.0, #53)Not yet
15Archive a retro to a GitHub Discussion (0.10.0, #57)Not yet
16Home, the landing page (0.11.0, #61)Not yet
17Report a Problem and wrap-upNot yet
18Release Confidence (0.13.0, #91, #93) and the Release Report (0.14.0, #94, #95)Not yet
19The GitHub client against the real GitHub (0.14.9, #112, #113, #114)Not yet

Screenshot runs

The matrix

One table per group, then the detail for each activity: the exact tests, the manual steps and what is not covered.

Set up and settings

ActivityConfidenceUnitIntegrationManual
Start in a repositoryMedium114On 0.7.1
Change settings in the Settings panelMedium63On 0.7.1

Home

ActivityConfidenceUnitIntegrationManual
Open HomeMedium21Not run
Review objectives from HomeMedium41Not run
Start the next retro from HomeMedium52Not run
Read the maturity curve and the empty statesMedium31Not run

New Retro and templates

ActivityConfidenceUnitIntegrationManual
Create a retroMedium276On 0.7.1
Choose a feedback template for the retroMedium23On 0.7.1
Use the built-in templatesMedium132On 0.7.1
Use templates from the repository and a shared template repositoryMedium71On 0.7.1
Create a new feedback templateMedium13Not run
Refresh feedback templatesMedium11On 0.7.1
Get help while editing a template fileLow10Not run
Customise the retro document layoutMedium21Not run
Use your own feedback types and entry format (older settings)Medium71Not run

Gather and feedback

ActivityConfidenceUnitIntegrationManual
Add feedbackMedium184On 0.7.1
Write to your own file (per-person files)Medium21On 0.7.1
Say you are doneMedium13On 0.7.1
Edit, delete or find a feedback entryMedium83On 0.7.1
Open the feedback fileLow01Not run
Gather as the facilitator: who is done, and nudgesMedium51Not run
Gather as a participantMedium31Not run
Lock feedback by leaving GatherMedium41Not run

Retro Dashboard and Retro panel

ActivityConfidenceUnitIntegrationManual
Open the Retro DashboardMedium32Not run
Prepare the retroMedium41Not run
Switch between Facilitator and Participant viewsMedium21Not run
Move through the later stages from the dashboardMedium31Not run
Read and edit a retro in the Retro panelMedium153Not run
Set the retro statusMedium22Not run
Keep what you have typed when the panel refreshesMedium83Not run

Insights and summaries

ActivityConfidenceUnitIntegrationManual
Analyse Git activity for the periodMedium122On 0.7.1
Add GitHub issues, pull requests and discussions to the InsightsMedium40On 0.7.1
Summarise feedback without CopilotMedium22Not run
Summarise feedback with CopilotLow50On 0.7.1
Keep working without Copilot or offlineLow02Not run
See release confidence in the retro's InsightsMedium31Not run

Objectives

ActivityConfidenceUnitIntegrationManual
Create and edit an objectiveMedium132On 0.7.1
Use GitHub issues as objectivesMedium31On 0.7.1
Open an objectiveMedium03On 0.7.1
Choose which objectives a retro focuses onMedium93Not run
Link feedback and action items to objectivesMedium31Not run
Add evidence and see automatic measuresMedium32On 0.7.1
Assess objectives with CopilotLow22On 0.7.1
Record the team's verdictMedium42On 0.7.1
Use release confidence as an objective measureMedium132Not run

Decide: action items

ActivityConfidenceUnitIntegrationManual
Write and carry forward action itemsMedium41On 0.7.1
Create GitHub issues for action itemsMedium114On 0.7.1
Update action items from GitHubMedium12On 0.7.1
Open an action itemLow00Not run
Tag feedback and action items with a release activityMedium11Not run

Close and archive

ActivityConfidenceUnitIntegrationManual
Check where the archive will goLow01Not run
Post the retro to a GitHub DiscussionLow172Not run
Remove the archived files safelyLow12Not run
Find archived retrosLow31Not run

@retro chat and language model tools

ActivityConfidenceUnitIntegrationManual
Ask @retro in Copilot ChatLow20On 0.7.1
Let an agent read retros (#retros tools)Medium02On 0.7.1
Let an agent read GitHub activityLow21On 0.7.1

Diagnostics and the Retros view

ActivityConfidenceUnitIntegrationManual
Report a problemMedium62Not run
Read the logMedium01On 0.7.1
Never get a top-of-window promptMedium01On 0.7.1
Browse retros, feedback, objectives and archived retros in the Retros viewMedium43On 0.7.1

Release confidence

ActivityConfidenceUnitIntegrationManual
Work out the release confidence scoreLow210Not run
See what changed since the last releaseLow110Not run
Record test results and real-world runs against a commitLow150Not run
See the release confidence dashboardMedium81Not run
Record a real-world run in a formMedium41Not run
Share the release reportLow91Not run
Check, score and list gaps of any repository from the command lineLow140Not run
Tag issues by activity and turn catalogue gaps into issuesLow100Not run

Detail by activity

Set up and settings

Start in a repository Medium

What it does · Impact if it breaks: high

Not covered: Non-Git folders are not exercised by any test, and the multi-root folder choice is unit tested but not in a real multi-root window.

Manual script steps 1.2, 1.3: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 11 unit, 4 integration

Unit tests

src/test/unit/build.test.ts

  • esbuild.js bundles every webview page script
  • the F5 watch task runs esbuild and both type checkers

src/test/unit/modules.test.ts

  • activating a module returns its API and registers it
  • two modules cannot share an id, a settings namespace or an activity
  • the optional extension points are collected from the modules that have them
  • a problem report's environment facts are collected from the modules that have them
  • a folder is owned when any module says it has data there; a module that throws claims nothing
  • no folder: nothing; one folder: that folder (whatever the module says)
  • several folders: the active editor wins, then the first folder with module data, then the first
  • a folder the module cannot inspect is skipped
  • the Retros module's activities are exactly the catalogue's retro activities

Integration tests

src/test/suite/extension.test.ts

  • registers its commands
  • ships a built script for each webview form
  • resolves the repository and the retros folder
  • the Retros view lists the retros

Change settings in the Settings panel Medium

What it does · Impact if it breaks: medium

Not covered: Settings that only VS Code can edit (the JSON ones) are linked, not edited, in the panel; that link is not tested.

Manual script steps 1.4, 12.5: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 6 unit, 3 integration

Unit tests

src/test/unit/release0110.test.ts

  • both settings are in the Settings form

src/test/unit/settingsValues.test.ts

  • trims text and falls back when nothing is left
  • folders are relative: no leading or trailing slashes, a fallback when empty
  • numbers must be whole and above zero
  • a choice must be one of the allowed values
  • a setting whose values a module supplies accepts only those, or nothing

Integration tests

src/test/suite/extension.test.ts

  • Open Settings shows the Settings panel (#36)
  • the Settings panel (#36) writes to the workspace or user settings, only what changed, and resets
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Home

Open Home Medium

What it does · Impact if it breaks: medium

Manual script steps 16.1, 16.2: not run by hand yet.

Screenshot runs: Home (0.11.0).

Tests: 2 unit, 1 integration

Unit tests

src/test/unit/release0110.test.ts

  • only when the view is visible, nothing is open, and not already this window
  • the setting values

Integration tests

src/test/suite/extension.test.ts

  • 0.11.0: Home with nothing yet: both calls to action, the empty states, the curve, opening by itself (#61-#65)

Review objectives from Home Medium

What it does · Impact if it breaks: medium

Manual script steps 16.3, 16.4: not run by hand yet.

Screenshot runs: Home (0.11.0).

Tests: 4 unit, 1 integration

Unit tests

src/test/unit/release0110.test.ts

  • the history is the newest retro with the objective and its most recent review
  • needs review: never assessed, not in the latest retro, or older than the threshold
  • the default is two default periods
  • daysBetween counts calendar days

Integration tests

src/test/suite/extension.test.ts

  • 0.11.0: Home with objectives and retros: needs review, Assess, Continue and the next retro (#63, #64)

Start the next retro from Home Medium

What it does · Impact if it breaks: medium

Manual script steps 16.5: not run by hand yet.

Screenshot runs: Home (0.11.0).

Tests: 5 unit, 2 integration

Unit tests

src/test/unit/release0110.test.ts

  • no retros: today, then a period later
  • continuous: the latest period end plus defaultPeriodDays
  • overdue: today, so the period catches up
  • dates that already have a retro are skipped; a bad period falls back to 14 days
  • the saved stage wins; otherwise it is worked out from the content

Integration tests

src/test/suite/extension.test.ts

  • 0.11.0: Home with objectives and retros: needs review, Assess, Continue and the next retro (#63, #64)
  • 0.11.0: Home with nothing yet: both calls to action, the empty states, the curve, opening by itself (#61-#65)

Read the maturity curve and the empty states Medium

What it does · Impact if it breaks: low

Not covered: The two linked articles open in the browser; only the link handling is tested, not the pages.

Manual script steps 16.6, 16.7: not run by hand yet.

Screenshot runs: Home (0.11.0).

Tests: 3 unit, 1 integration

Unit tests

src/test/unit/release0110.test.ts

  • Darrel's four drawings, one per stage, in order
  • each is theme-aware: currentColor ink, classes for the other colours, no inline styles (the CSP blocks them)
  • goes into the page without its prolog and comments

Integration tests

src/test/suite/extension.test.ts

  • 0.11.0: Home with nothing yet: both calls to action, the empty states, the curve, opening by itself (#61-#65)

New Retro and templates

Create a retro Medium

What it does · Impact if it breaks: high

Not covered: The progress messages while creating are checked, not how they look.

Manual script steps 3.1: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Screenshot runs: Retro Dashboard (0.8.0, Gather again at 0.11.0).

Tests: 27 unit, 6 integration

Unit tests

src/test/unit/formModel.test.ts

  • accepts valid values
  • rejects an existing retro date
  • rejects invalid and empty dates
  • requires start <= end
  • allows a one-day period, but not one that ends after the retro
  • limits text lengths
  • normaliseValues trims and removes line breaks
  • suggestPeriod is the default window ending on the retro date
  • describeContinuity flags an overlap or gap for the chosen start
  • parses valid messages from the webview
  • rejects malformed messages from the webview
  • parseFormValues drops unknown properties
  • parses messages from the extension

src/test/unit/naming-period.test.ts

  • round-trips retro and feedback file names
  • does not confuse retro, feedback and other files
  • defaults to a fixed window of N days, inclusive, ending on the retro date
  • finds the latest earlier retro, using its periodEnd when valid
  • ignores retros on or after the new date (e.g. back-filling an old retro)
  • reports continuity with the previous retro
  • rejects nonsense period lengths
  • periodLengthInDays handles daylight saving changes

src/test/unit/dates.test.ts

  • isIsoDate accepts real dates and rejects impossible ones
  • parseIsoDate trims and throws helpfully
  • addDays crosses month, year and leap-day boundaries
  • toIsoDate uses local time, not UTC
  • isWithinPeriod is inclusive at both ends
  • toLocalIsoTimestamp includes the local offset

Integration tests

src/test/suite/extension.test.ts

  • creates a first retro via the New Retro command (form values as argument), without AI
  • the New Retro form opens once, with defaults that follow the previous retro
  • the form shows inline validation errors instead of creating a duplicate retro
  • Cancel closes the form
  • a second retro covers the gap since the first and carries its action items forward
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Choose a feedback template for the retro Medium

What it does · Impact if it breaks: high

Manual script steps 3.1: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Screenshot runs: Gather (0.11.0).

Tests: 2 unit, 3 integration

Unit tests

src/test/unit/feedbackTemplates.test.ts

  • with no settings: the built-ins, defaulting to classic
  • an unknown default or retro template falls back

Integration tests

src/test/suite/extension.test.ts

  • the New Retro form opens once, with defaults that follow the previous retro
  • a retro with the health check template has its own types, ratings and headings
  • the Retro panel (#33): details, feedback done, and the Create GitHub Issues confirmation

Use the built-in templates Medium

What it does · Impact if it breaks: medium

Not covered: Only the classic and health check templates are exercised end to end; the other built-ins are covered by the template validity test.

Manual script steps 3.1: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 13 unit, 2 integration

Unit tests

src/test/unit/feedbackTemplates.test.ts

  • the built-ins are valid, with unique ids and types
  • with no settings: the built-ins, defaulting to classic
  • the legacy feedbackTypes setting becomes the "custom" template and the default
  • custom templates are validated, added, and replace built-ins with the same id
  • an unknown default or retro template falls back
  • a rating's section wins over the type's
  • questions render as a Markdown list

src/test/unit/release040.test.ts

  • the incident review has a root-cause type with an outline, and guidance for Copilot
  • custom types can have a prompt; submitting the outline unchanged is rejected
  • the built-in health check has an optional trend scale
  • the trend is stored in the marker, survives edits, and old entries are unaffected
  • summaries and prompts include the trend
  • the form validates the trend and parses the new messages

Integration tests

src/test/suite/extension.test.ts

  • a retro with the health check template has its own types, ratings and headings
  • the health check records a trend per dimension

Use templates from the repository and a shared template repository Medium

What it does · Impact if it breaks: medium · Needs GitHub

Not covered: The shared repository is read through a fake in the automated test; the real GitHub fetch was run by hand on 0.7.1 only.

Manual script steps 1.5: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 7 unit, 1 integration

Unit tests

src/test/unit/templateSources.test.ts

  • parseTemplateSources keeps known, unique sources in order
  • parseSharedTemplateRef understands owner/repo, paths, refs, URLs and local folders
  • isTemplateSourceFile picks template.json, retro.md and templates.json only
  • loadTemplateSource reads folders, ids, layouts and the default, and reports problems
  • precedence follows nabsConfidence.templateSources; personal settings win; the default comes from the sources
  • templateToJson and blankTemplateJson round-trip through the validator
  • the template.json schema matches the validator and the feedbackTemplates setting

Integration tests

src/test/suite/extension.test.ts

  • feedback templates from the repo and a shared repository (fake GitHub), with caching and offline fallback (#14)

Create a new feedback template Medium

What it does · Impact if it breaks: low

No step in the manual script covers this.

Tests: 1 unit, 3 integration

Unit tests

src/test/unit/templateSources.test.ts

  • templateToJson and blankTemplateJson round-trip through the validator

Integration tests

src/test/suite/extension.test.ts

  • feedback templates from the repo and a shared repository (fake GitHub), with caching and offline fallback (#14)
  • the Settings panel (#36) writes to the workspace or user settings, only what changed, and resets
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Refresh feedback templates Medium

What it does · Impact if it breaks: low · Needs GitHub

Not covered: The command itself is never invoked by a test; the forced refresh is exercised through the template library.

Manual script steps 1.5: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 1 unit, 1 integration

Unit tests

src/test/unit/templateSources.test.ts

  • precedence follows nabsConfidence.templateSources; personal settings win; the default comes from the sources

Integration tests

src/test/suite/extension.test.ts

  • feedback templates from the repo and a shared repository (fake GitHub), with caching and offline fallback (#14)

Get help while editing a template file Low

What it does · Impact if it breaks: low

Not covered: The editor experience itself (completion and hover in VS Code) is not tested.

No step in the manual script covers this.

Tests: 1 unit, 0 integration

Unit tests

src/test/unit/templateSources.test.ts

  • the template.json schema matches the validator and the feedbackTemplates setting

Integration tests: none.

Customise the retro document layout Medium

What it does · Impact if it breaks: low

Not covered: The command itself is never invoked by a test; the template rendering and the layout created on first use are.

No step in the manual script covers this.

Tests: 2 unit, 1 integration

Unit tests

src/test/unit/document.test.ts

  • the default template renders every placeholder
  • custom templates can drop sections and keep unknown placeholders

Integration tests

src/test/suite/extension.test.ts

  • creates a first retro via the New Retro command (form values as argument), without AI

Use your own feedback types and entry format (older settings) Medium

What it does · Impact if it breaks: low

No step in the manual script covers this.

Tests: 7 unit, 1 integration

Unit tests

src/test/unit/feedbackTypes.test.ts

  • missing settings use the defaults without complaint
  • valid custom types are used, with optional fields cleaned up
  • invalid entries are skipped with a problem; nothing valid means the defaults
  • the entry template must have {{text}} once and only known placeholders
  • unknown type ids display as Other

src/test/unit/feedback.test.ts

  • a custom template round-trips, and entries survive the template changing later
  • the template is filled in one pass (text containing placeholders is left alone)

Integration tests

src/test/suite/extension.test.ts

  • the legacy feedbackTypes setting becomes the "custom" template; the entry template comes from the settings

Gather and feedback

Add feedback Medium

What it does · Impact if it breaks: high

Manual script steps 4.1: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Screenshot runs: Gather (0.11.0).

Tests: 18 unit, 4 integration

Unit tests

src/test/unit/feedbackFormModel.test.ts

  • valid values have no errors
  • reports each invalid field
  • defaults to the newest in-progress retro, otherwise the newest
  • parses messages from the webview strictly
  • parses messages from the extension
  • a rating is required when the retro has a rating scale
  • in edit mode an entry may keep a type that is no longer in the template
  • finds the template of the selected retro

src/test/unit/feedback.test.ts

  • a new file has no entries (the format help is ignored) and lists the types
  • the default format is the v0.1.0 heading, preceded by a marker
  • appended entries round-trip through the parser
  • a custom template round-trips, and entries survive the template changing later
  • still reads hand-written v0.1.0 entries, including after marker entries
  • keeps entries with unknown types (shown as "other"), skips empty ones, reports bad markers
  • sanitises the author and text so the entry stays parseable
  • anonymous entries are written as "Anonymous" and flagged in the marker
  • the template is filled in one pass (text containing placeholders is left alone)
  • renders grouped lists and counts by configured type

Integration tests

src/test/suite/extension.test.ts

  • the Add Feedback form opens once, preselects the clicked retro, and supports "add another"
  • a retro with the health check template has its own types, ratings and headings
  • the health check records a trend per dimension
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Write to your own file (per-person files) Medium

What it does · Impact if it breaks: high

Manual script steps 4.1, 4.3: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Screenshot runs: Gather (0.11.0).

Tests: 2 unit, 1 integration

Unit tests

src/test/unit/release040.test.ts

  • person slugs make safe, stable file names
  • a personal file says whose it is and parses like a shared one

Integration tests

src/test/suite/extension.test.ts

  • per-person feedback files: each contributor writes their own file and marks it done

Say you are done Medium

What it does · Impact if it breaks: medium

Manual script steps 4.2: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Screenshot runs: Gather (0.11.0).

Tests: 1 unit, 3 integration

Unit tests

src/test/unit/release040.test.ts

  • the done signal is set and cleared in the front matter, keeping the entries

Integration tests

src/test/suite/extension.test.ts

  • per-person feedback files: each contributor writes their own file and marks it done
  • the Retro panel (#33): details, feedback done, and the Create GitHub Issues confirmation
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Edit, delete or find a feedback entry Medium

What it does · Impact if it breaks: medium

Not covered: Show in Feedback File is only checked to open the Retro panel; the jump to the entry in the file is not tested.

Manual script steps 4.4: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 8 unit, 3 integration

Unit tests

src/test/unit/feedbackEdit.test.ts

  • ids, ratings and edit times round-trip through the marker
  • parsing records where each entry is
  • updates one entry in place, keeping the others and the id
  • refuses to overwrite an entry that changed since it was opened
  • deletes one entry, tidying the blank lines
  • hand-written (v0.1.0) entries can be edited by fingerprint and become marker entries
  • keeps Windows line endings
  • comments in the file do not shift the positions

Integration tests

src/test/suite/extension.test.ts

  • the Retros view lists feedback entries, and the edit form updates and deletes them in place
  • per-person feedback files: each contributor writes their own file and marks it done
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Open the feedback file Low

What it does · Impact if it breaks: low

Not covered: Only the panel it opens is checked.

No step in the manual script covers this.

Tests: 0 unit, 1 integration

Unit tests: none.

Integration tests

src/test/suite/extension.test.ts

  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Gather as the facilitator: who is done, and nudges Medium

What it does · Impact if it breaks: medium

Manual script steps 13.4: not run by hand yet.

Screenshot runs: Gather (0.11.0).

Tests: 5 unit, 1 integration

Unit tests

src/test/unit/retroProgress.test.ts

  • Gather checks who is done, the feedback types and the lock
  • counts entries across people and the shared file
  • Gather for the facilitator: invite, nudge, then move on
  • invitation and nudge texts
  • "most people": everyone but one with three or more people

Integration tests

src/test/suite/extension.test.ts

  • the Retro Dashboard (#41): stages, readiness, the soft gate and the feedback lock

Gather as a participant Medium

What it does · Impact if it breaks: medium

Manual script steps 13.4: not run by hand yet.

Screenshot runs: Gather (0.11.0).

Tests: 3 unit, 1 integration

Unit tests

src/test/unit/retroProgress.test.ts

  • Gather for a participant: add feedback, mark done, then done
  • facilitator by default when it is you, or nobody yet
  • matches people by name or feedback file name

Integration tests

src/test/suite/extension.test.ts

  • the Retro Dashboard (#41): stages, readiness, the soft gate and the feedback lock

Lock feedback by leaving Gather Medium

What it does · Impact if it breaks: high

Not covered: The lock is enforced by the extension only: a hand edit of a feedback file, or a teammate on an older version, is not blocked.

Manual script steps 13.5, 13.6: not run by hand yet.

Screenshot runs: Retro Dashboard (0.8.0, Gather again at 0.11.0), Gather (0.11.0).

Tests: 4 unit, 1 integration

Unit tests

src/test/unit/retroProgress.test.ts

  • moving past Gather locks the feedback and lists what is still open
  • going back to Gather (or Prepare) unlocks; moving within the locked stages leaves it
  • applyMove writes the stage, who and when, and the lock
  • Gather when feedback is locked: only the facilitator can go back

Integration tests

src/test/suite/extension.test.ts

  • the Retro Dashboard (#41): stages, readiness, the soft gate and the feedback lock

Retro Dashboard and Retro panel

Open the Retro Dashboard Medium

What it does · Impact if it breaks: high

Manual script steps 13.1, 13.2: not run by hand yet.

Screenshot runs: Retro Dashboard (0.8.0, Gather again at 0.11.0).

Tests: 3 unit, 2 integration

Unit tests

src/test/unit/retroProgress.test.ts

  • six stages in order, only Prepare and Gather guided in this version
  • infers the stage of a retro without one
  • the stage in the front matter wins over the inferred one

Integration tests

src/test/suite/extension.test.ts

  • the Retro Dashboard (#41): stages, readiness, the soft gate and the feedback lock
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Prepare the retro Medium

What it does · Impact if it breaks: medium

Manual script steps 13.3: not run by hand yet.

Screenshot runs: Retro Dashboard (0.8.0, Gather again at 0.11.0).

Tests: 4 unit, 1 integration

Unit tests

src/test/unit/retroProgress.test.ts

  • Prepare is ready when the period, template, objectives and sharing are fine
  • Prepare warns about gaps, unknown templates, no objectives and unpushed files
  • Prepare warns when the period is missing
  • Prepare: the facilitator chooses objectives, shares, then moves on

Integration tests

src/test/suite/extension.test.ts

  • the Retro Dashboard (#41): stages, readiness, the soft gate and the feedback lock

Switch between Facilitator and Participant views Medium

What it does · Impact if it breaks: low

Manual script steps 13.4: not run by hand yet.

Screenshot runs: Retro Dashboard (0.8.0, Gather again at 0.11.0), Gather (0.11.0).

Tests: 2 unit, 1 integration

Unit tests

src/test/unit/retroProgress.test.ts

  • Gather for a participant: add feedback, mark done, then done
  • facilitator by default when it is you, or nobody yet

Integration tests

src/test/suite/extension.test.ts

  • the Retro Dashboard (#41): stages, readiness, the soft gate and the feedback lock

Move through the later stages from the dashboard Medium

What it does · Impact if it breaks: medium

Not covered: The buttons that open other panels from a later stage are not clicked in any automated test.

Manual script steps 13.2, 13.6: not run by hand yet.

Screenshot runs: Retro Dashboard (0.8.0, Gather again at 0.11.0).

Tests: 3 unit, 1 integration

Unit tests

src/test/unit/retroProgress.test.ts

  • the later stages have checklists too
  • the later stages link out to the existing panels
  • applyMove writes the stage, who and when, and the lock

Integration tests

src/test/suite/extension.test.ts

  • the Retro Dashboard (#41): stages, readiness, the soft gate and the feedback lock

Read and edit a retro in the Retro panel Medium

What it does · Impact if it breaks: high

Manual script steps 12.1: not run by hand yet.

Screenshot runs: Retro Dashboard (0.8.0, Gather again at 0.11.0).

Tests: 15 unit, 3 integration

Unit tests

src/test/unit/panels.test.ts

  • accepts only well-formed messages from a panel
  • accepts only well-formed messages from the extension
  • escapes everything from files and settings
  • validates the details
  • reads the details (TBC is no facilitator)
  • writes only what changed, and renames the heading with the title
  • leaves a heading that isn't the title alone
  • renders bold, italic and code, and escapes everything else

src/test/unit/document.test.ts

  • parses simple key/value pairs and quoted values
  • handles Windows line endings
  • returns the whole text as body when there is no front matter
  • setFrontMatterValue updates, adds, and creates

src/test/unit/panelUpdates.test.ts

  • splits a page into its regions, nested elements included
  • sends only the regions that changed: an action in Feedback leaves Details alone
  • reloads the page when its structure changes

Integration tests

src/test/suite/extension.test.ts

  • the Retro panel (#33): details, feedback done, and the Create GitHub Issues confirmation
  • the Retros view lists the retros
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Set the retro status Medium

What it does · Impact if it breaks: low

Manual script steps 12.1: not run by hand yet.

Tests: 2 unit, 2 integration

Unit tests

src/test/unit/panels.test.ts

  • validates the details
  • writes only what changed, and renames the heading with the title

Integration tests

src/test/suite/extension.test.ts

  • the Retro panel (#33): details, feedback done, and the Create GitHub Issues confirmation
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Keep what you have typed when the panel refreshes Medium

What it does · Impact if it breaks: medium

Manual script steps 12.2, 12.3, 12.4, 12.6: not run by hand yet.

Tests: 8 unit, 3 integration

Unit tests

src/test/unit/panelUpdates.test.ts

  • marks the page for unsaved changes: in-panel confirmation, restore notice, commit and discard buttons
  • a draft is a field that differs from what the extension rendered
  • unsaved input in section B survives an action in section A
  • the action that saves a section drops its drafts, and only those
  • a field changed elsewhere meanwhile keeps your version and is reported
  • drafts of a field that is gone are dropped (e.g. an action item that became an issue)
  • switching retro keeps the drafts for later, and brings them back
  • reads back what setState stored, and ignores anything else

Integration tests

src/test/suite/extension.test.ts

  • panels keep unsaved input (#37): only changed regions are sent, drafts survive actions, hiding and switching
  • the Add Feedback form opens once, preselects the clicked retro, and supports "add another"
  • the New Retro form opens once, with defaults that follow the previous retro

Insights and summaries

Analyse Git activity for the period Medium

What it does · Impact if it breaks: high

Manual script steps 5.1, 5.2: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 12 unit, 2 integration

Unit tests

src/test/unit/insights.test.ts

  • parses commits, numstat lines, binary files and renames
  • normalises rename notation
  • summarises authors, churn and hot files
  • handles an empty log

src/test/unit/gitHistory.test.ts

  • a very large history: thousands of commits and a vendor import of tens of thousands of files are read and summarised, only the period counts
  • output over the buffer is a plain sentence about the period, not Node's buffer message
  • a shallow clone says so and leaves out its oldest commit, which Git shows as adding every file
  • merges: merge commits are not counted, the commits they bring in are, once
  • unusual renames and names: moved to the root, into a new folder, spaces, non-ASCII, and a tab; each file is counted under its new path
  • git's quoted paths and rename notation are read back to the real path

src/test/unit/document.test.ts

  • reads and replaces only the block content
  • returns undefined when markers are missing

Integration tests

src/test/suite/extension.test.ts

  • creates a first retro via the New Retro command (form values as argument), without AI
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Add GitHub issues, pull requests and discussions to the Insights Medium

What it does · Impact if it breaks: medium · Needs GitHub

Not covered: The real GitHub client (sign-in, REST and GraphQL calls, paging, rate limits, private repositories) has no automated test, not even against a fake. Only the pure helpers are tested. It was checked by hand on 0.7.1.

Manual script steps 5.1, 5.2: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 4 unit, 0 integration

Unit tests

src/test/unit/insights.test.ts

  • parses common remote URL formats
  • builds search queries scoped to the repo and period
  • classifies issues, PRs and discussions by what happened in the period
  • extracts GraphQL error messages

Integration tests: none.

Summarise feedback without Copilot Medium

What it does · Impact if it breaks: high

Not covered: The modal that protects hand-edited sections is not exercised by an automated test.

Manual script steps 11.1: not run by hand yet.

Tests: 2 unit, 2 integration

Unit tests

src/test/unit/feedback.test.ts

  • renders grouped lists and counts by configured type

src/test/unit/release040.test.ts

  • summaries and prompts include the trend

Integration tests

src/test/suite/extension.test.ts

  • feedback can be added and summarised into the retro
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Summarise feedback with Copilot Low

What it does · Impact if it breaks: medium · Needs Copilot

Why this level: The Copilot call itself (model choice, consent, the request and the streamed answer) is never run by an automated test: the integration tests switch Copilot off. Only the prompts and the parsing of answers are unit tested. The last real run was on 0.7.1, before the prompts changed in 0.9.0.

Manual script steps 6.1, 14.7: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 5 unit, 0 integration

Unit tests

src/test/unit/insights.test.ts

  • feedback prompt includes the feedback and the markers
  • parses a well-formed model answer and rejects a malformed one

src/test/unit/release090.test.ts

  • the summary prompts focus on the chosen objectives, with an "Outside focus" line
  • without objectives the prompts are as before

src/test/unit/release040.test.ts

  • summaries and prompts include the trend

Integration tests: none.

Keep working without Copilot or offline Low

What it does · Impact if it breaks: medium · Needs Copilot

Why this level: Every integration test runs with Copilot off, so the no-Copilot path is well covered, but it was never run by hand on a current version (the offline and no-Copilot section was still to do after the 0.7.1 run) and the Copilot-on path is untested.

Not covered: Offline behaviour for GitHub calls is only covered for template caching; the manual offline section has not been run.

Manual script steps 11.1, 11.2, 11.3: not run by hand yet.

Tests: 0 unit, 2 integration

Unit tests: none.

Integration tests

src/test/suite/extension.test.ts

  • feedback can be added and summarised into the retro
  • feedback templates from the repo and a shared repository (fake GitHub), with caching and offline fallback (#14)

See release confidence in the retro's Insights Medium

What it does · Impact if it breaks: medium

Not covered: Only reports somebody saved and committed are read (it is not live). The section is not part of the Copilot summary prompt in the chat participant.

No step in the manual script covers this.

Tests: 3 unit, 1 integration

Unit tests

src/test/unit/releaseActivityLinks.test.ts

  • "Release confidence since the last retro": start and end, the trend and the activities that moved, falls first
  • the section says so when the period has no report, one report, or old reports without levels
  • Retros adds the sections other modules provide to the Insights, and a section that fails is only left out

Integration tests

src/test/suite/extension.test.ts

  • 0.14.4: an activity's level is an objective measure, the retro's Insights show release confidence since the last retro, and #act:<id> shows on the dashboard (#107)

Objectives

Create and edit an objective Medium

What it does · Impact if it breaks: high

Manual script steps 2.1, 2.2: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Screenshot runs: Objectives (0.9.0 to 0.11.0).

Tests: 13 unit, 2 integration

Unit tests

src/test/unit/objectiveForm.test.ts

  • validates the fields and criteria inline
  • a new objective can't reuse another objective's id
  • new criteria get the next free id; existing ids are kept
  • a new objective file round-trips through the parser
  • editing keeps other front matter keys and sections, and empties optional fields
  • the file New Objective used to write still parses after editing
  • a verdict must be chosen
  • an override of the AI's verdict needs a reason
  • agreeing with the AI, or no AI assessment, needs no reason
  • evidence must not be empty

src/test/unit/objectives.test.ts

  • parses an objective file: front matter, why, criteria with ids, done and automatic measures
  • reports problems instead of failing: bad ids, missing titles, unknown metrics, bad dates
  • slugs and new objective files

Integration tests

src/test/suite/extension.test.ts

  • the Objective form (#34) creates and edits an objective, with its acceptance criteria
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Use GitHub issues as objectives Medium

What it does · Impact if it breaks: medium · Needs GitHub

Not covered: Issues are read through a fake in the integration test and a fake GitHub (paging, pull requests left out) in the unit tests; the real read was done by hand on 0.7.1 only.

Manual script steps 2.3: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 3 unit, 1 integration

Unit tests

src/test/unit/objectives.test.ts

  • parses an objective issue: header lines, ids for this and other repositories, state
  • merges sources in order (the first wins) and validates nabsConfidence.objectiveSources

src/test/unit/githubClient.test.ts

  • objective issues: labelled issues are listed over pages without pull requests, with their state and assignees

Integration tests

src/test/suite/extension.test.ts

  • objectives (#20): files and GitHub issues (fake), the chosen objectives in each new retro (0.9.0), linking, evidence, assessment and verdicts

Open an objective Medium

What it does · Impact if it breaks: low

Not covered: Opening an issue-based objective in the browser is not tested.

Manual script steps 2.1, 14.9: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 0 unit, 3 integration

Unit tests: none.

Integration tests

src/test/suite/extension.test.ts

  • the Objective form (#34) creates and edits an objective, with its acceptance criteria
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)
  • objectives (#20): files and GitHub issues (fake), the chosen objectives in each new retro (0.9.0), linking, evidence, assessment and verdicts

Choose which objectives a retro focuses on Medium

What it does · Impact if it breaks: high

Not covered: An integration test walks section 14's steps 14.1-14.5 and 14.9 through the real New Retro form, dashboard and Retro panel code, but nobody has run section 14 by hand in VS Code, so what the pages do in the browser (ticking boxes, the confirmation dialogs) and the Copilot steps 14.6-14.8 are unchecked.

Manual script steps 14.1, 14.2, 14.3, 14.4, 14.5, 14.9: not run by hand yet.

Screenshot runs: Retro Dashboard (0.8.0, Gather again at 0.11.0).

Tests: 9 unit, 3 integration

Unit tests

src/test/unit/release090.test.ts

  • New Retro: the open objectives are ticked by default; achieved and dropped ones are listed unticked
  • an existing retro: its own objectives are ticked, ones no longer in the sources are kept
  • readSelection and cleanIds read ticked boxes and well-formed ids only
  • changeSelection keeps existing regions verbatim, adds new ones and reports removals
  • removing an objective with evidence, an assessment or a verdict needs confirmation
  • New Retro form values: objectives are optional and cleaned; zero ticked gives a soft warning
  • Dashboard: Prepare asks for at least one objective (a soft gate)

src/test/unit/objectives.test.ts

  • the layout: the section is dropped when nothing is open, and added to older retros
  • regions: set, missing markers, meta escaping

Integration tests

src/test/suite/extension.test.ts

  • objectives (#20): files and GitHub issues (fake), the chosen objectives in each new retro (0.9.0), linking, evidence, assessment and verdicts
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)
  • manual script section 14 (0.9.0, #53), steps 14.1 to 14.5 and 14.9 in order, as a person does them: New Retro, the dashboard, the Retro panel (#111)

Add evidence and see automatic measures Medium

What it does · Impact if it breaks: medium

Manual script steps 6.3, 12.5: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Screenshot runs: Objectives (0.9.0 to 0.11.0).

Tests: 3 unit, 2 integration

Unit tests

src/test/unit/objectiveForm.test.ts

  • evidence must not be empty

src/test/unit/objectives.test.ts

  • computes automatic measures from the numbers we already collect
  • renders the snapshot and reads it back (criteria, measures, evidence, links)

Integration tests

src/test/suite/extension.test.ts

  • objectives (#20): files and GitHub issues (fake), the chosen objectives in each new retro (0.9.0), linking, evidence, assessment and verdicts
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Assess objectives with Copilot Low

What it does · Impact if it breaks: medium · Needs Copilot

Why this level: The Copilot call itself (model choice, consent, the request and the streamed answer) is never run by an automated test: the integration tests switch Copilot off. Only the prompts and the parsing of answers are unit tested. The last real run was on 0.7.1, before the prompts changed in 0.9.0.

Manual script steps 6.2: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 2 unit, 2 integration

Unit tests

src/test/unit/objectives.test.ts

  • the assessment prompt and its answer
  • records the AI assessment and the team verdict, with the override and its reason

Integration tests

src/test/suite/extension.test.ts

  • every command opens its form or picker, whatever VS Code passes from its menus (#22)
  • objectives (#20): files and GitHub issues (fake), the chosen objectives in each new retro (0.9.0), linking, evidence, assessment and verdicts

Record the team's verdict Medium

What it does · Impact if it breaks: medium

Manual script steps 7.1, 7.2: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Screenshot runs: Objectives (0.9.0 to 0.11.0).

Tests: 4 unit, 2 integration

Unit tests

src/test/unit/objectiveForm.test.ts

  • a verdict must be chosen
  • an override of the AI's verdict needs a reason
  • agreeing with the AI, or no AI assessment, needs no reason

src/test/unit/objectives.test.ts

  • records the AI assessment and the team verdict, with the override and its reason

Integration tests

src/test/suite/extension.test.ts

  • objectives (#20): files and GitHub issues (fake), the chosen objectives in each new retro (0.9.0), linking, evidence, assessment and verdicts
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Use release confidence as an objective measure Medium

What it does · Impact if it breaks: medium

Not covered: Only reports somebody saved and committed are read (it is not live). A report saved before 0.14.4 has no levels in its front matter; they are read from its table. The activity measures are one per activity, so the objective form lists them all (by title). Nobody has run steps 18.12 and 18.13.

Manual script steps 18.12, 18.13: not run by hand yet.

Tests: 13 unit, 2 integration

Unit tests

src/test/unit/releaseRetroLink.test.ts

  • the newest score in the period, in plain words with the release and commit
  • the trend compares the newest score with the report before it
  • releases in a row at 80 or more: the evidence for "80 or more for three releases"
  • the Release module provides the measures through the core contract, reading the repository root it is given
  • a measure with a dot in its id is accepted in a criterion; other names still are not
  • the retro line for a release measure names the provider, the value and the period; one nobody provides says so
  • the snapshot and the refresh fill in the release measures; the measures Retros has are unchanged
  • Retros reads the providers it is given: every id gets a reading, a failing provider is only unavailable
  • the objective form accepts a release measure and the file keeps it as (auto: <id>)

src/test/unit/releaseActivityLinks.test.ts

  • a saved report lists every activity with its score and level, and it reads back
  • a report saved before 0.14.4 has the levels in its table only, and they are read from there
  • release.activity.<id>.level: the level (1 Low to 3 High) in the newest report of the period, with how it moved
  • the activities of the catalogue are on offer, one measure each, and the family answers any of them

Integration tests

src/test/suite/extension.test.ts

  • 0.14.1: an objective can use release confidence as its automatic measure, read from the saved release reports (#95)
  • 0.14.4: an activity's level is an objective measure, the retro's Insights show release confidence since the last retro, and #act:<id> shows on the dashboard (#107)

Decide: action items

Write and carry forward action items Medium

What it does · Impact if it breaks: high

Manual script steps 3.1, 8.1: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 4 unit, 1 integration

Unit tests

src/test/unit/document.test.ts

  • parses open and done task list items
  • renders a follow-up section

src/test/unit/release040.test.ts

  • parses mentions, due dates and issue links; ignores the placeholder
  • the follow-up section shows the follow-through rate

Integration tests

src/test/suite/extension.test.ts

  • a second retro covers the gap since the first and carries its action items forward

Create GitHub issues for action items Medium

What it does · Impact if it breaks: high · Needs GitHub

Not covered: The real client's requests are only tested against a fake GitHub written from GitHub's documentation (labels, assignees, the rate limit, 401/404/410 and outage failures, resuming); the fake can be wrong about what GitHub does. Real issue creation was checked by hand on 0.7.1 only, and `npm run verify:github` (section 19) has not been run.

Manual script steps 8.1, 8.2, 19.1, 19.2, 19.4: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 11 unit, 4 integration

Unit tests

src/test/unit/release040.test.ts

  • issue titles and bodies are clean
  • links and ticks are written into the right line, keeping the rest

src/test/unit/githubClient.test.ts

  • issues: the label is checked then created, the issue request has the documented shape, and the answer is read from its real shape
  • issues: an assignee who cannot be assigned gives an unassigned issue, a label made meanwhile is fine, and other refusals say what GitHub said
  • issues: the failures GitHub really answers with: no access, issues turned off, a revoked sign-in, the rate limit, an outage and a dropped connection
  • not signed in: no request is made, and the message says to run the command again and sign in
  • action item states: up to 50 issues per request, closed and open read, an issue that does not exist is left out
  • explainFailure leaves other messages alone and only mentions the rate limit for the codes GitHub uses for it
  • it reports a failure when GitHub disagrees, and skips the Discussion checks when there is no category
  • read-only: it reads, never writes, and skips the writing checks
  • with --write: it creates a label, two issues (closed again) and a Discussion with comments, and every check passes

Integration tests

src/test/suite/extension.test.ts

  • action items become GitHub issues, are linked in the retro, and tick themselves off when closed
  • the Retro panel (#33): details, feedback done, and the Create GitHub Issues confirmation
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)
  • action items become real GitHub requests (a fake GitHub that checks the documented rules): labels, assignees that cannot be assigned, the rate limit midway, resuming, then the ticking off (#112)

Update action items from GitHub Medium

What it does · Impact if it breaks: medium · Needs GitHub

Not covered: The state of real issues is read through a fake in the automated test.

Manual script steps 9.1: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 1 unit, 2 integration

Unit tests

src/test/unit/release040.test.ts

  • links and ticks are written into the right line, keeping the rest

Integration tests

src/test/suite/extension.test.ts

  • action items become GitHub issues, are linked in the retro, and tick themselves off when closed
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Open an action item Low

What it does · Impact if it breaks: low · Needs GitHub

Not covered: No automated test and no manual step covers this command.

No step in the manual script covers this.

Tests: 0 unit, 0 integration

Unit tests: none.

Integration tests: none.

Tag feedback and action items with a release activity Medium

What it does · Impact if it breaks: medium

Not covered: The ids are not checked against the catalogue when you type them (the table says when there is no level for one). The tag is not offered in the Add Feedback form.

No step in the manual script covers this.

Tests: 1 unit, 1 integration

Unit tests

src/test/unit/releaseActivityLinks.test.ts

  • #act:<id> links a feedback entry or an action item to an activity; the issue gets an activity:<id> label and a clean title

Integration tests

src/test/suite/extension.test.ts

  • 0.14.4: an activity's level is an objective measure, the retro's Insights show release confidence since the last retro, and #act:<id> shows on the dashboard (#107)

Close and archive

Check where the archive will go Low

What it does · Impact if it breaks: medium · Needs GitHub

Not covered: GitHub is a fake in the automated test; the real checks (Discussions turned off, a missing category) have never been run. Section 15 of the manual script has never been run by hand.

Manual script steps 15.1, 15.2, 15.3: not run by hand yet.

Tests: 0 unit, 1 integration

Unit tests: none.

Integration tests

src/test/suite/extension.test.ts

  • 0.10.0: archive a retro to a GitHub Discussion, resume without duplicates, then remove its files safely (#57-#60)

Post the retro to a GitHub Discussion Low

What it does · Impact if it breaks: high · Needs GitHub

Not covered: Posting is tested through the real client against a fake GitHub written from GitHub's documentation (the rate limit, a revoked sign-in, paging, GitHub's 65,536-character limit, resuming, finding the Discussion by its marker), not against GitHub: the fake can be wrong about what GitHub does (GitHub's real refusal messages, how soon a new Discussion can be found by search, permission errors on organisation repositories). `npm run verify:github` and manual steps 15.5-15.7, 15.13 and 19.3 have not been run.

Manual script steps 15.4, 15.5, 15.6, 15.7, 15.13, 19.1, 19.3, 19.4: not run by hand yet.

Screenshot runs: Archive (0.11.0).

Tests: 17 unit, 2 integration

Unit tests

src/test/unit/release0100.test.ts

  • the parts follow the stages: body, one comment per person, insights, objectives, decide, close
  • the body says which objectives it addresses, the details, and who is done with counts
  • a retro with no objectives says so
  • each person's comment groups their feedback by type, with the objective
  • insights, objectives (assessment, verdict, override reason), decide and close
  • long parts are split under the limit, by paragraph, each with its own marker and a "continued" line
  • splitMarkdown: paragraphs, then lines, then a hard cut; nothing is lost
  • markers find what is already posted, for this retro only
  • pendingParts skips the posted ones, in order
  • progress lives in flat front matter keys
  • helpers: title, person keys, table cells, placeholders and the upwards section
  • the Archive record comment links the last commit and says how to restore

src/test/unit/githubClient.test.ts

  • Discussions: the settings are read, and a repository without Discussions or without the category is told apart
  • Discussions: creating one sends exactly what GitHub documents, and refusals (no category, too long, Discussions off) are explained
  • Discussions: comments are added and read back over pages, a part is found by its marker over pages, and a deleted Discussion is undefined
  • Discussions: the rate limit (as GraphQL reports it, with HTTP 200), a revoked sign-in and an outage stop the call with a message that says what to do
  • with --write: it creates a label, two issues (closed again) and a Discussion with comments, and every check passes

Integration tests

src/test/suite/extension.test.ts

  • archive and remove through the real GitHub client (a fake GitHub that checks the documented rules): the rate limit midway, resuming, a 70,000-character entry, a failed record comment, a failed git rm, a detached HEAD, and the restore command (#113, #114)
  • 0.10.0: archive a retro to a GitHub Discussion, resume without duplicates, then remove its files safely (#57-#60)

Remove the archived files safely Low

What it does · Impact if it breaks: high · Needs GitHub

Not covered: Git is real in the integration tests (a local bare repository: not committed, no upstream, local changes, not pushed, a detached HEAD, a locked index, running the restore command), but the Discussion is a fake GitHub written from GitHub's documentation, and nobody has run the removal by hand. The record comment failing, git rm failing after it, and retrying are tested; `npm run verify:github` and manual steps 15.8, 15.9, 15.11 and 15.12 have not been run.

Manual script steps 15.8, 15.9, 15.11, 15.12, 19.3: not run by hand yet.

Screenshot runs: Archive (0.11.0).

Tests: 1 unit, 2 integration

Unit tests

src/test/unit/release0100.test.ts

  • the Archive record comment links the last commit and says how to restore

Integration tests

src/test/suite/extension.test.ts

  • archive and remove through the real GitHub client (a fake GitHub that checks the documented rules): the rate limit midway, resuming, a 70,000-character entry, a failed record comment, a failed git rm, a detached HEAD, and the restore command (#113, #114)
  • 0.10.0: archive a retro to a GitHub Discussion, resume without duplicates, then remove its files safely (#57-#60)

Find archived retros Low

What it does · Impact if it breaks: medium · Needs GitHub

Not covered: Not run by hand.

Manual script steps 15.10: not run by hand yet.

Screenshot runs: Archive (0.11.0).

Tests: 3 unit, 1 integration

Unit tests

src/test/unit/release0100.test.ts

  • round-trips, newest first, one record per date
  • bad JSON or bad entries are reported, not thrown
  • @retro learns which retros were archived, and where

Integration tests

src/test/suite/extension.test.ts

  • 0.10.0: archive a retro to a GitHub Discussion, resume without duplicates, then remove its files safely (#57-#60)

@retro chat and language model tools

Ask @retro in Copilot Chat Low

What it does · Impact if it breaks: medium · Needs Copilot

Why this level: No automated test registers or calls the chat participant, and no test reaches the tool loop; only some of the prompt text is unit tested. The Copilot call itself (model choice, consent, the request and the streamed answer) is never run by an automated test: the integration tests switch Copilot off. Only the prompts and the parsing of answers are unit tested. The last real run was on 0.7.1, before the prompts changed in 0.9.0.

Not covered: The participant, its three commands and the tool loop have no automated test. /objectives (0.9.0) has never been run by hand.

Manual script steps 10.1, 10.2, 14.8: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 2 unit, 0 integration

Unit tests

src/test/unit/release090.test.ts

  • @retro /objectives: the prompt has only the chosen objectives, their evidence and verdicts

src/test/unit/release0100.test.ts

  • @retro learns which retros were archived, and where

Integration tests: none.

Let an agent read retros (#retros tools) Medium

What it does · Impact if it breaks: low · Needs Copilot

Not covered: The tools are called directly by the test, not by Copilot, so the way Copilot chooses and calls them is not tested.

Manual script steps 10.3: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 0 unit, 2 integration

Unit tests: none.

Integration tests

src/test/suite/extension.test.ts

  • contributes its language model tools
  • the language model tools return data from the workspace

Let an agent read GitHub activity Low

What it does · Impact if it breaks: low · Needs GitHub and Copilot

Why this level: Only the registration of this tool is tested; it is never called by a test, and the real GitHub client has no automated test.

Not covered: Never called by an automated test.

Manual script steps 10.3: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 2 unit, 1 integration

Unit tests

src/test/unit/insights.test.ts

  • builds search queries scoped to the repo and period
  • classifies issues, PRs and discussions by what happened in the period

Integration tests

src/test/suite/extension.test.ts

  • contributes its language model tools

Diagnostics and the Retros view

Report a problem Medium

What it does · Impact if it breaks: low

Manual script steps 17.1: not run by hand yet.

Tests: 6 unit, 2 integration

Unit tests

src/test/unit/panelUpdates.test.ts

  • the new messages are checked on both sides
  • the environment lines end with each module's facts
  • needs a title and what happened
  • prefills GitHub's issue form (template, title and fields)
  • keeps the link short enough, dropping the oldest log lines first
  • copies the whole report as Markdown

Integration tests

src/test/suite/extension.test.ts

  • Report a Problem (#38) is a form that opens a prefilled GitHub issue; the Output channel logs what happened
  • every command opens its form or picker, whatever VS Code passes from its menus (#22)

Read the log Medium

What it does · Impact if it breaks: low

Not covered: The Show Log command itself is not invoked by a test.

Manual script steps 1.3: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 0 unit, 1 integration

Unit tests: none.

Integration tests

src/test/suite/extension.test.ts

  • Report a Problem (#38) is a form that opens a prefilled GitHub issue; the Output channel logs what happened

Never get a top-of-window prompt Medium

What it does · Impact if it breaks: low

Not covered: The check counts Quick Picks and input boxes only for the commands the test runs.

Manual script steps 1.6: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 0 unit, 1 integration

Unit tests: none.

Integration tests

src/test/suite/extension.test.ts

  • no command opened a Quick Pick or an input box (0.7.0: everything in forms)

Browse retros, feedback, objectives and archived retros in the Retros view Medium

What it does · Impact if it breaks: high

Not covered: The tree's look (icons, ordering by date in very long lists) is only checked by hand.

Manual script steps 1.2, 3.2: run by hand on 0.7.1 (the current version is 0.14.9). Any steps from sections the run did not reach are still to do.

Tests: 4 unit, 3 integration

Unit tests

src/test/unit/treeComposer.test.ts

  • the roots are the modules' roots in order, and the nodes are passed through unwrapped
  • children and tree items go back to the module that returned the node
  • no modules: an empty tree; a node nobody returned has no children and no item
  • a module that returns nothing, or a promise, is fine

Integration tests

src/test/suite/extension.test.ts

  • the Retros view lists the retros
  • the Retros view lists feedback entries, and the edit form updates and deletes them in place
  • registers its commands

Release confidence

Work out the release confidence score Low

What it does · Impact if it breaks: medium

Not covered: The numbers are proposals to calibrate on real releases; use the score as a trial, not as a gate.

No step in the manual script covers this.

Tests: 21 unit, 0 integration

Unit tests

src/test/unit/releaseScore.test.ts

  • the numbers come from catalogue.yml and are checked
  • the worked example in the design reproduces: 75, 50, 39, 25 and 0
  • with each older run counted as shared-only change, the score gives the matrix level for every combination
  • a failing test blocks the activity: score 0, and the reason names the test
  • open gaps take points off by severity, and never more than the maximum
  • a hand-set cap holds the score down and says why; a cap never raises it
  • no test at all: at most 39, even with a passed manual run
  • real-world credit is scaled by the steps that passed and fresh, and is 0 without declared code
  • tests with no result for this commit earn no credit; the explanation says to run them
  • every activity explains itself in one sentence, with what would raise it
  • what would raise it lists the most valuable change first and stops at the next level
  • the release score is the mean weighted by impact: high 3, medium 2, low 1
  • the gates are checked first: a failed test, a high-impact activity at Low, and an unowned change each hold the release
  • without a gate: Go from 80, Go with known risks from 60 to 79, Hold below 60
  • the verdict says in plain words what is holding the release back, most costly first

src/test/unit/releaseEvidence.test.ts

  • fresh results and a run on an older version give the worked example: 25 + 25 + 25 = 75, Medium
  • results for another commit earn nothing, and the notes say so
  • a run that fails a step is the evidence for that step, and a failed test is a gate
  • a run with a commit is compared with the code at that commit; without one, an older version counts as shared code changed, or as changed when strict
  • missing integration results are named, and written gaps count only when asked to
  • the report says the verdict, the score, what costs the most, the notes and every activity

Integration tests: none.

See what changed since the last release Low

What it does · Impact if it breaks: medium

Not covered: The 0.7.1 to 0.11.0 comparison in the design cannot be repeated: the repository history was squashed into one base commit.

No step in the manual script covers this.

Tests: 11 unit, 0 integration

Unit tests

src/test/unit/releaseChanges.test.ts

  • a file is the own code of the activities that list it (two can share a file), shared, ignored, or unowned
  • a change to a file nobody owns is reported as unowned
  • shared code affects the activities it says: all, a list, or (derive) those that reach the file
  • derive without an import graph affects every activity: it may over-warn, it never under-warns
  • a file that is both activity code and shared counts as owned by the activity
  • name-status output: added, modified, deleted, and a rename counts as a delete and an add
  • untracked files come from the status output and nothing else does
  • the files changed since a tag, with their status
  • the newest release tag, by version and not by name
  • the head commit, clean; then dirty, with uncommitted and new files counted as changes

src/test/unit/releaseReport.test.ts

  • a saved run record or report does not make the tree dirty, but any other change does

Integration tests: none.

Record test results and real-world runs against a commit Low

What it does · Impact if it breaks: medium

Not covered: The CI results are kept as an artifact; the dashboard does not download them yet, so a local score has no integration test results until you save the CI file under .nabs/results.

No step in the manual script covers this.

Tests: 15 unit, 0 integration

Unit tests

src/test/unit/releaseResults.test.ts

  • Mocha JSON becomes a result file: passed, failed and skipped per test, with the source file of each
  • output that is not Mocha JSON, or has no tests, is reported in plain words
  • a result file written out reads back the same, and a damaged one says what is wrong
  • results count only for the same commit with a clean tree; another commit counts as not run
  • the evidence for a list of tests counts passed ones and names failed ones; skipped and missing ones earn nothing
  • a test with the same title in another file does not count
  • stale results give no credit and say why
  • every failed test of a fresh result is listed, whether or not an activity lists it
  • a run that lists sections counts every step in them as passed; other steps are not covered
  • a run that lists steps counts only those, with the result it gives
  • the newest run that covers a step decides it, by date
  • how fresh a run is: from its commit when it has one, else from its version
  • is valid YAML that passes the run schema, whatever is in the text
  • lists a section as run only when every step of it was run
  • says what is wrong with the input instead of writing a bad record

Integration tests: none.

See the release confidence dashboard Medium

What it does · Impact if it breaks: medium

Not covered: The Run the Tests button runs only the fixed npm script (a workspace setting must not be able to run any command), so a repository without it cannot run tests from the dashboard. The dashboard does not download the CI results yet. The report that can be shared comes later (#94).

Manual script steps 18.1, 18.2, 18.3, 18.4, 18.5, 18.7, 18.8: not run by hand yet.

Tests: 8 unit, 1 integration

Unit tests

src/test/unit/releaseDashboard.test.ts

  • the dashboard says the verdict and the score first, with the reason in words
  • every activity is listed with its score, its level, where the score comes from and what would raise it
  • a gate is shown as the reason the release is held
  • the changed-code summary names the unowned files, and says so when there is no release tag to compare with
  • anything that comes from the catalogue or a run is escaped
  • no catalogue, or a broken one, says what is wrong and what to do instead of showing a score
  • the tree and Home show one line: the verdict and the score
  • the Release activities of the module are the activities of the catalogue group "release", and the settings and commands match package.json

Integration tests

src/test/suite/extension.test.ts

  • 0.13.0: Release Confidence: no catalogue, then the dashboard, Home card, tree node, Run tests and Record a real-world run (#91, #93)

Record a real-world run in a form Medium

What it does · Impact if it breaks: medium

Not covered: The form lists the script's steps by title only; open the script to read what to do. It cannot edit or delete a recorded run (edit the file).

Manual script steps 18.6: not run by hand yet.

Tests: 4 unit, 1 integration

Unit tests

src/test/unit/releaseDashboard.test.ts

  • the manual script's sections and steps are read from its headings and check boxes
  • both step formats are read, and a line that is not a step is ignored
  • the form has the run details, a choice for each section and one for each step, and escapes the script
  • a step takes its own answer, else its section's; steps that were not run are left out

Integration tests

src/test/suite/extension.test.ts

  • 0.13.0: Release Confidence: no catalogue, then the dashboard, Home card, tree node, Run tests and Record a real-world run (#91, #93)

Share the release report Low

What it does · Impact if it breaks: medium · Needs GitHub

Not covered: Posting a real Discussion is only tested against a fake GitHub (as for archiving); step 18.11 is the real test, and nobody has run it. The badge is a shields.io image address with the score written into it, not a live badge.

Manual script steps 18.9, 18.10, 18.11: not run by hand yet.

Tests: 9 unit, 1 integration

Unit tests

src/test/unit/releaseReport.test.ts

  • it states the verdict, the score, the commit and the tool, and explains the score in plain words
  • every activity is in a table with its score, level, impact and why, and the inputs are listed
  • a gate is the first thing said about why the release is held
  • text from the catalogue cannot break the table or add HTML
  • the badge is one line of text made now, with the score, the verdict, the date and the commit
  • the file name says the version, the day and the commit; the body ends with a marker so it is never posted twice
  • a body too long for a Discussion is shortened to the first thing that would raise each activity
  • the front matter is the history of the score: it reads back, and anything else is not a report
  • the saved reports are read oldest first, and a file that is not a report is skipped

Integration tests

src/test/suite/extension.test.ts

  • 0.14.0: Release Report: Markdown, copy, save in the repository, and post to a Discussion only after the form is confirmed (#94)

Check, score and list gaps of any repository from the command line Low

What it does · Impact if it breaks: medium

Not covered: The first pilot is Number Our Days (#96). The package is built and packed locally but has never been published or installed from a registry, and there is no importable library, only the command. There is no import-graph plugin for languages other than TypeScript (affects derive counts as all), and the Release Report and the dashboard are still VS Code and GitHub only.

No step in the manual script covers this.

Tests: 14 unit, 0 integration

Unit tests

src/test/unit/standaloneCatalogue.test.ts

  • the XML reader handles entities, CDATA, attributes and self-closing elements, and refuses malformed files
  • C# test names are the methods under [Fact] and [Theory], not the helpers
  • a test class is found by its namespace and name, also nested, or by its simple name when only one file has it
  • TRX results become test results by source file and method name; skipped, failed and unknown classes are handled
  • JUnit XML results use the file attribute when there is one, else the class; failure and skipped are read
  • method names drop arguments and namespaces; formats are told apart by extension and first element
  • the version comes from package.json, a JSON field, or a pattern, and is ? when it is not there
  • the bundled schemas are the files in catalogue/schema (run npm run gen:schemas after changing one)
  • a repository with no npm, no schemas, no readiness and no scoring section can be checked, scored and recorded

src/test/unit/catalogueInit.test.ts

  • init scaffolds a catalogue that passes check on a repository that has none, named after its package
  • the scaffold can be scored straight away, and --product and --code decide the name and the files the first activity owns
  • init never overwrites: when any file it would write exists, it writes nothing and says which
  • gaps lists each written gap with its stable id, for one group or as JSON, and says so when there is no catalogue
  • the npm package is the one bundle with a bin entry and no dependencies, and npm pack would publish exactly its files

Integration tests: none.

Tag issues by activity and turn catalogue gaps into issues Low

What it does · Impact if it breaks: medium

Not covered: Closed issues of a gap that is still there, and open issues of a gap that has gone, are only reported, not changed. The issue numbers are not kept in the catalogue.

No step in the manual script covers this.

Tests: 10 unit, 0 integration

Unit tests

src/test/unit/catalogueGitHub.test.ts

  • a tag is a marker comment or the Activities field of an issue form, and nothing else counts
  • an activity gives the label of its group, a group gives its own, and an unknown id is reported and never labelled
  • labels are planned from the tags: untagged items are never touched, labelled ones need nothing, typos are listed
  • the labels command is a dry run unless --apply: it writes nothing, then labels issues and Discussions that have a tag and no others
  • a gap has a stable id, and its issue carries the activity tag and the gap id
  • a gap is matched to its issue by the marker, never the title; a closed gap is only reported
  • gap-issues is a dry run unless --apply, refuses to open more than five without --limit, and never opens the same gap twice
  • gap-issues opens the highest-impact gaps first, and --only chooses gaps by id and refuses ids it does not know
  • an action item tagged #act:<id> makes an issue that carries the tag, so the labels script finds it
  • GitHub is reached with a token or through the GitHub CLI; no token is read by the tool itself

Integration tests: none.