Test Cases for new implementation of NuttX PR Labeling: https://github.com/apache/nuttx/issues/18359

pr-labeler-test-cases.md

Test Cases for new implementation of NuttX PR Labeling

Discussion: apache/nuttx#18359

Arch Labeling

• Simple PR should be labeled correctly and trigger a Simple Build: Arm32-only, Arm64-only, RISC-V-only, Arch + Board, Arch + Board + Doc, ...

  • Arch is Arm32-only: Correctly labeled as Arch: arm. Correctly triggers an Arm32-only Build
  • Board is Arm32-only: Correctly labeled as Board: arm. Correctly triggers an Arm32-only Build
  • Arch is Arm64-only: Correctly labeled as Arch: arm64. Correctly triggers an Arm64-only Build
  • Board is Arm64-only: Correctly labeled as Board: arm64. Correctly triggers an Arm64-only Build
  • Arch is RISC-V-only: Correctly labeled as Arch: risc-v. Correctly triggers a RISC-V-only Build
  • Board is RISC-V-only: Correctly labeled as Board: risc-v. Correctly triggers a RISC-V-only Build
  • Arch + Board are Arm32-only: Correctly labeled as Arch: arm, Board: arm. Correctly triggers an Arm32-only Build
  • Arch + Board are Arm32-only, with Doc: Correctly labeled as Arch: arm, Board: arm, Area: Documentation. Correctly triggers an Arm32-only Build

• Complex PR should be labeled correctly and trigger a Complete Build: Drivers, Include, Arm32 + Arm64, Arm32 + RISC-V, ...

  • Drivers PR: Correctly labeled as Area: Drivers. Correctly triggers a Complete Build
  • Include PR: Correctly labeled as Area: OS Components. Correctly triggers a Complete Build
  • Arch is Arm32 + Arm64: Correctly labeled as Arch: arm, Arch: arm64. Correctly triggers a Complete Build
  • Arch is Arm32 + RISC-V: Correctly labeled as Arch: arm, Arch: risc-v. Correctly triggers a Complete Build
  • Board is Arm32 + Arm64: Correctly labeled as Board: arm, Board: arm64. Correctly triggers a Complete Build
  • Board is Arm32 + RISC-V: Correctly labeled as Board: arm, Board: risc-v. Correctly triggers a Complete Build

• Doc PR should be labeled correctly and trigger a Doc Build only

  • Add 2 Doc files: Correctly labeled as Area: Documentation and triggers Doc Build only

Size Labeling

• Size XS, S, M, L, XL should be labeled correctly

  • Add 10 lines in 1 file: Correctly labeled as Size XS
  • Add 11 lines in 1 file: Correctly labeled as Size S
  • Add 100 lines in 1 file: Correctly labeled as Size S
  • Add 101 lines in 1 file: Correctly labeled as Size M
  • Add 500 lines in 1 file: Correctly labeled as Size M
  • Add 501 lines in 1 file: Correctly labeled as Size L
  • Add 1000 lines in 1 file: Correctly labeled as Size: L
  • Add 1001 lines in 1 file: Correctly labeled as Size: XL

• Added / modified / removed lines in a Single File: Should be labeled correctly

  • Add 11 lines in 1 file: Correctly labeled as Size S
  • Remove 11 lines in 1 file: Correctly labeled as Size S
  • Remove 5 lines and add 6 lines in 1 file: Correctly labeled as Size: S

• Added / modified / removed lines in Multiple Files: Should be labeled correctly

  • Add 10 lines across 2 files: Correctly labeled as Size: XS
  • Add 11 lines across 2 files: Correctly labeled as Size: S
  • Remove 11 lines across 2 files: Correctly labeled as Size: S
  • Remove 11 lines and add 11 lines across 2 files: Correctly labeled as Size: S

• Deleted files should be ignored

  • Add file with 11 lines, delete file with 2531 lines: Correctly labeled as Size: S

Response Time

• PR Labeling must complete within 1.5 minutes

  Why? The Build Workflow begins in the Fetch-Source stage, checking out the Entire Repo and uploading everything in 1.5 minutes, followed by the Select-Builds stage (arch.yml) reading the PR Labels. Before 1.5 minutes, rightfully our workflow_run trigger would have written the PR Labels to the PR.

  • New PR Labeling starts at 6:44:22, ends at 6:44:38. Total 16 elapsed seconds for New PR Labeling.

  • Old PR Labeling starts at 3:46:41, ends at 3:47:16. Total 35 elapsed seconds for Old PR Labeling. (Sometimes 24 seconds)

Zizmor Security Scan

• Zizmor Security Scan should not report any Security Issues. However Zizmor flags workflow_run as a Potential Security Issue, because it's unable to analyse the code inside the workflow.

$ git clone https://github.com/lupyuen6/nuttx
$ cd nuttx

$ zizmor .github/workflows/labeler.yml
🌈 zizmor v1.22.0
 INFO audit: zizmor: 🌈 completed .github/workflows/labeler.yml
No findings to report. Good job! (4 suppressed)

$ zizmor .github/workflows/pr_labeler.yml
🌈 zizmor v1.22.0
 INFO audit: zizmor: 🌈 completed .github/workflows/pr_labeler.yml
error[dangerous-triggers]: use of fundamentally insecure workflow trigger
  --> .github/workflows/pr_labeler.yml:22:1
   |
22 | / on:
23 | |   workflow_run:
24 | |     workflows: ["Pull Request Labeler"]
25 | |     types:
26 | |       - completed
   | |_________________^ workflow_run is almost always used insecurely
   |
   = note: audit confidence → Medium

7 findings (6 suppressed): 0 informational, 0 low, 0 medium, 1 high