summaryrefslogtreecommitdiff
path: root/tools/metrics.py
diff options
context:
space:
mode:
authorhathach <[email protected]>2026-08-21 12:41:47 +0700
committerhathach <[email protected]>2026-08-21 12:41:47 +0700
commite13eff8d4e757ebe7709a58fce44017b8be5a84d (patch)
tree32073e11437e057d92ff843290d3a5594c86d528 /tools/metrics.py
parentf96ddbaa1e11a98f8076df48ba73026191c28399 (diff)
ci: fix nine ways the selection under-selected or mismatched
Every one of these dropped coverage silently - the worst failure mode here, because the PR still goes green. Found by review, each reproduced first. Selection rules: * class_macros derived the config macro from the class DIRECTORY, so a change to src/class/midi/midi2_device.c selected the midi_test examples (which do not compile it) and never examples/device/midi2_device (the only one that enables CFG_TUD_MIDI2, and the only one that does). The file's own macro is unioned in where it differs - union, never replace: over-selecting costs a build, under-selecting merges a break. * the ${FAMILY_MCUS} fallback added for espressif fired on any family whose _family_mcus came back empty, and _cmake_sets is if()-blind and keeps the FIRST definition - so mcx/frdm_mcxn947 answered MCXA15, a token six examples' skip.txt names, dropping 12 firmware images CMake builds. Limited now to families that never spell set(FAMILY_MCUS ...) at all. * lib_examples read only an example's top-level CMakeLists.txt/Makefile; host/msc_file_explorer_freertos names lib/embedded-cli in src/CMakeLists.txt and survived by luck. The whole example tree is scanned. (SEGGER_RTT and rt-thread still resolve to nothing: all three references sit inside a LOGGER=rtt guard no CI build sets - the documented ruling, not a miss.) * get_family_boards applied ci_skip_boards/ci_preferred_boards only under GITHUB_ACTIONS/CIRCLECI, so the selector answered differently on a laptop than on a runner; _prune_buildable forces CI semantics. Its one-board pick also abandoned the whole preferred list when entry one could not build the -e set, and asked skip_example without the build's -D tokens. * _config_enables and lib_examples still read with the locale encoding - under LC_ALL=C the selector tracebacked on three tracked tusb_config.h files. The whole selector and its suite run clean there now. Workflows: * the Membrowse Upload step omitted $EX_ARGS, but --one-first now picks the board from the -e set, so it configured a different, empty build dir and uploaded --identical for a board never compiled. It takes $EX_ARGS for the BOARD; the target stays the aggregate, which has no DEPENDS and still records every example. * blanking FAM_REGEX reset only build_filtered, leaving the build scoped while code-metrics took the UNSCOPED branch and diffed a 1-family run against the full averaged baseline. All three drop together now, as CircleCI's fall-open does. * CircleCI's EX_ARGS had no character screen and is used unquoted, and its code-metrics job still exit 1'd on an empty metrics set - which a scoped build makes a legitimate outcome. * a `ci-full` PR label now turns the scoping off for one PR. A selector bug under-selects silently, and without a label the only ways back to a full matrix are accidental. Performance, since the selector gates every other job: family.cmake texts are read once rather than per changed directory (a 6,000-file dep bump re-read 84 files 99,892 times) and _scrape_mcu is cached: 2.2s -> 0.29s there, 0.8s -> 0.33s on a class diff. Tests: a drift guard for hw/bsp families absent from ci_set_matrix.family_list (they select zero legs now, where they used to ride the full matrix); the rule-4 port test asserted a SUBSET, which set() satisfies, so it could not fail on the empty selection it exists to catch; the GITHUB_ENV guard test counted a SUM of two guards. Drops metrics.py's --only-examples, which nothing called, and applies the TOTAL scrub to the by-example branch that skipped it.
Diffstat (limited to 'tools/metrics.py')
-rw-r--r--tools/metrics.py15
1 files changed, 7 insertions, 8 deletions
diff --git a/tools/metrics.py b/tools/metrics.py
index b97b2b206..27c995954 100644
--- a/tools/metrics.py
+++ b/tools/metrics.py
@@ -83,7 +83,7 @@ def parse_bloaty_csv(csv_text, filters=None):
return {"files": files, "TOTAL": total_all}
-def combine_files(input_files, filters=None, only_examples=None):
+def combine_files(input_files, filters=None):
"""Combine multiple metrics inputs (bloaty CSV or metrics JSON) into a single data set."""
filters = filters or []
@@ -105,9 +105,11 @@ def combine_files(input_files, filters=None, only_examples=None):
# rule and metrics_pair_compare all spell that suffix) - a shape
# sniff would silently reroute any coincidentally-shaped JSON.
for ex in sorted(json_data):
- if only_examples and ex not in only_examples:
- continue
- sub = {'files': list(json_data[ex]['files'])}
+ # same TOTAL scrub the shared path below applies: this branch
+ # `continue`s past it, so do it here or a by-example input keeps
+ # the fake TOTAL rows an ordinary input has stripped
+ sub = {'files': [f for f in json_data[ex]['files']
+ if str(f.get('file', '')).upper() != 'TOTAL']}
if filters:
sub['files'] = [f for f in sub['files']
if f.get('path') and any(x in f['path'] for x in filters)]
@@ -614,8 +616,7 @@ def render_compare_table(rows, include_sum):
def cmd_combine(args):
"""Handle combine subcommand."""
input_files = expand_files(args.files)
- only_examples = set(args.only_examples.split(',')) if args.only_examples else None
- all_json_data = combine_files(input_files, args.filters, only_examples=only_examples)
+ all_json_data = combine_files(input_files, args.filters)
json_average = compute_avg(all_json_data)
if json_average is None:
@@ -673,8 +674,6 @@ def main(argv=None):
help='Sort order: size/size- (descending), size+ (ascending), name/name+ (ascending), name- (descending). Default: size-')
combine_parser.add_argument('--by-example', dest='by_example', action='store_true',
help='Also write <out>_by_example.json: per-example file lists keyed by role/example')
- combine_parser.add_argument('--only-examples', dest='only_examples', default='',
- help='Comma-separated role/example ids to keep when reading by-example JSON inputs')
# Compare subcommand
compare_parser = subparsers.add_parser('compare', help='Compare two metrics inputs (bloaty CSV or metrics JSON)')