Skip to content

[tool] Add summarize_failures script to ci_info - #13722

Open
elliette wants to merge 1 commit into
flutter:mainfrom
elliette:ci-info-summarize-failures
Open

elliette wants to merge 1 commit into
flutter:mainfrom
elliette:ci-info-summarize-failures

Conversation

@elliette

Copy link
Copy Markdown
Member

Follow-up to #13580 which added the ci_info tool and the initial find_builds script.

This PR adds the summarize_failures script which can be used to print out the failures given a BUILD id/s.

Example

Command: script/ci_info/run.sh find_builds --pr 13505 --failed-only

Note: It now tells you the summarize_failures command to run next.

Screenshot 2026-11-19 at 11 08 05 AM

Command: script/ci_info/run.sh summarize_failures 8666496609127595697 8666496609241589041

Screenshot 2026-11-19 at 11 08 17 AM

Work towards flutter/flutter#196131

Pre-Review Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the gemini-code-assist bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.

Footnotes

  1. Regular contributors who have demonstrated familiarity with the repository guidelines only need to comment if the PR is not auto-exempted by repo tooling. ↩ ↩2

@elliette elliette added the CICD Run CI/CD label Nov 19, 2026
@elliette
elliette marked this pull request as ready for review November 19, 2026 19:59
@elliette
elliette requested a review from tarrinneal November 19, 2026 19:59
@elliette

Copy link
Copy Markdown
Member Author

cc @tarrinneal added you as reviewer since you did such a good job reviewing the last one but feel free to un-assign yourself if your PR queue is too much :)

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new tool, summarize_failures.dart, along with its supporting library failure_analysis.dart, to analyze and condense failure logs from LUCI/Buildbucket builds. It also updates find_builds.dart to suggest the failure summarization command for failed builds, adds helper functions to ci_lib.dart for fetching and caching LogDog logs, and includes comprehensive unit tests. The review feedback focuses on performance optimizations, specifically suggesting the reuse of a single http.Client across requests, executing log fetching and build summarization in parallel using Future.wait, and optimizing the loop in extractErrorPackages to avoid redundant line processing.

Comment on lines +239 to +245
Future<Map<String, dynamic>> getBuild(
String buildId,
String fields, {
Duration? cacheAge = const Duration(hours: 1),
}) async {
return prpc('GetBuild', <String, dynamic>{'id': buildId, 'fields': fields}, cacheAge: cacheAge);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To allow reusing a single http.Client across multiple requests (which improves performance by keeping TCP connections alive), add an optional http.Client? client parameter to getBuild and pass it to prpc.

Suggested change
Future<Map<String, dynamic>> getBuild(
String buildId,
String fields, {
Duration? cacheAge = const Duration(hours: 1),
}) async {
return prpc('GetBuild', <String, dynamic>{'id': buildId, 'fields': fields}, cacheAge: cacheAge);
}
Future<Map<String, dynamic>> getBuild(
String buildId,
String fields, {
Duration? cacheAge = const Duration(hours: 1),
http.Client? client,
}) async {
return prpc('GetBuild', <String, dynamic>{'id': buildId, 'fields': fields}, cacheAge: cacheAge, client: client);
}

Comment on lines +16 to +40
Future<Map<String, dynamic>> _summarize(String buildId, int maxLines) async {
final Map<String, dynamic> build = await ci_lib.getBuild(buildId, buildFields);
final Map<String, dynamic> builder =
build['builder'] as Map<String, dynamic>? ?? <String, dynamic>{};

return <String, dynamic>{
'id': buildId,
'number': build['number'],
'status': build['status'],
'builder': ci_lib.stringField(builder, 'builder'),
'url': ci_lib.miloBuildUrl(
ci_lib.stringField(builder, 'project', defaultValue: 'flutter'),
ci_lib.stringField(builder, 'bucket', defaultValue: 'prod'),
ci_lib.stringField(builder, 'builder'),
build['number'] as int?,
buildId,
),
'summary': ci_lib
.stringField(build, 'summaryMarkdown', defaultValue: '')
.replaceAll(RegExp(r'[\r\n]+'), ' ')
.trim(),
'minutes': buildMinutes(build)?.round(),
'steps': await analyzeSteps(buildId, build, maxLines),
};
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Accept an optional http.Client? client parameter in _summarize and pass it down to ci_lib.getBuild and analyzeSteps to enable connection reuse.

Future<Map<String, dynamic>> _summarize(String buildId, int maxLines, {http.Client? client}) async {
  final Map<String, dynamic> build = await ci_lib.getBuild(buildId, buildFields, client: client);
  final Map<String, dynamic> builder =
      build['builder'] as Map<String, dynamic>? ?? <String, dynamic>{};

  return <String, dynamic>{
    'id': buildId,
    'number': build['number'],
    'status': build['status'],
    'builder': ci_lib.stringField(builder, 'builder'),
    'url': ci_lib.miloBuildUrl(
      ci_lib.stringField(builder, 'project', defaultValue: 'flutter'),
      ci_lib.stringField(builder, 'bucket', defaultValue: 'prod'),
      ci_lib.stringField(builder, 'builder'),
      build['number'] as int?,
      buildId,
    ),
    'summary': ci_lib
        .stringField(build, 'summaryMarkdown', defaultValue: '')
        .replaceAll(RegExp(r'[\r\n]+'), ' ')
        .trim(),
    'minutes': buildMinutes(build)?.round(),
    'steps': await analyzeSteps(buildId, build, maxLines, client: client),
  };
}

Comment on lines +143 to +159
final results = <Map<String, dynamic>>[];
for (final String buildId in parsedArgs.rest) {
try {
results.add(await _summarize(buildId, maxLines));
} on Exception catch (e) {
stderr.writeln('BUILD $buildId: failed to summarize: $e');
// Keep going so the other builds are still summarized, but report the
// failure to callers.
exitCode = 1;
}
}

if (parsedArgs[_jsonFlag] as bool) {
print(jsonEncode(results));
} else {
results.forEach(_render);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Optimize the execution by:

  1. Creating a single shared http.Client and closing it in a finally block to reuse TCP connections.
  2. Fetching and summarizing multiple build IDs in parallel using Future.wait instead of sequentially.
  3. Catching any object (including TypeError or CastError) instead of just Exception to prevent the script from crashing on unexpected API payloads.
  final http.Client client = http.Client();
  try {
    final List<Map<String, dynamic>?> resultsWithNulls = await Future.wait(
      parsedArgs.rest.map((String buildId) async {
        try {
          return await _summarize(buildId, maxLines, client: client);
        } catch (e) {
          stderr.writeln('BUILD $buildId: failed to summarize: $e');
          exitCode = 1;
          return null;
        }
      }),
    );
    final List<Map<String, dynamic>> results =
        resultsWithNulls.whereType<Map<String, dynamic>>().toList();

    if (parsedArgs[_jsonFlag] as bool) {
      print(jsonEncode(results));
    } else {
      results.forEach(_render);
    }
  } finally {
    client.close();
  }

Comment on lines +126 to +156
List<String> extractErrorPackages(List<String> lines) {
final entries = <String>[];
for (var i = 0; i < lines.length; i++) {
if (!_errorsHeader.hasMatch(lines[i].replaceAll(_ansi, ''))) {
continue;
}
String? current;
final reasons = <String>[];
for (int j = i + 1; j < lines.length; j++) {
final String candidate = lines[j].replaceAll(_ansi, '').trimRight();
final String stripped = candidate.trim();
if (stripped.isEmpty || stripped.startsWith('See above')) {
break;
}
final int indent = candidate.length - candidate.trimLeft().length;
if (indent >= 4 && current != null) {
reasons.add(stripped);
continue;
}
if (current != null) {
entries.add(_withReasons(current, reasons));
}
current = stripped.endsWith(':') ? stripped.substring(0, stripped.length - 1) : stripped;
reasons.clear();
}
if (current != null) {
entries.add(_withReasons(current, reasons));
}
}
return entries.toSet().toList();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Optimize extractErrorPackages by advancing the outer loop index i to the end of the parsed block (j - 1). This avoids redundantly re-processing lines that have already been parsed by the inner loop.

List<String> extractErrorPackages(List<String> lines) {
  final entries = <String>[];
  for (var i = 0; i < lines.length; i++) {
    if (!_errorsHeader.hasMatch(lines[i].replaceAll(_ansi, ''))) {
      continue;
    }
    String? current;
    final reasons = <String>[];
    int j = i + 1;
    for (; j < lines.length; j++) {
      final String candidate = lines[j].replaceAll(_ansi, '').trimRight();
      final String stripped = candidate.trim();
      if (stripped.isEmpty || stripped.startsWith('See above')) {
        break;
      }
      final int indent = candidate.length - candidate.trimLeft().length;
      if (indent >= 4 && current != null) {
        reasons.add(stripped);
        continue;
      }
      if (current != null) {
        entries.add(_withReasons(current, reasons));
      }
      current = stripped.endsWith(':') ? stripped.substring(0, stripped.length - 1) : stripped;
      reasons.clear();
    }
    if (current != null) {
      entries.add(_withReasons(current, reasons));
    }
    i = j - 1;
  }
  return entries.toSet().toList();
}

Comment on lines +346 to +401
final entries = <Map<String, dynamic>>[];
final targets = <(Map<String, dynamic>, String)>[
for (final Map<String, dynamic> step in failedLeafSteps(build)) (step, 'failed'),
for (final Map<String, dynamic> step in hungLeafSteps(build)) (step, 'hung'),
];

for (final (Map<String, dynamic> step, String kind) in targets) {
final stepName = step['name'] as String;
final String? url = _stdoutUrl(buildId, step);
final entry = <String, dynamic>{'step': stepName, 'kind': kind, 'log_url': url};
entries.add(entry);

if (url == null) {
entry['error'] = 'no stdout log (a recipe/cleanup step; see any hung step below)';
continue;
}

final String text = await ci_lib.fetchLog(url, client: client);
if (text == ci_lib.logMissing) {
entry['error'] = 'log purged or unavailable (LogDog keeps ~6 months)';
continue;
}
if (text == ci_lib.logInaccessible) {
entry['error'] =
'log inaccessible (HTTP 403 from LogDog; the build may be private, '
'or requests are being rate limited)';
continue;
}

final List<String> lines = LineSplitter.split(text).toList();
entry['error_packages'] = extractErrorPackages(lines);

if (kind == 'hung') {
entry['signature'] = extractHang(lines, maxLines);
} else {
final List<String> signature = extractSignature(lines, maxLines);
entry['signature'] = signature;

if (signature.isEmpty) {
final List<String> nonEmpty = lines
.map((String l) => l.replaceAll(_ansi, '').trim())
.where((String l) => l.isNotEmpty)
.toList();
entry['signature'] = nonEmpty
.skip(nonEmpty.length > maxLines ? nonEmpty.length - maxLines : 0)
.map((String l) => ci_lib.truncate(l, 220))
.toList();
entry['signature_is_fallback'] = true;
}
}

entry['cached_log'] = ci_lib.logCachePath(url);
entry['log_lines'] = lines.length;
}

return entries;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To improve performance when a build has multiple failed or hung steps, fetch and analyze their logs in parallel using Future.wait instead of sequentially.

  final targets = <(Map<String, dynamic>, String)>[
    for (final Map<String, dynamic> step in failedLeafSteps(build)) (step, 'failed'),
    for (final Map<String, dynamic> step in hungLeafSteps(build)) (step, 'hung'),
  ];

  return Future.wait(targets.map(((Map<String, dynamic>, String) target) async {
    final (Map<String, dynamic> step, String kind) = target;
    final stepName = step['name'] as String;
    final String? url = _stdoutUrl(buildId, step);
    final entry = <String, dynamic>{'step': stepName, 'kind': kind, 'log_url': url};

    if (url == null) {
      entry['error'] = 'no stdout log (a recipe/cleanup step; see any hung step below)';
      return entry;
    }

    final String text = await ci_lib.fetchLog(url, client: client);
    if (text == ci_lib.logMissing) {
      entry['error'] = 'log purged or unavailable (LogDog keeps ~6 months)';
      return entry;
    }
    if (text == ci_lib.logInaccessible) {
      entry['error'] =
          'log inaccessible (HTTP 403 from LogDog; the build may be private, '
          'or requests are being rate limited)';
      return entry;
    }

    final List<String> lines = LineSplitter.split(text).toList();
    entry['error_packages'] = extractErrorPackages(lines);

    if (kind == 'hung') {
      entry['signature'] = extractHang(lines, maxLines);
    } else {
      final List<String> signature = extractSignature(lines, maxLines);
      entry['signature'] = signature;

      if (signature.isEmpty) {
        final List<String> nonEmpty = lines
            .map((String l) => l.replaceAll(_ansi, '').trim())
            .where((String l) => l.isNotEmpty)
            .toList();
        entry['signature'] = nonEmpty
            .skip(nonEmpty.length > maxLines ? nonEmpty.length - maxLines : 0)
            .map((String l) => ci_lib.truncate(l, 220))
            .toList();
        entry['signature_is_fallback'] = true;
      }
    }

    entry['cached_log'] = ci_lib.logCachePath(url);
    entry['log_lines'] = lines.length;
    return entry;
  }));

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CICD Run CI/CD

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant