Conversations
This page contains an auto-generated copy of discussions happened on various places. The original sources can be seen in scripts/generation/comments.
Main content starts roughly at 2022-09-15.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat.
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
@goderbauer Thanks! I only change 2 characters in a literal String, so I am a bit confused why it the tests are failed... I have looked at the CI logs, but seems no clues (since I am not very familiar with all the infrastructures of Flutter). Could you please give some hints?
There appear to be tests that verify the error message you're modifying. You can probably reproduce this locally by going into the packages/flutter directory and running flutter test there.
@goderbauer Thanks! I will have a look
@goderbauer Hi all checks have passed!
Wait a bit... The flutter-build fails? Hours ago it was all green...
Is that related to my PR or not? When I click "Details" I see a dashboard, which seems to have no relationship to me...
Fix duplicated documentation
Hi thanks for the lib! When reading doc, I find:
If multiple draw commands intersect with the clip boundary, this can result multiple draw commands intersect with the clip boundary, this can result in incorrect blending at the clip boundary.
That sounds like a typo (duplicated doc line) so I fixed it ;)
List which issues are fixed by this PR. You must list at least one issue. N/A but I can create one
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy. no
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above. -> see explanations
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat.
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
No tests needed - it changes the doc only
Fix typo that is introduced in hotfix 2.5.x
Well there is a typo when I examine the hotfix 2.5.x.
Fix #92744
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Fixing bug when ListenableEditingState is provided initialState with a valid composing range, android.view.inputmethod.BaseInputConnection.setComposingRegion(int, int) has NPE
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots. See https://github.com/flutter/flutter/issues/96570
List which issues are fixed by this PR. You must list at least one issue. Fix https://github.com/flutter/flutter/issues/96570
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
This pull request was opened against a branch other than main. Since Flutter pull requests should not normally be opened against branches other than main, I have changed the base to main. If this was intended, you may modify the base back to master. See the Release Process for information about how other branches get updated.
Reviewers: Use caution before merging pull requests to branches other than main, unless this is an intentional hotfix/cherrypick.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat.
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat.
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Well cannot understand that CI error... Have searched keyword ListenableEditingState but no related things. The log seems also unrelated to my code change... Could anyone plz give some hints?
Possibly related @chinmaygarde @darshankawar
The "Linux Unopt" CI error is asking for a change to the code formatting (see https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8824653695614065553/+/u/test:_format_and_dart_test/stdout)
The "build_and_test_linux_unopt_debug" CI error is a flaky test that has been seen before (see https://github.com/flutter/flutter/issues/95751)
@jason-simmons thanks! done
You are welcome!
@chinmaygarde yo, so when can we expect this fix to reach stable?
is there an HF version?
In flutter version "2.10.0-stable" the bug seems not resolved, I've tested with my app.
Fix wrong documentation: There is no LeaderLayer._lastOffset anymore
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.
Seems that LeaderLayer has been refactored (#96144, #96486) and no more _lastOffset exists. So doc oudated.
List which issues are fixed by this PR. You must list at least one issue.
Well this just fix code comments. If you need PR I can create one.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Gold has detected about 1 new digest(s) on patchset 1. View them at https://flutter-gold.skia.org/cl/github/100300
This pull request is not suitable for automatic merging in its current state.
- Please get at least one approved review if you are already a member or two member reviews if you are not a member before re-applying this label. Reviewers: If you left a comment approving, please use the "approve" review action instead.
This pull request is not suitable for automatic merging in its current state.
- The status or check suite Google testing has failed. Please fix the issues identified (or deflake) before re-applying this label.
Fix simple typo: or or
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.
List which issues are fixed by this PR. You must list at least one issue. Seems no need for issue since just typo
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
This pull request is not suitable for automatic merging in its current state.
- Please get at least one approved review if you are already a member or two member reviews if you are not a member before re-applying this label. Reviewers: If you left a comment approving, please use the "approve" review action instead.
Fix FollowerLayer (CompositedTransformFollower) has null pointer error when using with some kinds of Layers
Please see https://github.com/flutter/flutter/issues/100670
List which issues are fixed by this PR. You must list at least one issue. Close #100670
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Going to add tests, wait for a few minutes
Tests added
Fix typo (again)
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots. Just fix a typo (again)... Making a PR and requesting flutter devs to review it sounds resource-consuming, so I wonder whether there any approach that is lighter when I only want to fix a typo?
List which issues are fixed by this PR. You must list at least one issue. none
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
/cc @chunhtai @goderbauer (not sure whom to cc so sorry if disturbing)
Would be great if this could be merged into stable! This is our most occurring issue at the moment
Given my understanding of flutter, seems that we need to wait until next stable (which is roughly 2 months later, unfortunately) if it is not in 2.10. Or, we may make a cherry-pick request
Create ImageFilter.dilate/ImageFilter.erode
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots. Please see https://github.com/flutter/flutter/issues/100830
List which issues are fixed by this PR. You must list at least one issue. https://github.com/flutter/flutter/issues/100830
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
(Test-only) Add tests for new ImageFilter.dilate/ImageFilter.erode in flutter engine
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots. Please see https://github.com/flutter/flutter/issues/100830
List which issues are fixed by this PR. You must list at least one issue. https://github.com/flutter/flutter/issues/100830
Related: https://github.com/flutter/engine/pull/32334
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
This should fail until https://github.com/flutter/engine/pull/32334 is merged and flutter/flutter repo uses the latest flutter/engine.
The error seems to be:
02:50 [32m+259[0m[33m ~6[0m[31m -1[0m: test/html/paragraph/bidi_golden_test.dart: bidi with selection (DOM) [1m[31m[E][0m[0m
TimeoutException after 0:00:30.000000: Test timed out after 30 seconds. See https://pub.dev/packages/test#timeouts
/Users/chrome-bot/.pub-cache/hosted/pub.dartlang.org/test_api-0.4.1/lib/src/backend/invoker.dart 333:9 call
org-dartlang-sdk:///lib/async/zone.dart 1418:39 _rootRun
org-dartlang-sdk:///lib/async/zone.dart 1327:34 run
/Users/chrome-bot/.pub-cache/hosted/pub.dartlang.org/test_api-0.4.1/lib/src/backend/invoker.dart 331:5 _handleError
/Users/chrome-bot/.pub-cache/hosted/pub.dartlang.org/test_api-0.4.1/lib/src/backend/invoker.dart 326:8 _handleError
/Users/chrome-bot/.pub-cache/hosted/pub.dartlang.org/test_api-0.4.1/lib/src/backend/invoker.dart 287:9 call
org-dartlang-sdk:///lib/async/zone.dart 1426:12 _rootRun
org-dartlang-sdk:///lib/async/zone.dart 1327:34 run
/Users/chrome-bot/.pub-cache/hosted/pub.dartlang.org/test_api-0.4.1/lib/src/backend/invoker.dart 286:38 call
org-dartlang-sdk:///lib/_internal/js_runtime/lib/async_patch.dart 131:9 call
org-dartlang-sdk:///lib/_internal/js_runtime/lib/js_helper.dart 1818:14 invokeClosure
org-dartlang-sdk:///lib/_internal/js_runtime/lib/js_helper.dart 1852:1 <fn>
Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'pub run test --chain-stack-traces'.
Thus maybe unrelated to my PR?
By the way, I see 25. Upload artifacts android-arm64-profile in the CI. So I wonder where is the uploaded artifacts? I want to have a try on them to ensure they are really correct when used in my app.
Hi, is there any updates?
There were some CI issues today or yesterday - not sure if they're resolved yet but might be worth just merging up to head and pushing a new commit to kick CI.
[Proposal]Let Flutter run animations at 60fps even if there are heavy widgets, possibly using React Fiber-like or suspend-like algorithm?
EDIT: Design proposal https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko/edit?usp=sharing
Below (folded) are the initial proposal. However, I have realized the initial proposal has many drawbacks, and have raised new proposals. For example, the dual isolate (click to view that comment).
Details
Hi thanks for the framework! As we all know, React Fiber improves the performance and smoothness of React. Currently I am also observing some jank for Flutter app even after optimizing it using the tooltips in official doc, and I do hope there can be something similar to Fiber in Flutter side.
p.s. Some doc about react fiber: https://github.com/acdlite/react-fiber-architecture
I am interested in making contribution when having time as well.
Everything seems green now, what should I do next?
In general, we are moving to a state where all PRs have to use the "Waiting for tree to go green" label to get submitted. I seem to recall a requirement that developers need 2 approving reviews from authorized reviewers to be eligible to be pushed, but I don't see the checks asking for that. I'll add the label and see if it goes in.
If the bot comes along and removes the label, then tag @dnfield to complete a review and then the label should work.
Note that for now the tree is broken (the "luci-engine" failure in the list) so the label is currently in a wait state on that condition.
This pull request is not suitable for automatic merging in its current state.
- Please get at least one approved review if you are already a member or two member reviews if you are not a member before re-applying this label. Reviewers: If you left a comment approving, please use the "approve" review action instead.
if I understand correctly you compare reconciling DOMs to rebuilding the elements in a widget tree and you are proposing to rebuild only certain elements the same way react-fiber prioritize
Possibly not only Widget build, but also layout, paint, etc. Since it is often the case that the layout/paint cost time.
give the developer the ability to set a widgets to low rendering priority
Sounds reasonable.
@flar I see.
@dnfield Hi could you please approve this PR such that it can be merged?
Once the tree is open this should get landed by the bot.
@dnfield Thank you
Hi @fzyzcjy, Thanks for filing the issue. I am quite not sure about the algorithm and its effectiveness. Labeling this issue for further insights from the team.
cc: @dnfield
@maheshmnj Hi thanks for the reply.
I have made an attempt about doing async rendering without modifying Flutter framework: https://github.com/fzyzcjy/flutter_smooth_render But the result is not very interesting - seems that we really need to dig into the framework itself instead of making a wrapper layer around it.
I've been talking about something somewhat like this on the #hackers-framework channel in the past, but it's not a trivial problem to solve. I'd be interested in seeing more details about your designproposal and/or discussing on discord.
And FWIW, this is likely a pretty significant amount of work to do, but there are some people who have already started looking at parts of it @hixie @goderbauer
@dnfield Hi thanks for the reply!
but there are some people who have already started looking at parts of it @Hixie @goderbauer
To avoid reinventing the wheel, I hope to listen to the parts before thinking about what to do next
We know dilate/erode is not implemented for the web platform, just like the compose filter. So what should I do (skip tests when in web?)?
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following UnimplementedError was thrown running a test:
ImageFilter.dilate not implemented for web platform.
When the exception was thrown, this was the stack:
../dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 251:49 throw_
../lib/ui/painting.dart 411:5 dilate
image_filter_test.dart.js 428:133 <fn>
../dart-sdk/lib/_internal/js_dev_runtime/patch/async_patch.dart 84:54 runBody
../dart-sdk/lib/_internal/js_dev_runtime/patch/async_patch.dart 123:5 _async
image_filter_test.dart.js 427:80 <fn>
../packages/flutter_test/src/matchers.dart.js 4522:19 <fn>
/cc @dnfield
Skip these on web with a reference to the bug filed to implement them for web.
Thanks, I will do that.
@fzyzcjy Can you update the tests per the suggestion above to make the checks happy?
Sure. I forgot it
Golden file changes have been found for this pull request. Click here to view and triage (e.g. because this is an intentional change).
If you are still iterating on this change and are not ready to resolve the images on the Flutter Gold dashboard, consider marking this PR as a draft pull request above. You will still be able to view image results on the dashboard, commenting will be silenced, and the check will not try to resolve itself until marked ready for review.
For more guidance, visit Writing a golden file test for package:flutter.
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Changes reported for pull request #101036 at sha c39d7b80811c4903213d9559c764386ab26ada02
@fzyzcjy Can you take a look at the golden images linked in the previous comment to confirm that that's what you expected them to look like? One of the images looks empty, which seems odd...
@goderbauer I guess it is right, just b/c it erodes too much. I have updated the test so it will be less strange.
Golden file changes are available for triage from new commit, Click here to view.
For more guidance, visit Writing a golden file test for package:flutter.
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Changes reported for pull request #101036 at sha e3d3659c4911cfa6caa84c6c3e5ec0b6cf69b146
The gold looks better. (Notice it has very very thin lines)
Thanks, @fzyzcjy
@dnfield or @flar could you review this one (and also approve the goldens) since you reviewed the upstream PR in the engine?
Thanks, @fzyzcjy
@dnfield or @flar could you review this one (and also approve the goldens) since you reviewed the upstream PR in the engine?
I still don't see anything rendered in the erode test even if I click on it to see it full size. Am I clicking on the wrong link?
Actually, it looks like I was reviewing the goldens from before I requested a larger stroke width. Maybe they haven't been updated yet?
I still don't see anything rendered in the erode test even if I click on it to see it full size. Am I clicking on the wrong link? Actually, it looks like I was reviewing the goldens from before I requested a larger stroke width. Maybe they haven't been updated yet?
I guess the CI is even not executed after modifying strokewidth...
I guess the failure (and seems skipped executing all tests related to golden) is not caused by me. So maybe need to wait
I guess the failure (and seems skipped executing all tests related to golden) is not caused by me. So maybe need to wait
I mentioned it on the infra Discord chat to see if someone can poke it.
I would just LGTM to let it merge, but I'm worried it may merge with bad goldens and cause problems down the line.
Take your time. Since this is a test-only PR I am not that urgent :)
It looks like you need to rebase in order to get the ci.yaml to gel with the latest CI recipes and then hopefully final goldens will be generated.
Golden file changes are available for triage from new commit, Click here to view.
For more guidance, visit Writing a golden file test for package:flutter.
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Changes reported for pull request #101036 at sha 692b2715bd2b0cc9b076228b007c7506cc7d25e6
Now seems better
Where the goldens for this approved? It looks like this is breaking the build:
The following _Exception was thrown while running async test code:
Exception: Skia Gold received an unapproved image in post-submit
testing. Golden file images in flutter/flutter are triaged
in pre-submit during code review for the given PR.
Visit https://flutter-gold.skia.org// to view and approve
the image(s), or revert the associated change. For more
information, visit the wiki:
https://github.com/flutter/flutter/wiki/Writing-a-golden-file-test-for-package:flutter
Debug information for Gold:
stdout: Given image with hash 94028cfacbe41a55af47edcddeff6540 for test widgets.image_filter_erode
Untriaged or negative image:
https://flutter-gold.skia.org/detail?test=widgets.image_filter_erode&digest=94028cfacbe41a55af47edcddeff6540
stderr: Test: widgets.image_filter_erode FAIL
When the exception was thrown, this was the stack:
#0 SkiaGoldClient.imgtestAdd (package:flutter_goldens_client/skia_client.dart:216:7)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)
Can we just approve the new goldens? If not I will need to revert this to get the tree green again.
I do not have permission to approve the goldens, but seems that @flar has said that the goldens are ok, and he may have permission to approve
It looks like it has been resolved and the tree is green again, so it's all good now.
I did approve the goldens, so I'm not sure what happened.
Create takeExceptionDetails which is similar to takeException but allows the user to know details (e.g. stack trace)
Please see https://github.com/flutter/flutter/issues/103487
Close #103487
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
What's the use case for needing that? What can you test now that you weren't able to test before?
What's the use case for needing that? What can you test now that you weren't able to test before?
Not to test indeed, but to inform. Firstly, in https://github.com/fzyzcjy/flutter_convenient_test readme, there is a 1-minute video demonstrating how this open source lib looks like. Then you see this lib shows errors etc in the command panel in an intuitive way for the programmers. When talking about "errors", surely we want to show stack traces as well, but currently for errors like golden mismatch I have to use takeException and stacktrace is lost.
Where this lib needs it: https://github.com/fzyzcjy/flutter_convenient_test/blob/f9d38c9fd870a5dab5a61755f66214b29caadf2f/packages/convenient_test_dev/lib/src/functions/command.dart#L99
👀
Still get the error with:
Flutter 2.10.4 • channel stable
Tools • Dart 2.16.2 • DevTools 2.9.2
Specifically when I double-tap to select the text in a text field.
It is a little odd to expose that here and may lead to confusion in the API. Have you considered just overwriting FlutterError.onError in your framework yourself to catch the error directly?
Well I hope to reuse the logic about Flutter errors in testing...
(Triage) Hi @fzyzcjy, does @goderbauer's suggestion suit your needs? I do not know that we want to merge this given the previous comments.
@Piinks Hi that does not suit. Btw FlutterError.onError cannot be rewritten outside a widgetTest since widgetTest overrides it again.
FlutterError.onError cannot be rewritten outside a widgetTest since widgetTest overrides it again
I think that is by design. You can override it for the given test, or even use setUp to override it for a group of tests.
even use setUp to override it for a group of tests.
That seems not possible, surprisingly. Because FlutterError.onError is overrided in TestWidgetsFlutterBinding._runTest, which seems to be run after all setup?
Oh right. That is true! It looks like we usually set this in the given test.
Yes, so it is not very convenient as we cannot do it in setup
We need to come up with another solution for this that doesn't require adding confusing API. This will need some more thinking. I suggest you describe what you want to achieve in an issue without proposing a concrete solution just yet.
Remove workaround since #66006 is fixed
The comments say "@jiahaog Remove when https://github.com/flutter/flutter/issues/66006 is fixed." and that issue is already fixed
List which issues are fixed by this PR. You must list at least one issue. #66006
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Thank you! Can you take a look at the test failures? I think you need to remove https://github.com/flutter/flutter/blob/1add0d790d78918a560387c23802afd2979d15d7/packages/integration_test/test/binding_test.dart#L126-L131
You are right. I have removed them. (I forgot to check the CI just now...)
@jiahaog Now tests seem to pass
only remove code (no modified or added lines) to remove a feature or remove dead code.
So seems it is test-exempt
I ran a global presubmit (tap/458352815) for this change. This will require a g3fix internally.
Actually I'm not really sure why removing the repaint boundary results in a significant resize of the screenshot goldens. @dnfield can you take a quick look at shortn/_z3wWrukdZc?
What is tap/458352815 and shortn/_z3wWrukdZc...? Is it some internal links that I cannot view?
Oh don't worry about it, yes it's an internal system for running tests :)
From what I recall, this needed some changes internally to pump widgets - cl/392929256
Ah thanks, I didn't realise that the fix (PR https://github.com/flutter/flutter/pull/88609) for issue https://github.com/flutter/flutter/issues/66006 was actually reverted in PR https://github.com/flutter/flutter/pull/88889 so the issue shouldn't be closed, and I completely forgot all about the discussion we had previously regarding this.
Checking through internal tests failures again, it seems like this change causes some test to be especially flaky. This is not the same issue which caused it to be reverted though. I think the next step would be for us to investigate further to find out what is wrong with that test. We can track that on https://github.com/flutter/flutter/issues/66006, which I just reopened. Sorry about that!
Looks like this fix has not been included in stable channel yet. Does anybody know where it was included?
@gonzalonm May I know where you see it not in stable? from the release date it should have been stable
I didn't see it in the release notes. Anyways, I could not reproduce the issue with version 3.0.0, so in that version it has been fixed.
Thanks!
It may be in release notes of engine, while flutter/flutter repo (not flutter/engine) will not say this
Fix that RenderEditable (TextField) ignores offset in painting, making text selections shifted when offset is nonzero
I will start adding tests once flutter team thinks this PR is acceptable, since I am not sure whether the team like to fix it, and if the team does not like it I will not waste time on it :)
When implementing RenderBox.paint, we know we should respect the offset parameter. However, even though _paintContents respects it, the _paintHandleLayers does not - instead it completely ignores the offset parameter. This is logically wrong as it does not respect the definition and requirements of the function.
Luckily (or unluckily), currently it does not have visible harm yet. This is because RenderEditable is used by _Editable and future by EditableText. And, the _Editable's ancestors widgets do not provide any offset (because the widgets in the same Layer paint child without shifting). Thus, offset parameter is always zero in the current version of Flutter.
However, I am still proposing this PR because of the following reasons:
- It is logically wrong (violates definition of
paint), so by definition it is a bug and we should fix it. - It may be a visible bug in future releases. With adding more functionality to text fields, surely we cannot guarantee RenderEditable's ancestors always give a zero offset. (If we think so, I should immediately raise another PR with
assert(offset==Offset.zero)to enforce it.) - Users who need deep customization may directly use
RenderEditable(since it is public), and even copy some of the source code and hack it (e.g. me). With their customizations, it is highly possible that the offset is no longer always zero, then they will find suddenly the text selections shifted weirdly. (Disclaimer: That is my case, and why I find this bug)
I have not come up with a way to make tests about this, because _Editable is private, and as mentioned above, it is logically wrong but not visible now.
However, this PR does not add any new logic, so making existing tests pass IMHO may be sufficient.
Close #109289
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
P.S. https://discord.com/channels/608014603317936148/608018585025118217/1006808038914654218
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
@justinmc Hi, as discussed by @Hixie at here:
@Hixie : seems fine to me but @Justin McCandless (justinmc) is the person we should really ask
So I wonder whether this PR is ok for you? If so I will spend time adding tests.
As discussed on Discord, the approach in this PR looks good to me. Please tag me or request a review from me when you finish the tests.
@justinmc Hi, is there any updates?
gradlew/gradlew.bat should not be gitignored according to official doc
Close https://github.com/flutter/flutter/issues/109749
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
This is the repo's gitignore, not a template file gitignore.
I know. The problem is, maybe flutter repo itself also need to follow gradle's official suggestion?
We've done this historically because the tool can handle re-generating this and to avoid churn in the repo when the contents of this file changes.
I'm going to close this PR, as we don't intend to accept this change. If something changes we can always reconsider
I see. Thanks all the same
Fix Image's logical flow which disposes its image too early, causing errors such as "Cannot clone a disposed image"
Close #110129
Please see the issue for paragraphs about the bug cause and fix etc
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
👀 Any updates here after two months? "workaround" does not sound very good so hoping it can be removed
The test, without the fix, fails. This verifies the PR is needed.
══╡ EXCEPTION CAUGHT BY WIDGETS LIBRARY ╞═══════════════════════════════════════════════════════════
The following StateError was thrown building KeyedSubtree-[#c5088]:
Bad state: Cannot clone a disposed image.
The clone() method of a previously-disposed Image was called. Once an Image object has been
disposed, it can no longer be used to create handles, as the underlying data may have been released.
When the exception was thrown, this was the stack:
#0 Image.clone (dart:ui/painting.dart:1808:7)
#1 RawImage.createRenderObject (package:flutter/src/widgets/basic.dart:5951:21)
#2 RenderObjectElement.mount (package:flutter/src/widgets/framework.dart:5744:52)
... Normal element mounting (10 frames)
#12 Element.inflateWidget (package:flutter/src/widgets/framework.dart:3883:16)
#13 Element.updateChild (package:flutter/src/widgets/framework.dart:3606:20)
#14 ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:4920:16)
#15 StatefulElement.performRebuild (package:flutter/src/widgets/framework.dart:5060:11)
#16 Element.rebuild (package:flutter/src/widgets/framework.dart:4617:5)
#17 BuildOwner.buildScope (package:flutter/src/widgets/framework.dart:2687:19)
#18 AutomatedTestWidgetsFlutterBinding.drawFrame (package:flutter_test/src/binding.dart:1244:19)
#19 RendererBinding._handlePersistentFrameCallback (package:flutter/src/rendering/binding.dart:374:5)
#20 SchedulerBinding._invokeFrameCallback (package:flutter/src/scheduler/binding.dart:1175:15)
#21 SchedulerBinding.handleDrawFrame (package:flutter/src/scheduler/binding.dart:1104:9)
#22 AutomatedTestWidgetsFlutterBinding.pump.<anonymous closure> (package:flutter_test/src/binding.dart:1093:9)
#25 TestAsyncUtils.guard (package:flutter_test/src/test_async_utils.dart:71:41)
#26 AutomatedTestWidgetsFlutterBinding.pump (package:flutter_test/src/binding.dart:1079:27)
#27 WidgetTester.pump.<anonymous closure> (package:flutter_test/src/widget_tester.dart:618:53)
#30 TestAsyncUtils.guard (package:flutter_test/src/test_async_utils.dart:71:41)
#31 WidgetTester.pump (package:flutter_test/src/widget_tester.dart:618:27)
#32 main.<anonymous closure> (file:///b/s/w/ir/x/w/flutter/packages/flutter/test/widgets/image_test.dart:83:20)
<asynchronous suspension>
<asynchronous suspension>
(elided 5 frames from dart:async and package:stack_trace)
════════════════════════════════════════════════════════════════════════════════════════════════════
02:21 +3329 ~2 -1: /b/s/w/ir/x/w/flutter/packages/flutter/test/widgets/image_test.dart: Verify Image does not use disposed handles [E]
Test failed. See exception logs above.
The test description was: Verify Image does not use disposed handles
══╡ EXCEPTION CAUGHT BY WIDGET INSPECTOR ╞══════════════════════════════════════════════════════════
LGTM once tests are passing :)
What should I do now 👀
Sorry, thought I had added the label!
- Please get at least one approved review if you are already a member or two member reviews if you are not a member before re-applying this label. Reviewers: If you left a comment approving, please use the "approve" review action instead.
Validations Fail.
@dnfield That's OK. Oops seems to need one more reviewer
Improve performance by removing redundant null checks on hot paths
As we know, RenderObject.constraints field is called frequently, because every performLayout usually reads that. However, by looking at the code, it seems that it is having a redundant null check, which reduces the performance.
List which issues are fixed by this PR. You must list at least one issue. Close https://github.com/flutter/flutter/issues/110764
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Did you run any benchmarks by any chance to verify if this has any effect?
@Hixie I have done some further experiments: Look into its assembly code. The Dart compiler is much smarter than I thought before - it generates completely the same assembly!
https://godbolt.org/z/rKo77K5vn
@fzyzcjy Any update please? We are also interested in this feature.
@wangying3426 Well, no updates from me since I want to firstly listen to the "who have already started looking at parts of it @Hixie @goderbauer"
@fzyzcjy thanks for looking that deeply! That's awesome news.
😄
Add my name to authors list
Please see https://github.com/flutter/flutter/pull/90413#issuecomment-1245628544 for details :)
I have made a few merged PRs, including several bug fixes in flutter/flutter and also a little feature in flutter/engine as well.
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.
List which issues are fixed by this PR. You must list at least one issue.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
auto label is removed for flutter/flutter, pr: 111522, due to - Please get at least one approved review if you are already a member or two member reviews if you are not a member before re-applying this label. Reviewers: If you left a comment approving, please use the "approve" review action instead.
auto label is removed for flutter/flutter, pr: 111522, due to Validations Fail.
Link debugVisitOnstageChildren in Offstage
The Offstage has strong relationship with debugVisitOnstageChildren: When using Offstage widget, the widget subtree of debugVisitOnstageChildren is changed, and widget testers will not find an offstage widget. In addition, when we are talking about the word "offstage", we may need to refer to both pages, because they have slightly different meanings.
This origins partially from https://github.com/flutter/flutter/pull/111479#issuecomment-1245534877. Indeed, when talking about "offstage widgets", I quickly remembered and opened Offstage page, but never realize there is another page about debugVisitOnstageChildren, which even has a bit different explanation from Offstage. This PR fixes this problem.
List which issues are fixed by this PR. You must list at least one issue.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Hmm seems to need one more PR approval...
@Hixie @goderbauer @dnfield How do you think about this fiber proposal?
No one has come up with a workable proposal at this point in time. I think it's worth doing but it's not my top priority at the moment.
but it's not my top priority at the moment.
As mentioned earlier, I am willing to PR and contribute. But surely need some suggestions and discussions prior to start implementing :)
Btw I am not thinking about strictly implementing Fiber, since web model is not the same as Flutter model, but something inspired by it that can make our animations smoother.
@Piinks green now :)
but it's not my top priority at the moment.
As mentioned earlier, I am willing to PR and contribute. But surely need some suggestions and discussions prior to start implementing :)
Btw I am not thinking about strictly implementing Fiber, since web model is not the same as Flutter model, but something inspired by it that can make our animations smoother.
I think so. Maybe we can create a document and then a detailed description
- the fiber node archive and it can be interrupted by reconciler
- how does the Fiber reconciler work
- how does the flutter framework need to do and how to design
How do you think about it? @fzyzcjy
@JsouLiang Maybe we can create a document and then a detailed description
Sure! Maybe we can firstly discuss about it (maybe just here? - just like how I have seen many Dart/Flutter design discussions happen) and a detailed doc after we draw a (draft) conclusion
how does the flutter framework need to do and how to design
Btw, fiber can make animations smoother, but if I understand correctly, the smoothness is because that specific animation is driven by css, not js. This is contrary to flutter. For example, a CircularProgressIndicator, or even a scrolling of ListView, is driven by Dart code. Thus, we cannot easily say "let's give control to flutter engine / android / ios / whatever once in a while when we are doing build/layout/paint/whatever". If we simply do so, we will not get a smooth animation automatically. Instead, we may need to find out a more sophisticated approach.
but if I understand correctly, the smoothness is because that specific animation is driven by css, not js.
Yes, the CSS animation is driven by css, not through js That mean, the CSS associated with the HTML element can calculate the animation difference directly, without going through JS. @fzyzcjy
Yes, that is why fiber is so useful. Indeed it is like, the web ui is driven by two things - the JS and CSS. Fiber pause JS once in a while so CSS things can come in and animate.
@fzyzcjy Btw, fiber can make animations smoother, but if I understand correctly, the smoothness is because that specific animation is driven by css, not js. This is contrary to flutter. For example, a CircularProgressIndicator, or even a scrolling of ListView, is driven by Dart code. Thus, we cannot easily say "let's give control to flutter engine / android / ios / whatever once in a while when we are doing build/layout/paint/whatever". If we simply do so, we will not get a smooth animation automatically. Instead, we may need to find out a more sophisticated approach.
As you say, Web animation like css animate, Android ViewPropertyAnimator (maybe iOS also has similar animate mechanism), they all are driven by browser/system, but in Flutter it is driven by ourselves with all other business logic.
for more, Android's window transition animation is driven by WindowService seperately
@JsouLiang @xanahopper
So, it is possible we come up with something slightly different?
For example, if we want to make some animations faster, like CircularProgressIndicator and ListView-scrolling, is it possible to do the following: We give CircularProgressIndicator high priority, and it must be layout/paint at 60fps. In the meanwhile, all other widgets will run one layout/paint across multiple frames with suspending just like what Fiber does. In other words, when vsync comes, CircularProgressIndicator will do all the layout/paint job, while other widgets will continue working on its layout/paint but will pause once it is near 16ms. Then, we can see CircularProgressIndicator smooth at 60fps, while other widgets having similar rendering speed as before.
Btw, some side remarks that is less like Fiber: Here is a tool that defers Widgets from being built https://github.com/LianjiaTech/keframe. But I guess we can make it more fine-grained and with more improvements since we are going to modify the flutter framework itself. For example, (very rough draft idea), can we modify the layout phase (or paint, or others), such that it pauses layouting the remainder (and will do it in the next frame), and let painting and other phases go first?
The hard part of all of this is to figure out how to do it without breaking existing Framework code.
I think it's probably possible, but it's not easy.
@dnfield without breaking existing Framework code
We are allowed to modify anything in Flutter, don't we :) Just not allowed to break existing API that is used by flutter users.
Then, maybe we can have a feature flag?
@fzyzcjy Yes, we all farmilar with KeFrame and has already applied some optimize like it.
I guess we can make it more fine-grained and with more improvements since we are going to modify the flutter framework itself.
I think this may the point we are going to discuss.
The hard part of all of this is to figure out how to do it without breaking existing Framework code.
I think it's probably possible, but it's not easy. @dnfield we cannot just stop because is not easy. if it is a right way to improve it.
we cannot just stop because is not easy. if it is a right way to improve it.
Same here :) I like challenging, i.e. exciting, work!
@fzyzcjy I'm interested in this topic and have been trying to go in a direction where Keframe can make the best use of each 16.7ms, since now each item will take the full 16.7ms (even though it may only take 1ms on some good devices). I'm trying to count the time taken by individual items to determine how many items should be rendered in the next frame.
Continue from the animation proposal above, with @dnfield's "without breaking existing Framework code":
Maybe we can have a global flag, say, bool enableFiber = false. By default it is false, so users can use existing API freely without any change. When user manually set it to true, our new feature runs.
The API may be as simple as a Widget, say, HighPrioritySubTree(builder: (context, child) => build_your_subtree_here, child: put_static_child_here), just like animation builder widgets. That builder should wrap the CircularProgressIndicator in the example above. We may also add a CancelHighPrioritySubTree if needed. For example, when scrolling ListView, we may want the scrolling animation be at 60fps, while we have to accept that a big widget in ListView is slow to build. Then, we may wrap ListView with HighPrioritySubTree, and each child of ListView with CancelHighPrioritySubTree. By doing so, our ListView will be forcefully built at each frame, while its contents will be stale for a few frames.
@Nayuta403 I have had similar thoughts before. The problem is, build phase is not the most costly one. There are layout and paint phase, etc, as well. What's worse, Flutter has C++ engine code which rasterizes and flush to the screen. That one can take a long time in some cases (for example, in my own app, when there is a ton of bezier curves). A widget may, for example, have very short build phase time but very long C++ rasterize time.
The API may be as simple as a Widget, say,
HighPrioritySubTree(builder: (context, child) => build_your_subtree_here, child: put_static_child_here), just like animation builder widgets. That builder should wrap the CircularProgressIndicator in the example above. We may also add aCancelHighPrioritySubTreeif needed. For example, when scrolling ListView, we may want the scrolling animation be at 60fps, while we have to accept that a big widget in ListView is slow to build. Then, we may wrap ListView with HighPrioritySubTree, and each child of ListView with CancelHighPrioritySubTree. By doing so, our ListView will be forcefully built at each frame, while its contents will be stale for a few frames.
@fzyzcjy I agree with switch flag, but some individual widget may still look verbose. I'd rather like to add a optional parameter to base class Widget to specify it's build/layout/render priority.
Than change default page transition widget, scrollable container to high priority and wrap its content to low priority.
And here is another case may need to be consider: the list. in general container such as page, content size has no effect with container and other siblings, but things are different in list. if we have different size of different item, we cannot just show placeholder with same size, scrolling when and after content item is building/layouting may cause a sudden change in list.
I'd rather like to add a optional parameter to base class Widget to specify it's build/layout/render priority.
That sounds good to me. With that flag, we can also very easily create the widgets I mention. Just like the repaintBoundary is a flag and we create a widget to set it.
if we have different size of different item, we cannot just show placeholder with same size, scrolling when and after content item is building/layouting may cause a sudden change in list.
That's true. keframe workaround by letting the developer specify a placeholder size manually. But surely, for complex list items, we can never predict the size in advance so it still "jumps" when real content loads.
Maybe this is inevitable, and we have to live with it? Or, maybe we just place background color on those non-built entries?
we have to live with it, but we can give some different solutions, such like allow jumps, background color or some other...
I remember that iOS has very high priority with scrolling. If we can get item's size before build, this may not be a problem.
pre-measure for many things is possible but we have two considerations:
- it cannot block UI/main thread otherwise it means nothing
- it should has slice cost for developer to do that
this may conflict with principle of Flutter for single pass measure……but I think it has already has many cases in practice against that, it may not be a big deal.
@xanahopper I am not sure whether that is another isolated problem, or we can directly solve it within our proposal about this issue. For example, if we are to add a pre-measure phase, we may add computeSomething in addition to existing computeLayout, computeDryLayout etc, and that may be orthogonal to this issue.
Btw, I suspect whether pre-measure can happen before build phase, since we even do not know the widget tree then. Maybe it can happen before layout phase?
@fzyzcjy reply to @Nayuta403 I have had similar thoughts before. The problem is, build phase is not the most costly one. There are layout and paint phase, etc, as well. What's worse, Flutter has C++ engine code which rasterizes and flush to the screen. That one can take a long time in some cases (for example, in my own app, when there is a ton of bezier curves). A widget may, for example, have very short build phase time but very long C++ rasterize time.
Just to make it a little bit more detailed: On the contrary, if my proposal above works, the following may happen -
- Animations are in perfect 60fps, since low-priority job auto pause when near timeout. If we use keframe or similar solution, and give too many widgets in one frame, our animation will stuck.
- No cpu cycles are wasted, because we will never early-pause but will only pause when near timeout. For example, suppose widget A needs 160ms to build+layout+paint+raster, then it will be done in (roughly) 10 frames. If we use keframe or similar solution, and give too little widgets in one frame, we are wasting cpu cycles.
- It avoids our need to measure, or guess, the time needed for a widget in build/layout/paint/raster phase. Just as I mentioned above, I personally find it hard to guess how long a widget will need in those phases, especially raster phase which is C++ and varies greatly on different CPU/GPUs (different phones).
- It is OK to have a non-separable widget that is heavy in one phase.
- It is automatic and declarative. Programmers only need to specify priorities and that's all.
Btw I also like your (@Nayuta403) keframe solution :) Just trying to propose something that we can make flutter even better
@fzyzcjy Thank you, I think we all want to make Flutter better. ❤️
So I think of a few problems we might have to solve:
- How to get the current UI cost, I think we still need to know this information even if we put the animation in the high priority queue, so that we can determine when the low priority task should end.
- How does the ListVIew item handle sliding when there is no width and height information
- How Fiber builds interruptible. It might be a little easier for a ListVIew, because its items are siblings. But what about parent-child nodes like Container?
More thoughts here.
1. For CircularProgressIndicator, or high-priority widgets without low-priority children
A very draft idea:
We may have multiple sub-trees, i.e. have a forest, in flutter. In this example, CircularProgressIndicator may be subtree 1, and everything else may be subtree 2. The subtree 1 goes through build/layout/paint/raster etc for each frame, and subtree 2 may go slowly, i.e. suspend.
Suppose it needs 10 frames for subtree 2 to finish the whole build/layout/paint/raster process. Then, we just allow all inconsistent and dirty states to exist during that 10 frames. For example, a node may have several layouted children and several un-layouted children. Same goes for rasterizing etc. We also need to ensure nobody can mutate the state accidentally when they are dirty.
In addition, I think we may not need to add this suspend feature to the build phase, but only add to layout/paint/raster if possible, contrary to React. This is because, if the time-consuming operation is only at build phase, keframe or similar solutions should already work. It may be deep in the rendering pipeline that makes this proposed method more interesting.
Surely this is just a draft and brainstorm, and I am willing to hear any thoughts!
2. For ListView scrolling problem, or high-priority widgets with low-priority children
The problem is, those big low-priority children may need a lot of frames (say 10 frames) to build/layout/paint/rasterize, and during those 10 frames, their internal data structure are not ready for use. For example, we cannot let it to paint at 5th frame, because its layout tree (or layer tree or something like that) may have a child that has been layouted and another child that has not yet been layouted.
However, we are doing nothing but scrolling. Then what about simply raster cache the screen, and scrolling is nothing but shifting this ui.Image. More details can mimic this PR: https://github.com/flutter/flutter/pull/106621 In that PR, during a "zoom page transition", no real widgets are built in each frame. Instead, a ui.Image snapshot is taken in the first frame, and during the whole transition we are just zooming that Image. Our solution is different from #106621, though. In that PR, no work is done during the whole page transition, but in our case, we can perform useful build/layout/paint/raster in the remaining time of each frame.
This solution also has some spirit similar to React Fiber: In Fiber, our JS-driven DOM elements are freezed indeed, and it is the CSS animation that still works. In our case, the "scrolling ui.Image" is a bit mimic a scrolling CSS animation.
How to get the current UI cost, I think we still need to know this information even if we put the animation in the high priority queue, so that we can determine when the low priority task should end.
Seems we do not need? We just blindly run whatever should be done next, and suspend when we are near 16ms.
We do need to let the the high priority job (say CircularProgressIndicator or ListView-scrolling) finish within the totally 16ms though.
My first thought is that, it would be best if we execute all phases of this subtree first, and then execute (and suspend when timeout) all phases of the second subtree in whatever time remain. Then we never need to get the cost.
If that is impossible, I wonder whether we can use some heuristics. We all know a CircularProgressIndicator should be very lightweight, so is a ListView-scrolling (if using the ui.Image approach above). We may also learn from the history.
How does the ListVIew item handle sliding when there is no width and height information
If using the approach mentioned above, it will just be blank. But not blank whenever there is a scrolling! Because we know Flutter has some cache extent for ListViews, we can also capture those cached extents in our ui.Image snapshot. Then, only if the following happens, we will see blank:
- The user scrolls so much that all cache extent are used up
- Our heavy widgets are so heavy that it even does not finish one frame up to now
How Fiber builds interruptible. It might be a little easier for a ListVIew, because its items are siblings. But what about parent-child nodes like Container?
As a very rough draft, I am considering yield. For example:
Iterable<void> performLayout() sync* {
yield* myFirstChild.layout();
some_computation_here;
yield* mySecondChild.layout();
}
Each yield point is suspendable.
IIRC, Redux Saga https://redux-saga.js.org/docs/introduction/BeginnerTutorial/ uses something similar to this.
Have not digged into React Fiber's source code yet. Have you checked it, how does it implement it?
But as I am not an expert in Dart compiler implementation, I am not sure about the performance penalty. (Hope it to be tiny!)
A few pointers:
- We cannot use sync generators, they create code that is large and slow.
- A good canonical case here would be something like https://github.com/flutter/flutter/blob/master/dev/benchmarks/macrobenchmarks/lib/src/list_text_layout.dart. This ends up being janky because layout gets expensive for all that text (on a lower end phone it can easily take 20-30+ms just to layout all the text there, and the ListTile is a little deceptive because Material introduces expense - this is the kind of thing we want to figure out how to break up "automatically").
- We should probably worry about prioritization of jobs until after we figure out how to sensibly budget and interrupt layout/painting/compositing. It doesn't matter what priority we'd want to give things if we can't do that, and it will probably be hard to come up with a good/fair prioritization scheme.
@dnfield Thanks for the ideas!
how to sensibly budget and interrupt layout/painting/compositing.
Quick answer to budget: As suggested in my comments above, we may not need to budget things (unlike the keframe-like solution). We just run the high-priority subtree (one with animation) until it finishes, and then run low-priority heavy subtree until whenever timeouts.
Animations might not ever finish.
And you might be animating the entire screen, e.g. for a route transition
Animations might not ever finish.
Well, I mean, run its build+layout+paint+raster fully instead of partially, not wait until there is no animations at all. For a CircularProgressIndicator it may take, say, <1ms. The rest 16.66-1=15.66ms will be given to low-priority subtree.
And you might be animating the entire screen, e.g. for a route transition
That sounds similar to the "a scrolling ListView" example above in https://github.com/flutter/flutter/issues/101227#issuecomment-1247625317. Just as mentioned there (and a little bit similar to https://github.com/flutter/flutter/pull/106621), we may take a snapshot of the heavy children, when the heavy widgets are rebuilding.
Seems we do not need? We just blindly run whatever should be done next, and suspend when we are near 16ms.
Well, I think there should be a timer for how long the UI is currently built, since you also mentioned near 16ms, and the remaining time. I think it's easier (and that's what I'm going to try) if I just count the time spent on the framework. But as you say, the problem becomes more complicated when you consider the Raster thread.
Well, I think there should be a timer for how long the UI is currently built
I guess that is easy :) Maybe as simple as DateTime.now(), but probably there are something with higher precision etc.
@Nayuta403 I have had similar thoughts before. The problem is, build phase is not the most costly one. There are layout and paint phase, etc, as well. What's worse, Flutter has C++ engine code which rasterizes and flush to the screen. That one can take a long time in some cases (for example, in my own app, when there is a ton of bezier curves). A widget may, for example, have very short build phase time but very long C++ rasterize time.
Yes, the timing of the statistical framework is not complicated, so I'm just trying to perform more tasks in a frame based on that time, regardless of Raster
@Nayuta403 Yes, the timing of the statistical framework is not complicated, so I'm just trying to perform more tasks in a frame based on that time, regardless of Raster
Sorry I do not quite get it. Are you using history timing information to estimate future timing?
It's Keframe. I'm trying to count the time it takes to build/layout/paint item widgets so that each frame can be rendered as many times as possible (currently only one item per frame is rendered).
(Am I making myself clear? (*￣︶￣))
@Nayuta403 Clear :)
So seems that it is based on history. Then what if different items have (very) different time needed? That happens frequently IMHO. For example, suppose we have a ListView of posts. Post 1 may be a simple sentence so it is fast. Post 2 may be a long rich text paragraph and complex Paths etc, so it is slow.
Yes, you are absolutely right, because now every task is setState() and only goes back to rendering the real widget on the next frame. One idea I have now is to make this task a real rendering task, similar to marking it as dirty and then executing drawFrame() to get the real time.
I can create an issue later to describe my thinking in detail and make the issue clearer :>
I can create an issue later to describe my thinking in detail and make the issue clearer :>
Looking forward to it :)
More about "how to build suspendable/interruptable", given that sync generators are slow
Is it possible we create a RenderSuspendable RenderObject (and corresponding Suspendable widget) which does the following:
- Users need to insert this widget into tree whenever they want suspendable. This may be reasonable given this spirit is similar to RepaintBoundary. And users will not need to insert too much, just insert at coarse subtrees.
- It behaves like a most naive proxy render box in normal cases.
- When time is near used up, and when RenderSuspendable.layout is called, it will not call child.layout, but instead set a flag (say
needsLayoutLaterWhenPossible) and directly return. As for the return value, it may return the last layout size or user-defined default size (similar to what keframe does in widget-build level). By doing this, ancestor render objects will be happy and finish its layout function very fast. - For a
RenderSuspendablewithneedsLayoutLaterWhenPossible=true, when a new frame comes in, it willthis.markNeedsLayout(), and thus get a chance to execute itslayoutmethod again in this new frame. If time is enough, it is done normally as in "2.", and the needsLayoutLaterWhenPossible is cleared; otherwise, it is done as in "3.".
Remark: May need a tweak a bit about layout's caching mechanism.
Remark: RenderSuspendable's sub-tree will not be redundantly layouted more than once. For example, say we have a Column with two Suspendable children, the first one has done layout, and the second one does not because of timeout. Then, when the next frame comes, Suspendable 2 calls markNeedsLayout, and Column starts performLayout. Then Suspendable 1 does have layout() called. However, we should recognize it (possibly flutter caching already does so?), and no need to layout its child at all.
Features
- Solves the problem of "how to build suspendable/interruptable", without sync generators
- No need to modify existing render objects, only need to add a new one
Potential problem
Unnecessary (i.e. redundant) relayout will happen for ancestors of Suspendable, until meeting a relayout-boundary.
Not sure how large the penalty is. If we can give near enough relayout boundary, looks like it is no problem? In addition, if we wrap all expensive subtrees into Suspendables, then the rest may be quite cheap.
How is layout / paint / rasterize related?
Done one by one. Layout of everything will be firstly finished. Only after that, we start doing painting of everything. And then rasterizing.
What about paint tree? layer tree? engine(c++) rasterizer?
TBD, I guess will be similar to above. Looking forward to hearing some feedbacks about the approach for layout first!
How can we paint UI onto screen, if we are in half-way of layout/paint/rasterize, and many nodes are dirty / half-way updated?
Basically I have two draft ideas:
Firstly, we may hack the Flutter engine. Let it keep the old content available until the new content is fully available.
Secondly, we may be able to solve it without big modifications to engine. We may just "take a screenshot" before starting the journey of heavy updating. For example, suppose we need 10 frames to fully build/layout/paint/rasterize this widget subtree. Then, we use the new toImageSync() to take a photo of it. Then, during the 10 frames, we can do anything to the render/layer/engineLayer trees, and whenever the parents let us to paint, we just canvas.drawImage() using that. After 10 frames when we are done, we will finally paint the new thing.
By the way, this also has a bonus about predictable time consumption. IMHO, the time of drawing (paint+rasterize+...) a ui.Image may be easily computed, given it is nothing but a rasterized image.
@dnfield Given that you implemented this great new toImageSync feature (https://github.com/flutter/engine/pull/33736), I have a question about its performance:
In the solution above, instead of painting normally, we may have to convert child into ui.Image for each and every paint call.
In other words, in pseudo code:
class OurRenderObject {
void paint() {
// child.paint(); // cannot do this
if (everything_is_not_dirty) {
image = toImageSync(child_render_tree); // save a screenshot
}
// ...do some expensive work here if time is sufficient...
canvas.drawImage(image);
}
}
So, will this have performance penalty or not?
Fix typo
Just fix 2 char typo... Hope there is a more lightweight method that creating a PR!
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.
List which issues are fixed by this PR. You must list at least one issue.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Looking forward to some early feedbacks about the proposal :)
Maybe /cc @dnfield @JsouLiang @Nayuta403 @xanahopper (based on today's activity)
P.S. I am starting to work on a prototype about smoothing the "layout" phase. Will report any progress I make :)
Progress: 62ms -> 22ms for 99th build time of list_text_layout, and its limitations
(Limitations is discussed in the last section of this comment)
The list_text_layout is still too fast on my old android, so I enlarged its scale (have more items in column, more text in each item, etc) a little bit. Code is seen in https://github.com/fzyzcjy/flutter/commit/857210213531e76b3eb5c256a8ef3599ed434703. This yields:
{
"average_frame_build_time_millis": 2.8446626506024097,
"90th_percentile_frame_build_time_millis": 1.213,
"99th_percentile_frame_build_time_millis": 62.531,
"worst_frame_build_time_millis": 63.101,
"missed_frame_build_budget_count": 14,
"average_frame_rasterizer_time_millis": 3.296915662650604,
"90th_percentile_frame_rasterizer_time_millis": 8.09,
"99th_percentile_frame_rasterizer_time_millis": 13.82,
"worst_frame_rasterizer_time_millis": 15.178,
"missed_frame_rasterizer_budget_count": 0,
"frame_count": 249,
"frame_rasterizer_count": 249,
"new_gen_gc_count": 34,
"old_gen_gc_count": 4,
"frame_build_times": [
Then, I implement a proof-of-concept Suspendable. Code is at https://github.com/fzyzcjy/flutter/commit/0babd5b6856bc799c9f369bce75aada7c10fcd0b. Code diff can be found in https://github.com/flutter/flutter/compare/master...fzyzcjy:flutter:feat-smooth?expand=1.
It yields:
{
"average_frame_build_time_millis": 4.24028,
"90th_percentile_frame_build_time_millis": 17.769,
"99th_percentile_frame_build_time_millis": 22.235,
"worst_frame_build_time_millis": 23.829,
"missed_frame_build_budget_count": 41,
"average_frame_rasterizer_time_millis": 3.9516548672566385,
"90th_percentile_frame_rasterizer_time_millis": 8.949,
"99th_percentile_frame_rasterizer_time_millis": 11.202,
"worst_frame_rasterizer_time_millis": 11.604,
"missed_frame_rasterizer_budget_count": 0,
"frame_count": 225,
"frame_rasterizer_count": 226,
"new_gen_gc_count": 17,
"old_gen_gc_count": 4,
Limitations
This is just proof-of-concept and is very naive.
- It only suspends the layout and build phase. (The build phase is wrapped inside layout phase by adding a LayoutBuilder.) Indeed, it does not suspend the paint or raster phase, which should be done in future work.
- It paints nothing (i.e. do not call child.paint) if a Suspendable is suspending. This will destroy the layer tree and C++ engine layer trees, making performance much worse. We should address this problem later, possibly by keeping the layer tree not used but not removed.
- It lets the whole ancestors (up until relayout-boundary) to relayout in each frame.
- Overhead will become non-neglectable, if we want it to run in 60fps. In other words, if we want each frame to be under 16ms, looks like we will only have <10ms for handling the suspendable widgets (rough estimate, but anyway numbers differ on different phones). Then, the price of 60fps smooth animation is that, the suspendable needs longer time to be loaded.
- The current implementation does not run suspendable layouts last. Instead, they are run inside non-suspendable layout. Thus, we have to set a "earlier" deadline (e.g. 12ms, instead of 16.6ms in the example above), and hope that the remaining job will finish quick enough.
- Element.performLayout says, "In implementing this function, you must call layout on each of your children". But, when implementing Suspendable, we have to violate this. We will face troubles, or just minor changes are enough?
- If a child under Suspendable mark itself as needed to relayout/rebuild, and there is relayout boundary between that child and Suspendable, then the suspending mechanism will not work at all.
- Originally all code (implicitly) assume that, when a frame ends, build/didUpdateWidget has been called. But now this no longer holds. That will make a ton of widget fail to work, including those inside flutter framework, and many external packages. For example, those who assume this inside their addPostFrameCallback.
- The demo does not yet provide any animations (e.g. a CircularProgressIndicator), so by merely looking at the screen, we cannot see it becomes much smoother ;)
Just curious how many approved PR are required? I also fixed something in the engine
Furthermore, I think should we able to break the Build call if the Widget is complex and the Widget Build call is too deep and stalling?
@fzyzcjy
New idea: Dual isolates
(This comment is updated)
Advantages
The main goal is similar: No matter how heavy your widget build/layout/... is, animations/gestures should be 60fps.
It does not require existing Flutter/Dart code to accept new assumptions. For example, in the old proposal, the layout may not be called within a frame, and thus build will also not be called. This may violate many existing code. For example, addPostFrameCallback may assume build is done when post-frame.
On the contrary, the "Dual isolates" solution will not have those assumptions at all. It seems not to break existing explicit or implicit assumptions about the code. Except that it will make Dart isolate "freeze" once in a while - but that should not be a problem, since we are all happy with stop-the-world GC and OS's suspending a thread.
In addition, it should have much lower overhead, indeed almost zero. No wasted build/layout happens (unlike RenderSuspendable approach). No unnecessary tree destory and recreate happens.
Background
The approach above, with the minimal sample in https://github.com/flutter/flutter/issues/101227#issuecomment-1248894781, has many known problems which I am not sure whether can all be overcome. I will probably also experiment further on that path as well. At the same time, I find out a new approach without most problems above.
I am not an expert in flutter/engine, so please correct me if I am wrong!
Design
Originally, IMHO we have a UI thread, which runs both C++ code and Dart main isolate code. Now, we have three (but no worries, they will not be parallel most of the time!):
- C++ UI thread.
- Dart main isolate: Run everything you know, i.e. the heavy build/layout/paint/.... Say it takes 2 frames to finish.
- Dart sidecar isolate: Run CircularProgressIndicator, or ShiftTheChild(for scrolling ListView, to be explained below).
An UML diagram is attached below (best read with text explanations here).
Here is what happens when a vsync comes in:
- C++ ui thread receives the vsync. In the old days, it will call dart's DrawFrame. But now, it will set up a timer for a bit less than ~16.67ms (say 15ms), pause self thread, and call Dart main isolate's DrawFrame.
- Dart main isolate's DrawFrame starts running. It runs build/layout happily.
- At 15ms, timer wakes up C++ ui thread. C++ ui thread then immediately "pause" the dart main isolate. This is done by "safepoints". In other words, we insert
safepoint()call tolayout()function of Dart RenderObjects. And that function is a native function reading, say, a mutex lock. When C++ ui thread wants to pause the dart main isolate, it simply acquire the mutex. When Dart goes to the next safepoint() call, it will simply be pause there forever waiting to acquire the mutex (until next frame indeed). - C++ ui thread calls sidecar isolate to compute the whole build/layout/paint procedure. This is done serially now for simplicity, but should be easily parallizable with some locks.
- Sidecar Dart code is a bit different from the traditional widget/renderobject/layers. Instead, it knows which EngineLayer it owns, and only mutates it. For example, for a CircularProgressIndicator in sidecar, it will know it owns a DisplayListLayer, and only modify pictures in it, without touching other layers. For a ShiftTheChild, it owns a OffsetLayer and modifies its offset.
- Now go back to our C++ ui thread. We will simply utilize the current engine layer tree in C++, and the rest is the same, such as giving data to rasterizer thread and render to the screen.
This is not the end of story - notice our main isolate is still computing some layout and is hanging. Now suppose 2nd vsync comes in.
- Again, C++ ui thread receives vsync. It notices there is still remaining job in main isolate. Then it just resume the main isolate, without telling it anything about the second frame. Thus, in the eyes of main isolate, it will think the whole phone just "freezed" for a few milliseconds without other problems, and will happily continue build/layout/etc.
- Suppose the heavy job of the main isolate is finally finished in this frame. Then, it will do painting. In other words, it will mutate the Layer tree in C++ code. We deliberately put no safepoint() during painting, so the C++ layer tree will either be non-mutated or fully-mutated without intermediate case.
- The rest is similar to the first frame, except that our engine layer tree is updated to the new one.
Further improvements
- sidecar isolate should be executed concurrently
- main isolate should also be executed concurrently, with locks protecting critical regions such as mutating the engine layer tree. But otherwise, it should run freely. By doing this, we are guaranteed that, we can let main isolate run using almost a full cpu core. On the contrary, the "RenderSuspendable" approach above will only give, say, 10.67ms out of 16.67ms for heavy widget build/layout, because it need (say) 6ms to paint/rasterize existing things.
What is ShiftTheChild
I want to solve the problem of "ListView scrolling". In other words, when scrolling a ListView, the widget build/layout may be arbitrary heavy, while we should get 60fps.
Thus, let us do the following:
ParentWidgets(
child: ShiftTheChild(
child: ListView.builder( ... )
)
)
Suppose ListView subtree takes 10 frames to rebuild/layout/etc, and suppose the user is scrolling it. Then, during the 10 frames, ShiftTheChild will receive data packets about user dragging and perform a shift (i.e. OffsetLayer's offset) to its child content. ShiftTheChild will be in the sidecar isolate.
P.S. It may not even be a widget or RenderObject, but may be built on some other lower level primitives mutating corresponding C++ engine Layer. But surely we can wrap those primitives and maybe create a RenderSidecar or something new, that should not be a problem.
Minimal example
I plan not to implement sidecar isolate in the minimal example. Instead, just create a C++ function that shifts an OffsetLayer in each frame, as if a sidecar isolate is doing so. This is because the sidecar isolate is nontrivial engineering work but is not the core problem.
UML Diagram
@JsouLiang For the "RenderSuspendable" proposal, I guess we can have nested ones. For the "Isolates" proposal just now, I guess we do not have this problem - the main isolate will be paused at any safepoint, i.e. any layout function.
@dnfield @JsouLiang @Nayuta403 (and other engine masters)
For the new proposal, I hope to see some feedback... Since I am not an expert in flutter/engine (and few materials are about it on the internet). Thus:
- Is there any suggested materials (docs/articles/...) to understand the engine? How do you learn the engine?
- Does my proposal above looks OK?
@JsouLiang For the "RenderSuspendable" proposal, I guess we can have nested ones. For the "Isolates" proposal just now, I guess we do not have this problem - the main isolate will be paused at any safepoint, i.e. any layout function.
multiple isolates is one of my optimize and working in progress, the key to this is some build/layout callback function/method should not be called in non-main isolate, or just serialize/deseralize build/layout request and response to another isolate like a local RPC service. But! Multiple isolates may agains Flutter's principle, I don't sure whether it can be merged.
(Dude, you are really high-producing and I'm reading your new comments try to catch up
And for more, I may offer you some complex card widget case for benchmark.
For this Suspendable render, we may introduce structure like Fiber, I think it is Threaded tree.
First thing to drawing a frame including heavy/suspendable part is transform tree to a list (or just a threaded tree)
Render task 5 and 6 should and can be suspended at any place in it. (What if a widget/node cost timeout?)
we can tell from figure that suspendable is contagious, content in suspendable cannot be non-suspendable.
@xanahopper
multiple isolates is one of my optimize and working in progress
Looks interesting, could you please share the link? I have checked your github but seems cannot find anything. (All my Flutter work are done open-source and can be found at my github).
the key to this is some build/layout callback function/method should not be called in non-main isolate, or just serialize/deseralize build/layout request and response to another isolate like a local RPC service.
Could you please provide an example? Thanks
Btw, my solution does not use non-main isolate with callbacks :) Indeed, I only put CircularProgressIndicator and ShiftTheChild and things like that there. No normal user code should be done in the sidecar isolate, because otherwise it is quite unfriendly to the users (the sidecar isolate has no memory sharing w/ the main isolate).
So I hope it is not a blocker!
But! Multiple isolates may agains Flutter's principle, I don't sure whether it can be merged.
Could you please elaborate a little bit more?
My solution is still mainly single-isolate, and the sidecar isolate (as mentioned above) is just used very limitedly to support animations.
In addition, my multi-thread is still mostly serialized instead of parallel running. There are multiple threads, simply because I want to suspend/pause one thread easily.
Dude, you are really high-producing and I'm reading your new comments try to catch up
Haha take your time! It takes me a day thinking and trying all these things :)
And for more, I may offer you some complex card widget case for benchmark.
Sure, looking forward to that. Btw I also have very complex cases for my own app, but I decide to start from the simple - you know, one of the fundamental rules in software engineering.
@xanahopper's comment in https://github.com/flutter/flutter/issues/101227#issuecomment-1249172293
If I understand correctly, the figure is a bit like a extended version of my prototype https://github.com/flutter/flutter/issues/101227#issuecomment-1248894781 above.
The question is, how are you going to transform a tree to a (suspendable) list - In other words, for example, how to make subtree rooted at 5 become a list that can be paused?
I have proposed using yield in performLayout but @dnfield mentioned it is very slow. Given that your suspendable widgets are contagious, we cannot use yield at all (otherwise we will be using it in a big subtree).
Then in my prototype above I decide to let a Suspendable return zero size when it is near timeout (note: different from your figure, but the problem to solve is similar). But such approach seems not possible for your proposal.
This is solved very easily with my "Dual isolates" proposal. It just call safepoint() in every RenderObject's layout(). Then, whenever the C++ code wants to suspend Dart, C++ will just let safepoint() hang (probably by occupying a mutex). Then Dart code is just hang there, without doing anything, without feeling anything. In Dart code's view it is like a stop-the-world GC indeed.
@wangying3426 https://github.com/flutter/flutter/issues/101227#issuecomment-1240156979
Any update please? We are also interested in this feature.
Btw I forget to mention you (too above in the comments). Yes, now I have many updates :)
Multiple isolates and optimize with it is before the specification phase, just for you known that we both have the same idea that I and my colleagues are working on it. for very early part we think that this way may need modify engine or even the Dart VM.
Last time we coming with a issue #110063 and got a refuse with tough attitude.
Transform
When we need build a Widget, we must already got a widget or state(element), that means we have a factory for children widgets. All Flutter's build (as long as other declaration UI) is a function call, just like
UI = f(g(h(state)))
We just change this to
ui1 = h(state) ui2 = g(ui1) UI = f(ui2)
wrap ever build into a node/task and change all that to a chain list. In practice, we can use a deque to collect deeper call.
Widget tree build:
- Got a node to build from queue, we dont know whether it will be a leaf node.
- Execute node's build, add all children to queue.
- Add executed node to a deque tail.
- Repeat goto 1
Element & RO tree build
Because elements generally need children to be ready, so we have to produce it from leaf.
- Take a node from tail of executed node deque(this will like a stack)
- Produce Element/RenderObject
- repeat
It just like traversal a tree without recursion, so we can suspend and resume at any iteration of traversal. this is a prototype of pseudo code, hope it help.
@xanahopper
Multiple isolates and optimize with it is before the specification phase, just for you known that we both have the same idea that I and my colleagues are working on it. for very early part we think that this way may need modify engine or even the Dart VM. Last time we coming with a issue https://github.com/flutter/flutter/issues/110063 and got a refuse with tough attitude.
I see. Willing to collaborate to make it into reality as soon as possible!
I looked at #110063 now. If I understand correctly, seems that @jonahwilliams refuses because "Splitting the UI thread work into multiple theads is infeasible for several reasons", such as "a single thread means that newspace allocations don't need any locking". However, my proposal above deliberately avoids these problem. In my case, the c++ ui thread is sleeping while dart main isolate is running, and (if flutter does not like multi concurrent isolates) the thread and main isolate can also be sleeping while dart sidecar isolate is running. So, we are still running single isolate, and no lock is needed at all!
In short, I am not using multi threading. Instead, all threads are there only to implement suspending.
or even the Dart VM.
This inspire me of something: If we can implement a suspend mechanism in Dart VM, maybe we do not need that safepoint + one extra thread approach.
Widget tree build
Fully understand now :) That should be very workable, just like React Fiber does.
Element & RO tree build so we have to produce it from leaf Produce Element/RenderObject
Sorry I do not get it. We are not going to produce RenderObjects, but (most of the time) modify (update) them. For example, say you have a RenderPadding. Then we will only modify its padding field and markNeedsRelayout, instead of throwing away the old padding and create a new one.
Most importantly, how can we get the BoxConstraints (suppose we are dealing with RenderBox)? For example, when we are layout() for a leaf, we must know the BoxConstraints its parent wants to give it. But the parent is not yet layouted.
@xanahopper In addition, I have mentioned many limitations of the suspendable tree traversal in https://github.com/flutter/flutter/issues/101227#issuecomment-1248894781 (see last section there). Looking forward to see some solutions about it!
For example, a big problem: Originally all code (implicitly) assume that, when a frame ends, build/didUpdateWidget has been called. But now this no longer holds. That will make a ton of widget fail to work, including those inside flutter framework, and many external packages. For example, those who assume this inside their addPostFrameCallback.
Update: More problems are added to that comment. For example, "If a child under Suspendable mark itself as needed to relayout/rebuild, and there is relayout boundary between that child and Suspendable, then the suspending mechanism will not work at all."
Just one is enough.
Fix UiKitView which wrongly unconditionally repaints
How the bug is found
This bug is surely not found by human eyes reading each and every line of Flutter code :)
I proposed the idea that a linter can be created (https://github.com/dart-code-checker/dart-code-metrics/issues/997) to validate whether RenderObject field setters have correct early-return code. @incendial implemented it and ran it (https://github.com/dart-code-checker/dart-code-metrics/pull/1003).
Thus, thanks @incendial for implementing the linter and run it through flutter framework code!
Bug description
As we know, when implementing a setter in RenderObject, we usually early halt if the new value is equal to the old value (such as this one, indeed almost all fields in RenderObject child classes are examples). This is very necessary, because we are setting fields in updateRenderObject unconditionally. If we do not early return, we will execute the full logic like markNeedsPaint and so on unconditionally on every updateRenderObject. That will be performance penalty.
The current PR fixes one bug of such case. In more details, the viewController field setter lacks such early-return, and thus unconditionally calls markNeedsPaint. The setter is called from updateRenderObject as follows:
And _UiKitPlatformView is used here:
In other words, the controller is not changed in every frame. Instead, in common cases, it should be the same one for many frames. However, it triggers repaint for each and every rebuild.
Close #111788
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Update: I am thinking whether we can remove the need of new threads in https://github.com/flutter/flutter/issues/101227#issuecomment-1249005541. If we can pause a Dart isolate without needing new threads, we can remove those threads.
Details can be found in:
Update: I am trying to use the spirit of stackful coroutines to implement it.
I do get stuck. We have a ton of callbacks from C++ calling into Dart, such as when the image data has been loaded successfully. If the dart main isolate is freezed (either by stackful coroutine, or by a normal thread with mutex), C++ code cannot call Dart at all. Delaying those calls also seem very troublesome because of resource deallocation problems.
Update: Search a bit on Discord history and here is a summary.
- [cannot find earlier discussions]
- 20220111-20220114, hackers-framework, mentioned in 20220520 by hixie, https://discord.com/channels/608014603317936148/608021234516754444/930241489374683157
- [not found] "Hixie tried an experiment that didn't seem to get to a working point", said here, but I cannot find the experiment code
- 20220520, "general" https://discord.com/channels/608014603317936148/608014603317936150/977074969542553600
- 20220520, "hackers-performance-", with a pointer to "general" as previous discussions, https://discord.com/channels/608014603317936148/613398126367211520/977109431408009317
Well I see some parts of my experiment above has already been discussed there
auto label is removed for flutter/flutter, pr: 111790, due to - Please get at least one approved review if you are already a member or two member reviews if you are not a member before re-applying this label. Reviewers: If you left a comment approving, please use the "approve" review action instead.
auto label is removed for flutter/flutter, pr: 111790, due to Validations Fail.
Oops seems to because need 2 approvals
Fix typo again
Well I come again... Again fix one single word typo... Hope there exist a lighter method to do so!
List which issues are fixed by this PR. You must list at least one issue.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Rethinking (overcoming) the shortcomings of the Suspendable "62ms->22ms" experiment
The quoted text are the shortcomings mentioned in the experiment, and black text are my re-thoughts.
It only suspends the layout and build phase. (The build phase is wrapped inside layout phase by adding a LayoutBuilder.) Indeed, it does not suspend the paint or raster phase, which should be done in future work.
Given the discord discussions among @Hixie and @dnfield etc, seems build/layout is mostly the expensive one. So paint or raster may not needed to be considered at the highest priority, at least not implemented in this issue and may defer to future work.
It paints nothing (i.e. do not call child.paint) if a Suspendable is suspending. This will destroy the layer tree and C++ engine layer trees, making performance much worse. We should address this problem later, possibly by keeping the layer tree not used but not removed.
This is not a problem if we only consider the jank caused by widget creation/deletion (like going to a new page or ListView scroll to make a new widget visible).
It lets the whole ancestors (up until relayout-boundary) to relayout in each frame.
But I guess this should not be a big problem in real world, because we should keep the heavy things in Suspendable subtrees and keep the ancestors simple.
Overhead will become non-neglectable, if we want it to run in 60fps. In other words, if we want each frame to be under 16ms, looks like we will only have <10ms for handling the suspendable widgets (rough estimate, but anyway numbers differ on different phones). Then, the price of 60fps smooth animation is that, the suspendable needs longer time to be loaded.
However, if we want to keep it single-threaded (single isolate), as #110063 (multi isolate) is refused, this is the price we have to pay.
If a child under Suspendable mark itself as needed to relayout/rebuild, and there is relayout boundary between that child and Suspendable, then the suspending mechanism will not work at all.
We should add more Suspendables if we observe such situation. More specifically, we should add a Suspendable (or, if using keframe-like solution, the FrameSeparateWidget) near that specific widget. Then, this is no longer a problem.
p.s. This is not a problem if we only consider the jank caused by widget creation/deletion (like going to a new page or ListView scroll to make a new widget visible).
The current implementation does not run suspendable layouts last. Instead, they are run inside non-suspendable layout. Thus, we have to set a "earlier" deadline (e.g. 12ms, instead of 16.6ms in the example above), and hope that the remaining job will finish quick enough.
Not a critical problem indeed.
Element.performLayout says, "In implementing this function, you must call layout on each of your children". But, when implementing Suspendable, we have to violate this. We will face troubles, or just minor changes are enough?
Will see whether it is a problem after doing more experiments.
Originally all code (implicitly) assume that, when a frame ends, build/didUpdateWidget has been called. But now this no longer holds. That will make a ton of widget fail to work, including those inside flutter framework, and many external packages. For example, those who assume this inside their addPostFrameCallback.
Since this feature is completely opt-in (you have to manually put the Suspendable widget into your tree), users may be able to migrate their widgets when they decide to use Suspendable.
The problem is, it may take efforts to migrate each and every widget, and it also takes time to migrate all inside flutter framework itself. Luckily, it is opt-in, so we can do it steadily and slowly, just like how we migrate to Material 3 theme (it has been months but still not finished).
Many code may migrate smoothly without any problem. (For example, I personally used MobX for my Flutter app, which has reactive states and automatic rebuild, so I seldom touch the raw frame callbacks. For many widgets in flutter framework we can reason about it in our heads and they seem ok as well.)
We may need to provide some information to the users, indeed States or BulidContexts, telling them they have been suspended. A simple method may be adding a field to State/BuildContext, or use a InheritedWidget. I may defer this work after seeing what info a real widget wants when migrating real widgets.
Enhance keframe: Now seems it can build/layout as many items as possible until time is up, i.e. have strategy similar to the "layout" proposal above
@Nayuta403
The problem
As is discussed in https://github.com/LianjiaTech/keframe/issues/12#issuecomment-1238873216 and (IIRC) earlier comments, keframe now blindly builds one widget per frame, even if it can build (for example) 5 widgets. This makes the UI need much longer time to display fully. In addition, it always lag by one frame, because it uses setState in a addPostFrameCallback to update its widget.
The solution
IMHO, the following suggestion can avoid the problems above. Now it can build/layout as many items as possible until time is up, i.e. have strategy similar to the "layout" proposal above. Please correct me if I am wrong!
As can be seen in the code example below, the key point is a LayoutBuilder wrapped as parent of FrameSeparateWidget. By doing so, we ensure that the build and layout phase of widgets prior to the current widget has already been done. Now, FrameSeparateWidget can do a simple decision in its build method - if time is sufficient just return new child, otherwise return the old one and rebuild in the next frame.
By the way, this is partially equivalent to the "layout" proposal because of the following: IMHO, the builder callback inside a LayoutBuilder is called within performLayout. Therefore, the build of the child widget is strongly related to the layout of the LayoutBuilder render object. Then, I can partially migrate the idea in the "layout" proposal (where I hacked the performLayout) to this case (where I hack the build).
Full code example and output
The dummy timeRemain simulates the real world where we may have (e.g.) 16ms for each frame.
Details
// ignore_for_file: avoid_print, no_runtimetype_tostring
import 'package:flutter/material.dart';
import 'package:flutter/rendering.dart';
import 'package:flutter/scheduler.dart';
import 'package:flutter_test/flutter_test.dart';
late int timeRemain;
void main() {
testWidgets('example', (tester) async {
print('frame #1');
timeRemain = 3;
await tester.pumpWidget(MaterialApp(
home: Column(
children: [
for (var i = 0; i < 5; ++i)
LayoutBuilder(
builder: (_, __) => FrameSeparateWidget(
name: '$i',
child: i.isOdd ? SlowBuildWidget(name: '$i') : SlowLayoutWidget(name: '$i'),
),
),
],
),
));
print('frame #2');
timeRemain = 3;
await tester.pump();
print('frame #3');
timeRemain = 3;
await tester.pump();
});
}
// the `keframe` one
class FrameSeparateWidget extends StatefulWidget {
final String name;
final Widget child;
const FrameSeparateWidget({super.key, required this.child, required this.name});
State<FrameSeparateWidget> createState() => _FrameSeparateWidgetState();
}
class _FrameSeparateWidgetState extends State<FrameSeparateWidget> {
Widget build(BuildContext context) {
if (timeRemain > 0) {
print('$runtimeType#${widget.name} build: time is ok, give normal child');
return widget.child;
} else {
print('$runtimeType#${widget.name} build: time is up, give dummy');
SchedulerBinding.instance.addPostFrameCallback((_) => setState(() {}));
return Container();
}
}
}
class SlowBuildWidget extends StatelessWidget {
final String name;
const SlowBuildWidget({super.key, required this.name});
Widget build(BuildContext context) {
print('$runtimeType#$name simulates slow build (timeRemain: $timeRemain -> ${timeRemain - 1})');
timeRemain--;
return Container();
}
}
class SlowLayoutWidget extends SingleChildRenderObjectWidget {
final String name;
const SlowLayoutWidget({super.key, super.child, required this.name});
RenderSlowLayout createRenderObject(BuildContext context) => RenderSlowLayout(name: name);
void updateRenderObject(BuildContext context, RenderSlowLayout renderObject) => renderObject.name = name;
}
class RenderSlowLayout extends RenderProxyBox {
RenderSlowLayout({RenderBox? child, required this.name}) : super(child);
String name;
void performLayout() {
super.performLayout();
print('$runtimeType#$name simulates slow layout (timeRemain: $timeRemain -> ${timeRemain - 1})');
timeRemain--;
}
}
outputs
frame #1
_FrameSeparateWidgetState#0 build: time is ok, give normal child
RenderSlowLayout#0 simulates slow layout (timeRemain: 3 -> 2)
_FrameSeparateWidgetState#1 build: time is ok, give normal child
SlowBuildWidget#1 simulates slow build (timeRemain: 2 -> 1)
_FrameSeparateWidgetState#2 build: time is ok, give normal child
RenderSlowLayout#2 simulates slow layout (timeRemain: 1 -> 0)
_FrameSeparateWidgetState#3 build: time is up, give dummy
_FrameSeparateWidgetState#4 build: time is up, give dummy
frame #2
_FrameSeparateWidgetState#3 build: time is ok, give normal child
SlowBuildWidget#3 simulates slow build (timeRemain: 3 -> 2)
_FrameSeparateWidgetState#4 build: time is ok, give normal child
RenderSlowLayout#4 simulates slow layout (timeRemain: 2 -> 1)
frame #3
@Nayuta403 If you are interested, I can try to make it a full library. Given that it is based on keframe's idea (hack widget build), but at the same time it is quite different from the existing implementation (do not use addPostFrameCallback and use the LayoutBuilder hack), I am not sure whether I should make a PR to keframe, or I should create a separate lib by myself (and mention keframe)?
@fzyzcjy Hi man, you are very thoughtful and full of passion, thank you for your thoughts. Recently I have been busy with work.I want to first communicate with you about Keframe idea and then follow up your discussion.
As can be seen in the code example below, the key point is a LayoutBuilder wrapped as parent of FrameSeparateWidget. By doing so, we ensure that the build and layout phase of widgets prior to the current widget has already been done.
👍 👍 Your idea is great, we can hit the timer at the beginning of a frame and it seems to calculate the timeRemian. If you don't mind, I think you can create a branch/PR in KeFrame for discussion (I've given you a Write access) because there's some basic mechanics in there and a ready-made example in there.
In addition, it always lag by one frame, because it uses setState in a addPostFrameCallback to update its widget.
I think this will not happen, because KeFrame calls addPostTimeCallBack during initState (i.e.https://github.com/flutter/flutter/blob/5816d20b86b95205c40921fa91ee3434b9c97ac6/packages/flutter/lib/src/scheduler/binding.dart#L1197-L1201) and _postFrameCallbacks call after _persistentCallbacks is finished (i.e.
https://github.com/flutter/flutter/blob/5816d20b86b95205c40921fa91ee3434b9c97ac6/packages/flutter/lib/src/scheduler/binding.dart#L1203-L1210), They're both in the handleDrawFrame method, so I think they're still in the same frame.
Ps: I actually think Fiber and Keframe will end up with similar results, but Keframe will work within the existing framework and won't require a lot of changes to the framework and engine. I think we can contribute it to the flutter after we've optimized it, like nested in a ListView or a Column or something, and open it with flags, like a RepaintBoundary.
@Nayuta403 You are welcome!
we can hit the timer at the beginning of a frame and it seems to calculate the timeRemian
Yes, just like my "layout" demo, which I recorded when the frame begins.
In addition, it always lag by one frame, ... I think this will not happen
Well yes the function call is in the same frame; but indeed, if I understand correctly, the build will lag one frame. Consider the simplest example, where we are building a new widget tree (thus initState) with a child. Then, in frame 1, FrameSeparateWidget has initState and build called. But it is only at the post-frame callback phase that FrameSeparateWidget.result is filled with the real child. So it is only at frame 2 that FrameSeparateWidget really renders the child onto the screen.
Ps: I actually think Fiber and Keframe will end up with similar results, but Keframe will work within the existing framework and won't require a lot of changes to the framework and engine. I think we can contribute it to the flutter after we've optimized it, like nested in a ListView or a Column or something, and open it with flags, like a RepaintBoundary.
That looks interesting, and I love to contribute to Flutter :) But I am worried whether Flutter will accept such widgets that can live in thirdparty packages. On the contrary, if we need to modify the framework and it has to be integrated with the framework, then surely we need to put it into flutter framework.
create a branch/PR in KeFrame for discussion (I've given you a Write access)
Thanks for your invitation (I see it). However, I realize keframe is under LianJia, a commercial company. It is not a person (e.g. you), a nonprofit organization (e.g. the flutter organization, the llvm org, the mobx org), a company known to have a ton of open source contributions (e.g. google), or something like that. So I am very sorry I cannot join it. But anyway, all my work will be open-sourced, and under license like MIT, so everyone can use it!
@Nayuta403 A bit more explanation: Why we do not need to worry about "the child subtree build&layout for a FrameSeparateWidget is so long that it makes everything slow"?
Because if that is the case, we can wrap several FrameSeparateWidget in the heavy parts of that subtree. Then, because each (new version I proposed yesterday) FrameSeparateWidget builds normally if not timeout, it will behave normally if time is ok; on the contrary, as long as time is up, subtree will pause to build. By doing this, we can ensure every FrameSeparateWidget takes moderate time length (say, 1ms), and there is no such case as one FrameSeparateWidget taking (e.g.) 100ms so everything is jank.
Update: Some experiments here using the new implementation (proposed here).
https://github.com/fzyzcjy/flutter_smooth
Btw I find that performance boost varies a lot when considering different experiments.
But anyway, all my work will be open-sourced, and under license like MIT, so everyone can use it!
Yes, I was negligent. Lianjia is the company I used to work for. Just because this project is completed by me and has a certain number of users, I am still maintaining it personally. You are absolutely right, I also wish we had some open-sourced work available to everyone. We can work on your project https://github.com/fzyzcjy/flutter_smooth
A bit more explanation: Why we do not need to worry about "the child subtree build&layout for a FrameSeparateWidget is so long that it makes everything slow"?
Yes, I can understand that. For the subtree to time out, we can delay the build again by nesting the FrameSeparateWidget, which I've used before. I think from this point of view, all widget builds are interruptible, this Fiber-like mechanism. I have a crazy idea that if we add a placholder property to all widget(Not all, we can add this property to some base class), we will build the placholder if the frame timeRemain time is 0. Then the jank will never happen ! HHHHH
Update: Some experiments here using the new implementation.
I like your new implementation, which seems to have solved the problem we mentioned earlier by laying out as many widgets as possible in each frame. I think there may be some details that need to be added ：
- Whether other lifecycle states should be considered for
_SmoothState, such asdidUpdateWidgetoronDispose. For example, I encountered an error in keframe when setState was called from outside becauseresultwas cached in State. If thewidget.childis changed externally so that it does not work (It doesn't look like it's going to happen because you're using widget.child directly in build, but I think you might need to think about it when using State, I can add a https://github.com/LianjiaTech/keframe/blob/master/example/lib/page/complex_list_example.dart in your example) - In your example, the height of the item is 24. But for the list, many times we don't know the width and height at code time, and jitter will occur when the placeholder and the actual list are not the same width and height.(because placeholder becomes item, Causing sibling layout changes). like this example in keframe. I did this by using
SizeCacheWidgetto cache the width and height of the item and force it to the placeholder so that it would not shake the second time the item was displayed. (You can't avoid it the first time, because the Item doesn't have a layout.) Do you have any other ideas?
On the contrary, if we need to modify the framework and it has to be integrated with the framework, then surely we need to put it into flutter framework.
Yes, I think if we do it well enough, we can communicate with the Flutter Team and submit it to the Flutter Framework. I communicated with @dnfield a long time ago and he was also interested in it. discord
If we want to commit to Flutter Framework , what is the value of kTimeThreshold? Since we are only counting the build/layout time now, using 16.7 doesn't seem particularly appropriate, and for 120HZ devices, this value should be 16.7/2 ms
How do you think about it?
Yes, I was negligent. Lianjia is the company I used to work for. Just because this project is completed by me and has a certain number of users, I am still maintaining it personally. You are absolutely right, I also wish we had some open-sourced work available to everyone. We can work on your project https://github.com/fzyzcjy/flutter_smooth
Sure! Looking forward to collaborations :)
I have a crazy idea that if we add a placholder property to all widget(Not all, we can add this property to some base class), we will build the placholder if the frame timeRemain time is 0. Then the jank will never happen ! HHHHH
Haha that is really a crazy idea! The problem is overhead will be very big though :)
Whether other lifecycle states should be considered for _SmoothState, such as...
Agree! At least we should add a test in our code, asserting its correctness
I did this by using SizeCacheWidget
I think that is a pretty smart idea, and has not found other solutions yet. If you approve I will add things similar to that into the codebase. The idea will be the same, while implementation will differ slightly (e.g. use a InheritedWidget + StatefulWidget + controller).
Yes, I think if we do it well enough, we can communicate with the Flutter Team and submit it to the Flutter Framework. I communicated with @dnfield a long time ago and he was also interested in it. discord
Totally agree. (Btw I have searched through the history a few days ago: https://github.com/flutter/flutter/issues/101227#issuecomment-1249961627)
If we want to commit to Flutter Framework , what is the value of kTimeThreshold? Since we are only counting the build/layout time now, using 16.7 doesn't seem particularly appropriate, and for 120HZ devices, this value should be 16.7/2 ms
The 120hz should be simple since we can detect what frequency we are under.
The problem is "we are only counting the build/layout time now".
Btw, I realized that, for a scrolling list, the "finalizing" phase also takes time. Let alone the paint/compositing phase we all have known.
They (paint/compositing/finalizing) each take a little of time, but when accumulated, it is non-neglectible for that 16ms.
Btw, recent ideas:
- I am considering halting the
paintphase as well: Maybe we can directly reuse the old Layer, so we can get the same UI and at the same time do not call paint on subtree. This is just very naive idea and I will make an experiment later. - "for a scrolling list, the "finalizing" phase also takes time" - Maybe we can hack ListView itself, and control when it disposes its widgets.
Change type in ImplicitlyAnimatedWidget to remove type cast to improve performance and style
Hi, hope to have a quick view whether this PR is acceptable or not? If yes I will add an issue and maybe ask for test-exempt.
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.
List which issues are fixed by this PR. You must list at least one issue.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
New idea: Preemption
Advantages
It can nearly achieve my goal: 60fps, no matter how heavy your build/layout are. Without limitations of other approaches, such as the ones in flutter_smooth, or the ones in "layout" proposal where the widget subtree has to allow their build/layout not called in some frames.
It also has zero overhead about re-layouting, i.e. it will never need to pay any extra cost to layout, compared to the widget/layout based approaches. It also solves the problem of "how to suspend a layout". I can explain more advantages and comparisons if needed.
Compared with the "dual isolate" proposal above, that one seems very hard to implement as it requires threads or coroutines, but this proposal is not. In addition, this proposal eliminate the second "sidecar" isolate, and everything is in main isolate, so we can run any code with all data in main isolate memory visible.
Details
Continue and modified from https://github.com/flutter/flutter/issues/101227#issuecomment-1249005541 (the "dual isolates" idea, but without the need of adding new threads (very troublesome to do syncing), or c++ coroutines (troublesome when c++ wants to call dart callback).
Notice that, the c++ code, main isolate, and sidecar isolate all run on "ui thread". No new threads, no coroutines, etc. So this time, the diagram draws nothing but very normal function calls.
Description of the figure:
- vsync comes.
- As normal, C++ calls Dart's drawFrame.
- Suppose Dart has 3 widgets to build/layout. It build/layout the 1st, then 2nd.
- Then it realizes time has up (say, 15ms has come), when
layout()the 2nd widget. Then it callspreemptRaster()(a dart function). - In
preemptRaster, we firstly callpreemptModifyLayerTreeto modify the layer tree a bit, like CircularProgressIndicator or the scrolling ListView wrapper widget case, described in "dual isolate" proposal above. For simplicity, imagine thispreemptModifyLayerTreeis implemented via very low level API, such ascontainerLayer.offset = Offset(10,20). - In
preemptRaster, we then call a probably modified version of FlutterView.render. In other words, we provide layer tree to C++ code, and c++ code provide it to raster thread. Notice what layer tree we provide here: BecausepreemptRasteris called within alayout(), thepaintphase has not started, so the layer tree is completely old (instead of mixed). ThusIn addition,preemptModifyLayerTreewill modify the layer tree a bit. That's all. We will send this to raster. - Raster thread renders that layer tree as usual, so we see beautiful things on screen.
- UI thread C++/Dart goes on, because
preemptRasterfunction returns. The Dart code will continue from wherepreemptRasteris called (you know, just very plain function calls; but this solves the "how to suspend a layout call" implicitly indeed). In Dart's view, it thinks it is still the 1st frame. Let's say it continues layouting the 2nd widget. Then 3rd widget. Then paint, flush compositing bits, semantics, etc. - Then finally, as a normal pipeline stage, dart provides the new layer tree and let c++ to throw it to the raster thread.
- Raster thread renders it to screen in the background.
- Then, just like what will be done normally in frame 1, call post frame callbacks, c++ calls dart for some callbacks, etc.
- Now ui thread is idle. When next vsync comes, the same loop will go.
I think that is a pretty smart idea, and has not found other solutions yet. If you approve I will add things similar to that into the codebase. The idea will be the same, while implementation will differ slightly (e.g. use a InheritedWidget + StatefulWidget + controller).
Yeah, I think the code willn't be much different, or I can directly PR to your repo? This jitter usually occurs in ListView, which needs to be nested with SizeCacheWidget in KeFrame, and LayoutInfoNotification is emitted in FrameSpeWidget. So the user has to specify SizeCacheWidget if they want to user ListView. if it's in Flutter framework, we can add it directly, or do you have other ideas?
The 120hz should be simple since we can detect what frequency we are under.
Yes, we can get it directly from the engine, but I have to see how to get it in the framework. It may be necessary to add an API
"for a scrolling list, the "finalizing" phase also takes time" - Maybe we can hack ListView itself, and control when it disposes its widgets.
Yes, I think we can ignore this factor for now as I understand it is not particularly time consuming. Or we can directly change the "finalizing" timing of the ListView.
I am considering halting the paint phase as well: Maybe we can directly reuse the old Layer, so we can get the same UI and at the same time do not call paint on subtree. This is just very naive idea and I will make an experiment later.
I agree with that, I think you just need to nest RepaintBoundary on the subtree, right? Just like ListView item, avoid subtree paint causing pain in other widgets.
@fzyzcjy
I got a bad cold yesterday, so I was late in answering the message
@dnfield @Nayuta403 @JsouLiang (and other experts) I have made a "preemption" proposal, which is like a easy-to-implement version of "dual isolate". Looking forward to any feedbacks! I am going to implement a prototype tomorrow :)
Same thing in discord: https://discord.com/channels/608014603317936148/608021234516754444/1021783497112821861
There are some discussions going on there as well. For completeness, a reader of this github thread may need to go to this link and view comments there.
@Nayuta403
or I can directly PR to your repo?
Sure! But I am hesitate whether going on flutter_smooth now, as the "preemption" proposal seems quite appealing and addresses many problems of flutter_smooth, the layout proposal, and the keframe.
Could you please have a look at "preemption" proposal :) I want to implement a prototype tomorrow (UTC+8 timezone).
This jitter usually occurs in ListView, which needs to be nested with SizeCacheWidget in KeFrame, and LayoutInfoNotification is emitted in FrameSpeWidget. So the user has to specify SizeCacheWidget if they want to user ListView. if it's in Flutter framework, we can add it directly, or do you have other ideas?
That LGTM. Indeed I will do something like: The SizeCacheWidget (I may call it SmoothParent) has some inherited widget to provide a controller to its child subtree. Then child can save anything they want to that controller. Anyway, those are simple things, and I can also do it if you like (just need e.g. 15 minutes).
Yes, we can get it directly from the engine, but I have to see how to get it in the framework. It may be necessary to add an API
I remembered I did that via calling java/swift. Anyway this is minor problem :)
I agree with that, I think you just need to nest RepaintBoundary on the subtree, right? Just like ListView item, avoid subtree paint causing pain in other widgets.
Yes, but I hope not too many RepaintBoundary in the meanwhile. IIRC during some testing they add overheads. Btw the "preemption" proposal does not have this problem.
I got a bad cold yesterday, so I was late in answering the message
Sorry to hear that, and hope you are getting well!
Hi, I have proposed an approach for 60fps smooth animation no matter how heavy widget tree build and layout is, without paying extra cost (such as redundant re-layout). https://github.com/flutter/flutter/issues/101227#issuecomment-1252379787
@Hixie @dnfield (since @dnfield mentioned that layout suspending has been discussed between them; this one does suspend layout and render first)
I am planning to start working on a prototype ~9hr later, but want to hear some hints from you experts, since I have not quite hacked the engine before
I'm curious about how preemtRaster would modify the layer tree and how it would know where to resume. Those are the more difficult bits.
Let's say, for simplicity as a demo, just modify a layer directly via lowest level api. - But we can definitely wrap it to some higher level. And with future thinking maybe we can also do something with existing widget framework.
how it would know where to resume: Just call function and it returns! Example:
class RenderObject {
void layout() {
if (time_is_nearly_out) preemptRaster();
normal_layout_things;
}
}
void preemptRaster() {
modify_layer_tree_for_animation();
FlutterView.render(the_layer_tree);
}
@dnfield
Anyway, those are simple things, and I can also do it if you like (just need e.g. 15 minutes).
Haha OK, you do it 👍🏻 If I do I think it will probably take more than 15 minutes to communicate. hhhh
Could you please have a look at "preemption" proposal :) I want to implement a prototype tomorrow (UTC+8 timezone).
I wonder how this Frame1 is generated, now there are only two widgets with build/layout and neither of them have paint/comp etc. If you use the LayerTree from the previous frame that It looks the same as it does now. (A jank happened) Am I getting it wrong? I'm looking forward to seeing your prototype : )
A question: Take RenderOpacity for example, and suppose we want that in preemptRaster. Currently it is:
layer = context.pushOpacity(offset, _alpha, super.paint, oldLayer: layer as OpacityLayer?);
// definition
OpacityLayer pushOpacity(Offset offset, int alpha, PaintingContextCallback painter, { OpacityLayer? oldLayer }) {
final OpacityLayer layer = oldLayer ?? OpacityLayer();
layer
..alpha = alpha
..offset = offset;
pushLayer(layer, painter, Offset.zero);
return layer;
}
So, my naive thought of directly manipulating it:
void directlyManipulateOpacityInPreemptRaster() {
final OpacityLayer layer = layer_tree.children.where(...); // find opacity layer
layer.alpha = some_new_value_for_animation;
}
If I call merely this, without all those PictureRecorder/create-new-layer etc, will it be acceptable?
If I do I think it will probably take more than 15 minutes to communicate. hhhh
Haha I think so!
I wonder how this Frame1 is generated, now there are only two widgets with build/layout and neither of them have paint/comp etc
Well I should say, this diagram happens after we have rendered a lot of frames. So the layer tree is already there, just without the several newly added/modified widgets.
If you use the LayerTree from the previous frame that It looks the same as it does now. (A jank happened) Am I getting it wrong?
No, I call preemptModifyLayerTree. That one handles animations, e.g. CircularProgressIndicator, or ListView scrolling, or opacity changing animation. For simplest example, for opacity, it may update a OpacityLayer.opacity from 0.1 to 0.2 etc.
I'm looking forward to seeing your prototype : )
Thanks :)
What happens if a render object doesn't implement preemptRaster?
Just put it into RenderObject.layout
e.g. add if (time_is_nearly_out) preemptRaster(); as 1st line of RenderObject.layout. (Surely we may optimize it to be faster, but idea is same)
What happens if devs write something slow into preemptRaster? 🙂
I'm curious about this, maybe it'd help to see a little bit more detail around implemenation. Perhaps write up a doc?
Just do not do that 😉 It is like asking "what if dev use a million of pushLayer today (answer: it will be slow)"
Sure! Do you mean https://docs.google.com/document/d/1SFRO8U2toOlAaZ38dsuEU7Wm5fn41wvBCWKiwADqfmw/edit the flutter design doc template?
So I think an ideal solution will not require developers to update their widgets/render objects, and will not break if developers decide to just throw tons of work into a new method they have to implement
Yes, that would be good
not require developers to update their widgets/render objects -> Yes, this solution do not require
will not break if developers decide to just throw tons of work into a new method they have to implement -> dev do not implement preemptRaster
Instead, the API we give is like this:
No dev will know what is preemptRaster. They only know that, when they want a smooth CircularProgressIndicator, or a smooth ListView scroll, or a smooth opacity animation, they put a special widget into the tree, say, PreemptCircularProgressIndicator()
Under the hood, our PreemptCircularProgressIndicator will utilize preemptRaster.
Btw, preemptRaster is not a method in RenderObject that everyone needs to implement, unlike layout/paint/... which everyone should impl
preemptRaster is like some utility function, that only we flutter framework dev need to impl once
when they want a smooth CircularProgressIndicator, or a smooth ListView scroll, or a smooth opacity animation Isn't that always?
Haha, then maybe make it the default 🙂
Or, maybe add a flag into it
say, CircularProgressIndicator(preempt: true/false)
Anyway I have not think about the details about the high level apis inside preemptRender. It may or may not support arbitrary widgets. I am thinking about making it run firstly.
Btw, what are you guy's timezone? I will create a design doc probably within an hour, not sure whether you guys are online or not
Just sharing some experience I've had working on performance: it's very rare to find that applications are blocked on the UI thread, except for 1) incorrectly implemented scrolling 2) lack of isolate usage for data processing. Thus I would be quite skeptical that this sort of change would actually be beneficial to most Flutter developers, and design doc or not there is almost no chance I would be in favor of adding this to the framework.
@dnfield pointed out another case to me, which is that on particularly low end android devices, even simple UIs can jank due to text layout costs. Though I think its also fairly common for these particularly low end devices to have very few cores, or only one or two fast cores - meaning that multithreading may not help much either.
I think the only thing that would cause me to change my mind is a prototype that demonstrated substantially better performance on a real-ish app; that is one that did not intentionally do way too much work.
Not trying to be too discouraging, but I want to make sure that we're on the same page on the expectations for a feature like this.
it's very rare to find that applications are blocked on the UI thread, Me 😦 Quite complex UI, on very low end devices
I understand that, but then every time I get source code access, its always 1) or 2).
not saying that you're wrong, just that I'm not willing to take anyone at their word for this. I want to see the example code
(Sorry you already mentioned that example)
(wait a minute I first read all messages)
Though I think its also fairly common for these particularly low end devices to have very few cores, or only one or two fast cores - meaning that multithreading may not help much either. My suggestion is not multithreading, it is still single thread 🙂
I might be mixing this up with the other github issue on a separate animation thread
I think the only thing that would cause me to change my mind is a prototype that demonstrated substantially better performance on a real-ish app; that is one that did not intentionally do way too much work. @Jsouliang @Nayuta I think their
keframehas demonstrated some real cases where it boosts performance. I will find an article. Wait for a minute
A document would be a good place to start then 🙂
The readme explains a bit
They also use it in LianJia app IIRC, a somewhat large company
And in Bytedance, they said they have done sth similar to optimize
so I guess these are evidence of optimization of speed in real world
https://github.com/flutter/flutter/issues/101227#issuecomment-1247545240 Yes, we [people in bytedance] all farmilar with KeFrame and has already applied some optimize like it.
Note that my solution is quite diff from keframe.
If your solution is quite different, definitely write up a doc
The only similarity is that, we both want to address the less-than-60fps jank
Sure! I will do that in an hour
Btw there is a brief (1-page) proposal currently: https://github.com/flutter/flutter/issues/101227#issuecomment-1252379787
I thought that was the dual isolates/multithreading idea?
Nonono
I have removed dual isolates and multithreading or coroutine 🙂
Now look at the biggest fig in that comment
it is nothing but NORMAL function calls
Btw, keframe has popularity of 93% with 100+ likes in pub https://pub.dev/packages/keframe
If nobody is facing build/layout jank, I guess it should not be a popular lib at all 😉
keframe has some hard-to-overcome shortcomings, but it is still already this popoular
most popular packages don't get folded into the SDK
@fzyzcjy I would second @dnfield 's recommendation to write a doc. Most of us are going to have trouble following a github issue with dozens of comments, and we're not sure which parts of the proposal are still valid and which aren't
93% isn't actually as popular as it sounds; there are enough packages uploaded now that the percentage can be somewhat misleading. That puts it at something like 2000th.
most popular packages don't get folded into the SDK Sure, I know that 🙂 I just want to say "there do exist real-world cases who needs to be extra smooth"
Most of us are going to have trouble following a github issue with dozens of comments, and we're not sure which parts of the proposal are still valid and which aren't
I see, just ate and now start writing
93% isn't actually as popular as it sounds; there are enough packages uploaded now that the percentage can be somewhat misleading. That puts it at something like 2000th.
Did not know that before 🙂
But anyway, hopefully my comments above already show realworld cases: The bytedance and the lianjia. Especially bytedance (IIRC it is even on flutter.dev frontpage?).
I understand that this technique has been successful for bytedance and others, I'm not disputing or disagreeing with this. But adding something to the SDK, especially if it is a large intrusive change, is going to be held to the standard of whether it will be successful for all or most users of Flutter today. To evaluate this, we'll need at least a design doc, so we can understand the change you're trying to make. Ideally we would also have some sort of prototype, so that we can understand the behavioral changes, if any, required.
"large intrusive change" - I am trying to make it small 🙂 "we'll need at least a design doc" - writing! https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko/edit# (surely not finished yet though) " Ideally we would also have some sort of prototype, so that we can understand the behavioral changes, if any, required." - I also want to do that
Will ping here when finish writing
Thank you!
You are welcome!
https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko/edit?usp=sharing is ready
@Jonah Williams @dnfield (who said I should provide a doc)
Design proposal: https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko/edit?usp=sharing
If we can stop rendering at any point, how do we resolve the sizes of render objects that depend on their children?
What about something like a layout builder? If we pre-empted layout then we may not actually be able to finish building?
If we stop rendering based on time elapsed at an arbitrary RO, there is a risk we end up with a UI that makes no sense. i.e. we could get buttons with no labels or half filled in text. I'd be concerned that without a developer making an intentional choice of where to stop rendering, we'd be worse off than if we janked and took longer to render
It might be worth contrasting this approach with keframe, or elaborating on what problems this solves that keframe cannot
We do not resolve sizes. Indeed, we are using the previous fully rendered UI (plus modifications in preemptModifyLayer).
Well we are using the previous fully rendered UI + modifications in preemptModifyLayer. We will never see half filled UI!
The flutter test framework generally allows developers to elapsed arbitrary amounts of time with fake async usage. This is intentional to ensure that unit tests can be reasonably deterministic.
It would be massively breaking for unit tests to take a different number of frames to reach the same conclusion, depending on the speed of the host hardware.
Indeed, imagine the whole proposal like this: We are still running the janky slow UI that is less than 60FPS. But, once in a while, we "secretly" flush old layer tree + some preemptModifyLayer modifications to the screen.
It seems like that would only work if the previous UI was quite similar to the current UI, but that may not be the case.
How do you connect the current render object with the previous UI? The ROs are stateful objects, the only stable representation may be the old layer tree.
We should fake the meaning of "time" in this proposal as well. For example, we may provide a variable called preemptStrategy:
abstract class PreemptStrategy {
bool shouldWePreemptNow();
}
and call it in place of "checking whether 15 ms has passed".
Then, when testing, we are in full control. For example, we can disable the whole preempt. We can decide to preempt at a specific RenderObject we like to test. etc
Sure. I will do that in a minute.
Seems reasonable. FWIW, the amount of time available will vary per platform, and in the case of devices with dynamic refresh rates it may even vary frame to frame. I believe we should know the approximate target time for each frame when it starts though
I am mainly thinking about janks in animations. The examples - progress indicator, scrolling listview, enter page transition, all are examples.
If your previous UI does not look similar to current, it is also OK. My proposal is just like, "originally the UI is janky, now we add some extra frames into its normal frames".
Anyway they can always disable it by a simple flag in widgets.
When flushing ui during animation, we just provide (old + minor modified) layer tree. We do not touch RO indeed.
Agree, and that should be fetchable from some kind of platform APIs. I am saying 15ms or 16.6ms just because it is simple to explain :)
Yes, understood! :)
What I mean is - how do you detect that the previous UI is not like the current UI? Its OK if this isn't a performance improvement for that case, but I don't see how you would actually determine that it was "Safe" to use an old layer tree
It is always safe. Because we are just "inserting" extra frames into the plain old frames!
For example, suppose it originally runs at 10 fps. Now, between frame 1 and frame 2, we insert frame 1a,1b,1c, ..., which is very alike frame 1 except for minor modifications (such as one OffsetLayer.offset).
If the users find this UI weird in their special case, they can also choose to disable surely.
Thanks @fzyzcjy , I left some comments in the doc which I see you've responded to. In general, my feedback is that I don't see how some parts of your proposal would actually work in practice. Re-using the previous frame and yielding during layout may break many fundamental assumptions we've made throughout the framework, and without a runnable example/prototype I don't think we're going to be able to understand the trade-offs you're making.
I'd also add that I don't think you need new engine APIS for this.
@Jonah Williams Hi thanks for the suggestions!
I don't think you need new engine APIS for this. I am worried about this. FlutterView.render says: /// If this function is called a second time during a single /// [PlatformDispatcher.onBeginFrame]/[PlatformDispatcher.onDrawFrame] /// callback sequence or called outside the scope of those callbacks, the call /// will be ignored.
So in our case it will be ignored. We have to change the engine...
that is pretty fundamental
you can only submit a single frame in a vsync
without a runnable example/prototype I don't think we're going to be able to understand the trade-offs you're making. I will try to do so today
otherwise you're just doing extra work
you'd have to submit a frame with you pre-empted frame, and then schedule a new frame to continue running
I think you'd want that approach anyway.
Yes, only one in single vsync. But when the main code is very slow (say 10 frames to build/layout/paint/...), we want to respond to 2nd 3rd ... vsync
So engine has to be modified IMHO
I am not an expert in engine TBH (never did such big changes before!), but I will try my best to prototype it.
That is a very substantial change. What is the advantage to that approach over scheduling a new frame after the preemption?
Sorry not quite get it. If we schedule a new frame, and vsync comes, what should we do? We cannot call Flutter's onDrawFrame definitely, because Flutter is still busy doing build/layout/... of the first frame.
So backing up a bit, you're not really pre-empting in the way I thought you were. My idea was something like:
- Start drawing a frame
- Exceed threshold
- Submit frame
- Schedule new frame with some metadata that allows continuation.
- Repeat
Then your idea is more like:
- Start drawing a frame
- Exceed threshold
- Submit frame
- Continue building
- Submit frame
- Finish building
Is that correct?
Yes!
That is why I have zero overhead for suspending the layout phase
"with some metadata that allows continuation" - that seems to be the approach that was discussed in Jan 2022 by @Hixie etc, and discussed again in May (?) 2022 by bytedance people, and discussed again by me in github. I am writing comparison about it in google doc now (WIP), under the title"Compared with modify-the-layout-function methods".
one disadvantage of not yielding is that I don't think you'll receive input events. i.e. you start a page transition, start janking and then the user cancels. How do you avoid just continuing the animation?
At least if you yield, you could receive input events and run event handlers
Please have a look at preemptHandleTouchEvents
The preempt-aware special widgets will do that. For example, see "scrolling ListView" example. The scrolling will be 60fps.
You're going to end up rebuilding widgets recursively from within layout of a layer tree. I'm not really sure if that would work.
I guess have pre-emption you can yield from layout, handle events, and then go back?
But TBH that seems a lot more complicated than using the existing drawFrame APIs and scheduling new tasks.
and preemptHandleTouchEvents isn't sufficient, because you need to account for the existing gesture areas, otherwise event behavior may change between non pre-empted and pre-empted rendering
rebuilding widgets recursively from within layout of a layer tree Well no? I will just grab the layer tree and send to raster thread, without touching RenderObject, let alone element or widget
I think if you broke it down so that each pre-emption behaved like a regular flutter frame though, it might work
but then you might as well just schedule a frame
but event handlers can setState
and preemptHandleTouchEvents isn't sufficient, because you need to account for the existing gesture areas, otherwise event behavior may change between non pre-empted and pre-empted rendering It is just for simple things like "shifting a listview". For normal gestures, let it be done in normal frame pipeline. In other words, suppose it takes 1s to run a full pipeline, then the ListView will see all touch events during this 1s.
No, you can't create a distinct pre-empty only set of gestures
The preemptHandleTouchEvents will see events in each of 60fps frame, but that is like "secretly peeking at it"
you'll get different behavior between pre epmted and non pre-empted rendering
you have to go through the full event dispatch for correctness
each pre-emption behaved like a regular flutter frame though, it might work I am worried about that, b/c we will be build/layout a whole tree, inside the middle of build/layout a whole tree
consider the case where a ListView is behind something like a pointer interceptor
you don't want the listview to get scroll events in pre-empted frames because the pointer interceptor isn't aware of the pre-empt behavior
consider the case where a ListView is behind something like a pointer interceptor That's why preemptHandleTouchEvents is for animations, not general-purpose
Users may set a preempt handler in this case, specifying it not to scroll
It is like how React Fiber does things
With fiber, JS animation still jank. Only css animation is smooth
React does not have a layout or paint phase, and ultimately works much different from flutter
preemptHandleTouchEvents and its brothers are like "css animation" - another thing, parallel to traditional flutter widgets etc
i.e. React essentially yields during the equivalent of build
Sure, I am just analogy 🙂
I am analogy about the framework users' feeling: They have to write down something different (CSS instead of JS) for smooth animation.
Introducing a different set of event handlers, that may only fire sometimes during scrolling, is not a reasonable change IMO
it will be too hard for users to predict the behavior
Anyway for standard cases, such as ListView scrolling, or any animation that requires one DisplayListLayer, we can embed into framework
Hmm
So what solution do you think?
adjust your design 🙂
you are essentially proposing a new, Flutter-like framework
which, maybe that is the right thing for your use-case
We cannot run hitTest for not-yet-layout widgets I guess
What?
Well I am considering the general use case: Loading indicator, scrolling ListView, enter page transition. Isn't that many people needs 🙂
event handling is pretty core to how the framework behaves. not that we can't adjust the behavior, but what you are describing sounds like a huge departure.
and, if you are seriously dropping frames on the UI side of things with those examples, you will also drop raster frames
Indeed for my own case, I have heavy rasterize. But I do not propose PRs about this, since I guess few people have my own case
so that may not help reduce the perception of jank
Sorry I do not quite get it. I send layer tree to rasterizer at 60fps, so no jank?
I see.
no, because each layer tree also has to be rasterized and then submitted
and that requires work on the engine side and then the GPU
which can also jank
I think the React generator the VirtualDom Tree and Diff VDomTree will cost too long time, so in React18 it use a Fiber to interrupt the process if a recursion level is too deep
You mean rastierzed at ui thread or raster thread?
raster thread
"raster thread runs too slow that it causes visual jank" is not addressed in this proposal indeed.
That may be a separate proposal
Yes I am not analogy to that specific algorithm, just analogy about dev experience.
Which I think is essentially yielding during build, for the Flutter analogy. But Flutter runs layout/paint in the same thread, whereas browsers already have separate threads
Yes, so my design is not an analogy to fiber when it comes to implementation
html is also much more tolerant of half finished UIs....
Indeed this design very different from fiber 🙂
But the widget build or layout is run on the UI Thread, if build or layout is cost too long time, it will jank
Yes, and this proposal tries to address the problem
that same like the vdom diff or build in the js thread
So, currently our problem is, how to handle input events?
i think they are the same issue
I think you should handle input events by using the regular event handling pipeline
Yes that is our goal
which requires you to implement this by scheduling new frames instead of allowing multiple submissions from a single frame
I am thinking about it - is it possible to do this inside preempt...
Is it possible to do like this:
Do not propagate to ROs that are dirty. Only propagate to those who are clean.
i.e. call hitTests etc
All done inside the preemptRender.
@Jonah Williams Will we have trouble?
TBH I think you'll have a tremendous amount of trouble
good luck!
Ah??
For design of event handler, or for the whole proposal?
whole thing
Ah
I would really be interested in a using a prototype
Yes I will do that
do you think it is worthwhile to prototype it
but I am worried this would break in many unexpected ways
if it is a meaningless proposal, I will just halt now
It depends on what your goal is. If you want to keep exploring this, I don't see any way forward besides building a prototype. I may yet be wrong, or we may learn something from the proposal even if it doesn't get accepted.
But building a prototype doesn't mean that we're committed to accepting it in the framework or engine
or you may find ways to adjust the proposal to implement it with few or no changes to the framework
But building a prototype doesn't mean that we're committed to accepting it in the framework or engine
Sure I know that
@Jonah Williams About the event handler problem: In React Fiber example, a programmer has to think about two systems as well - the JS animation system and CSS animation system. They have to think about "the JS animation is jank while CSS animation runs smoothly" and collaborate with that. So in our Flutter even handler problem, maybe it is OK for dev to think about "the ListView event handler is janky, while the preempt scrolling is smooth"?
No
I mean, maybe
but my guess is no
Hmm
It is opt-in. If someone hates it just set flag to false.
HTML+CSS+JS Is not exactly a high water mark of ease of use
and its not a goal to be as complicated as it can be
because the problem isn't just that "the ListView event handler is janky, while the preempt scrolling is smooth"?, its that the ListView only scrolls when pre-empted because I'm actually expecting the events to be sent to a different event handler
I know its limitation and love Flutter (as you can see - I write Flutter code now instead of web code)
or consider the case where the listview is literally covered
"the ListView only scrolls when pre-empted because I'm actually expecting the events to be sent to a different event handler" When
preempt: truethey are expecting it to happen maybe?
so it wouldn't get events normally because the dispatchEvent code handles that
I will think about thta
but now suddenly its receiving scroll events?
With my apologies to @Hixie 😆
I agree that should never happen
Is it possible we let developers manually handle it? Flutter is like a automatic car, but when really needed (for performance), give dev a manual control button?
everything is opt in, not opt out
If I'm trying to provide a generic ListView like widget, how can I tell within a frame if I am obscured or not?
(generally you don't need to handle this, because the hitTest system does for you)
Let the dev do the job. For example, suppose we are developing this Discord mobile app. Then I will specify "it is always visible" if I am the dev.
If we are developing docs.google.com I will also specify "always visible"
Of course, except that it is not the latest route entry - but that case is simple to be built-in
that means that nothing in the framework will be able to use this
That maybe means framework needs to accept a parameter
by users
eg. ListView(prempt: PreemptStrategy?)
where abstract class PreemptStrategy { bool shouldWeAcceptThatHitTest(); }. And provide null to disable preempt.
I know it is not an automatic car in such cases 🙂
And, for the enter page transition, and the loading indicator, we even do not need gestures handling
you do, because they can be cancelled
I mean, no extra gesture handling at 60fps. Just normal handling at low fps
I don't really think it would be OK. You've got a performance improvement that is off by default, and using requires knowing exactly how your layout will look at all times. And nothing in the framework can use it....
For example, I tap a button and enter a page. Maybe we do not care that the system cannot respond to my touch when the new page is transitioning in (indeed, what gesture will we have there?)
But, when new page is transitioning in, if that transition is, say, 15fps, humans eyes can see it and feel it not good
have you looked at how we made the Android ZoomPageTransition faster (on master currently)? FWIW It was almost entirely raster thread issues
Yes I followed that github issue
ahh right, I remember you were in the Github review
haha
But when the new page has a ton of widgets, we will face build/layout jank I guess?
raster slowness is like a separate issue that my current design doc does not address
even on the low end android devices I tested, all of the jank was raster jank
that or GCs, which you will hit even with your system
Hmm so you are saying we do not need to optimize build/layout jank?
@Jsouliang do you see build/layout jank in your bytedance app? Or are all of them raster jank?
Personally speaking my app has build/layout jank as well
But given it is not open sourced and I cannot say I am the expert in optimization this may not be a big evidence
text layout can be quite slow
Raster jank we call look forward to impeller to solve it, case most raster thread jank are caused by share compile
Yes, some UI jank are cased the measure text content
I wish that were true, some Skia functionality is actually quite slow the way we use it. Clips for example, not slow due to shader compile. Same with ImageFilters
that will block the UI Layout
do you have some idea to optimize the text measure?
We've not had any success
Then maybe my proposal is a bit useful I guess?
If your prototype works and doesn't require Flutter 4.0, then maybe
Haha surely not 4.0 - all user visible API are opt in (and btw Flutter 3.0 has no breaking change)
the problem with opt in, is that most folks won't. Similarly, I don't think creating a parallel event handling system is reasonable. I think you will have better luck trying to break apart expensive scenes into multiple frames. But ultimately its your time + resources, so spend them how you see fit
break apart expensive scenes into multiple frames I guess that is how hixie, dnfield, bytedance people, and I have tried. No success yet 😦
Anyway I will also try on that
@Jonah Williams Another workaround: What if we throw away support for gesture system in preemptRender? When scrolling a ListView, it is mainly the inertia the drives the list to move, and seems that human finger speed will not dramatically change during a swipe. Then, if we never support gesture in preemptRender, the following will happen: (1) ListView is scrolled at 60fps, instead of (e.g.) 15fps, so user eyes will not see jank. (2) ListView is responding to user finger at only 15fps, but since this is not a game but just a scrolling, users may not feel it
Now we can by default enable the feature, and no parallel event handling system
Btw for the loading indicator and the enter page transition example, already no need for event handling by default
I looked at the document, an interesting idea for sure with the synchronous/parallel tree. But I also think it won't be feasible to implement the preemptModifyLayerTree() unless the animation is very simple. The more applicable it is, the more you are just reimplementing the framework.
But I also think it won't be feasible to implement the preemptModifyLayerTree() unless the animation is very simple. The more applicable it is, the more you are just reimplementing the framework. It already supports the following with simple code, see doc for details:
- any widgets fit within DisplayListLayer, such as CircularProgressIndicator. Arbitrarily fancy animation goes here, as long as in one DisplayListLayer. Or more generally, if they fit in a leaf subtree.
- let ListView scroll at 60fps
@Jonah Williams what's the 'ghost text' ?
@Callum We may integrate with the existing framework and allow a lot of widgets to animate (just a very rough proposal): Indeed what is done in preemptModifyLayerTree is just to modify arbitrary layers, so maybe we can reuse existing framework to arbitrarily render a subtree etc. Not come up with details yet though. But anyway, I guess most people just need a smooth loading indicator, a smooth scrolling listview?
@Jonah Williams So is this proposal looks ok? (repeat here in case the original comment is already so above that it is not read)
Rendering placeholder text, like a bunch of grey blocks in place of your actual text on first frames or when doing larger transitions
If the behavior of the framework is changing substantially in pre-empty frames then I think most developers would interpret that as a bug
But I wonder how should we know how many blocks? You know each english letter has different width, and paragraphing is quite hard to guess. If we put wrong number of grey blocks, the UI will be of wrong height. Then, after it gets the real height, all widgets below it (suppose we have a ListView) will jump
@Jonah Williams Well, what about this: It is still the plain old Flutter with (e.g.) 15fps. But we just have some "magic" here, such that it automatically generates something as a "tween" to have 60fps.
IIRC, some researches or nvidia has some "magic" such that, they input some low resolution low fps frames, and output high resolution high fps frames Consider this proposal as such kind of "tween creator", then maybe dev will feel it natural
Started a thread.
This is a good part of what makes it hard. The difficult case right now isn't the progress indicator, it's when you have a large scene change (like a route transition or some other full screen animation).
And lets say you want to render a lot of small pieces of text, which now all have to get laid out for the first time and will send you over budget for a single frame.
I see, I guess my proposal can solve the problem?
@dnfield Btw what do you think about the proposal 🙂
Layout for a single widget can easily blow through frame budget. That's part of what we'd like to solve with an interruptible approach, assuming such an approach is possible.
This will cause problems with scrolling/touch events.
These are not particularly pressing examples right now - showing a single progress indicator or simple animation tends not to be the issue, it's more like doing a full screen route transition where you're building a a large new tree or subtree with a few hundred widgets to inflate and layout.
This needs a lot more details about how you would sensibly interrupt and restart layout at a meaningful point.
I think it's interesting but it's missing a lot of important details
I am willing to fill in details
and will prototype as well
so what is missing?
Hmm do you mean one single leaf widget?
Indeed that is also very simple: I am proposing adding if (timeout) preemptRender() at the beginning of each RenderObject.layout.
Now, we can add more. Say, for YourHeavyRenderObject.performLayout() function, add 10 of such if timeout preemptRender
That's "that" approach which I am comparing to. My proposal does not have this :)
showing a single progress indicator or simple animation tends not to be the issue
Well the example is indeed, "have progress indicator 60fps, while we are doing something really heavy". i.e. just the interesting case you said :)
Content edited
The proposal do not interrupt, neither restart. It just calls a function (the preemptRender).
Not sure but maybe you have the same understanding as @JonahWilliams? This comment may be helpful: https://discord.com/channels/608014603317936148/608021234516754444/1021963804747255929
I have replied to google doc (not see that just now)
PerformanceOverlay's multiple fields are not updated when the user wants to update it
Close #112038. Please see description there.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Fix logical error in TimePickerDialog - the RenderObject forgets to update fields
Close #112038. Please see description there.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Fix CupertinoAlertDialog and CupertinoActionSheet, which mis-behave when orientation changes
Close #112038. Please see description there.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Fix SliverScrollingPersistentHeader not able to update stretchConfiguration
Close #112038. Please see description there.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Fix SliverPinnedPersistentHeader, also not able to update stretchConfiguration and showOnScreenConfiguration
Close #112038. Please see description there.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Add assertion to _CupertinoSwitchRenderObjectWidget, otherwise it is confusing why updateRenderObject omits state update
Close #112038. Please see description there.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Fix RenderEditable not able to update backgroundCursorColor when the user provides a new one
Close #112038. Please see description there.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
How is this Layertree generated (since the current RO is not painted), it still looks like the LayerTree from the previous frame?
If I understand correctly, you want to use the full LayerTree result from the last time. Like this object in the Raster thread?
In the current Framework design, once entering the RenderFlexObject, all child nodes will be Layout, and when the RO tree Layout is completed, paint will be submitted. Now we add a timer to the RenderFlexObject layout, the first child node layout is almost finished (16ms), at which point we interrupt the layout and submit the current RO tree for Paint. At this point, RenderFlexObject contains a child node. When the next schedule comes, do I continue to start from this RenderFlexObject node or something else?
So I guess you mean your example has an app which has Flex as its root widget, and thus RenderFlex as its root RenderObject?
When the next schedule comes, do I continue to start from this RenderFlexObject node or something else? We do not continue to "start from" anywhere. We just call the preemptRender function inside
RenderObject.layout(). And then, when preemptRender function returns, we continue running our code.
preemptRender is nothing but a very normal function
Yes, from previous frame, but modified a bit from preemptModifyLayerTree.
Almost the same full layer tree, except for preemptModifylayerTree
I want to know more details of preemptModifyLayerTree CircularProgressIndicator such as you mentioned. What will do in preemptModifyLayerTree?
void preemptModifyLayerTree() {
for (final renderObject in renderObjectsWhoWantsToPreemptModifyTheLayerTree) renderObject.preemptModifyLayerTree();
}
class RenderPreemptDisplayList extends RenderBox {
void paint(PaintingContext context, Offset offset) {
layer = paint_child_subtree_inside_a_DisplayListLayer();
}
void preemptModifyLayerTree() {
layer = paint_child_subtree_inside_a_DisplayListLayer();
}
}
So, it calls RenderPreemptDisplayList.preemptModifyLayerTree
Then, RenderPreemptDisplayList.preemptModifyLayerTree paints its child.
For simplicity, let's say, we do not want a CircularProgressIndicator, but only want a color rect
Then it is like:
class RenderWhatever extends RenderBox {
void paint(PaintingContext context, Offset offset) {
layer = create_DisplayListLayer_and_paint_a_colored_rect(color: red);
}
void preemptModifyLayerTree() {
layer = create_DisplayListLayer_and_paint_a_colored_rect(color: whatever_color_you_like_in_animation);
}
}
What would you do in the paint_child_subtree_inside_a_DisplayListLayer method at this point in the interrupt, maybe the child node hasn't been built/laid out yet?
layer = create_DisplayListLayer_and_paint_a_colored_rect(color: red);
like placeholder ？
For circular progress indicator, just paint a indicator you need
for scrolling listview, it is a longer story, maybe see doc
For readers of GitHub and not yet read Discord: Some discussions happen in Discord as well, see - https://discord.com/channels/608014603317936148/608021234516754444/1021783497112821861
As well as the sub-discussion in discord: https://discord.com/channels/608014603317936148/1021987751710699632
@fzyzcjy The build/layout phase is a recursive call, how do you break it?On keframe, we're actually doing recursion on the current frame, just using placeholders
Just do not break it 🙂 We are calling preemptRender. The call stack become deeper when calling that function
In that figure, a rectangle means the life of a function.
so for not layout node we will paint a placeholder rect like this?
We will know nothing for a node that has not been painted before this frame
Because it is not in the layer tree
No placeholder rect indeed. But if you like, what about just putting a placeholder rect there, in previous frame
Well the story is like:
Some researches or nvidia has some "magic" such that, they input some low resolution low fps frames, and output high resolution high fps frames. Consider this proposal as such kind of "tween creator".
So this proposal will output some extra frames, which are just "previous jank frame's layer tree + preemotModifyLayerTree minor changes like animation"
Does this mean that the build/layout of all nodes needs to be synchronized for this to happen? Normally build, layout are recursive calls.
Does this mean that the build/layout of all nodes needs to be synchronized for this to happen? Normally build, layout are recursive calls. It is still recursive call. Just like plain old. Except that one extra line:
class RenderObject {
void layout(Constraints constraints, { bool parentUsesSize = false }) {
if (nearTimeout) { preemptRender(); }
… the original layout code …
}
}
I kind of figured it out. The core thing is that when my time is running out, submit a LayerTree to the engine in preemptRender for rendering, and then continue with layout (The recursion does not exit at this point
).
Yes!
Then we get zero overhead for a lot of thing, etc
What if the whole element tree and RO tree change in the next frame
Just do the same thing in the next cycle
Is there any problem
Just use this mimic
Is paint applied to the parent of the interrupt node? I'm more concerned with the generation of this Layertree.
No
no paint is happened in preemptRender, except for preemptModifyLayerTree - which only few specialized RO will handle
but anyway, if you use PreemptDisplayList and put progress indicator as its child, then that child does get painted
The key point I think is that the generation of this Layertree, if we just call preemptRender(), For example, layer = create_DisplayListLayer_and_paint_a_colored_rect(color: whatever_color_you_like_in_animation); , it could be very different from the previous UI
We do not generate a whole new layer tree. Instead, we modify it a little bit in the preemptModifyLayerTree
For example, modify the color of a box a little bit, modify the shifting of the listview a bit
so it will be very similar to previous (janky) frame except for a bit of animation change
surely it may be different from next frame, if your prev frame is diff from next frame - but that definitely should be like that
Yes, I think it depends on the height similarity between the two frames
There are two issues I would love to see more detail on:
- How to generate this Layertree to the Engine when an interrupt occurs (e.g., two frames are completely different)
- How to handle vsync scheduling in case of interruption (for example, the whole Widget tree changes in the next vsync)
Would you consider making a prototype of this solution any time soon?
How to generate this Layertree to the Engine when an interrupt occurs (e.g., two frames are completely different) Since when preemptRender happens, we are in build & layout phase, we never touch layer tree yet. So we have the old layer tree. Now, we provide the old layer tree (with minor modify from preemptModifyLayerTree), to the engine
How to handle vsync scheduling in case of interruption (for example, the whole Widget tree changes in the next vsync) No vsync is passed to dart layer during jank. Say we have a build/layout that takes 1s. Then during the 60 hardware vsync, we render 60 frames to screen via preemptRender. However, for the Dart code (excluding the preemptRender part), it only sees one vsync, one build/paint/layout/etc.
Would you consider making a prototype of this solution any time soon?
Sure 🙂
I think the core is the design of this preemptModifyLayerTree which sounds similar to some low FPS conversion high FPS (inserting excessive frames in the middle).
Yes, I have said that here:
Indeed I am inserting a paragraph saying similar things into the doc at the same minute haha
Yes, (sorry I didn't really understand that before)
That's OK
If the core is preemptModifyLayerTree, that means when you want do some high priority render, you have to implement it?
Yes and no.
- For whatever widget tree that fit in a layer subtree without disturbing others, like a CircularProgressIndicator, reuse existing (well, to-be-written) PreemptDisplayList
- For listview scrolling, maybe just let ListView's RO implement preemptModifyLayerTree to implement inertia etc. (The design doc is a bit old, see discussions above for details)
- But I will think about more general solution to support arbitrary case. Anyway this is not the top priority - we should have the main idea running
that means a new mechanism to implement for widgets who want use it, and has a cost to migrate
For widgets who want 60fps, I guess not much - you can reuse it in many places
It is like, we write down computeIntrinsicWidth, performLayout, performDryLayout, etc, a lot of things manually nowadays
Notice we never need to implement new functions for things like CircularProgressIndicator
for those, just use the existing (will-be-in-framework) PreemptDisplayList
And according to you pseudo code, one RO has a layer for previous frame, if it laid with all its relative parts, what should we do for its layer, will it be updated? and what happened when it completed? Did that means layer painting is un-interrupted?
if it laid with all its relative parts Do you mean, what happen if that RO has its layer be the new data (instead of old data of previous frame)?
That will not happen. We only preempt during build & layout phase. Paint is not happen yet
I got it, layers only change when paint commit
Yes, IIRC
So there is no such thing like Fiber do, build/layout, record and diff?
Maybe yes?
What if the preemptModifyLayerTree overcost?
Then it janks, surely
but why it overcost
when the implement sucks
e.g. painting a tiny progress indicator, should be fast; for listview scrolling example, just modify a little bit of OffsetLayer.offset
It is like asking "what if you misuse flutter" - answer is you get trouble 🙂
A question popped into my mind, I'm thinking of what the difference between this and Keframe(Especially after your optimization, build/layout as many widgets as possible in one frame)? For example, for some Wigdet you mentioned, need to implement preemptModifyLayerTree, and in this case, what if you use KeFrame and you set a placeholder like that?
tell me the difference between this and Keframe I am going to sleep in a minute, will compare it tomorrow morning. Indeed the design doc has talked a little bit, and github issues also talked a bit.
@fzyzcjy Hey! I've been lurking on the GitHub thread and didn't want to ask this there as I feel like I'm missing something.
Does your most recent proposal allow trivial layout changes to take priority over multi-frame changes? It seems a lot of nontrivial animations require some amount of layout shifting, but your diagram seems to indicate that all widgets must still build and layout in turn, so if an expensive widget needed to rebuild in a different part of the tree then it would still jank since the animation builder would still need to wait its turn.
Sorry if this is somewhat ignorant, I'm still trying to wrap my mind around it all.
Also please sleep if you need to, the question can wait 🙂
@SecondFlight Hi, I will reply tomorrow morning 🙂
Well, I actually know the answer to that, but I think it might need to be spelled out a little bit more clearly
I'm still looking forward to your prototype ，Well done
good night 🌙
Thanks, good night
I've already lost track of this thread, but I think there's some good questions here about how this would work in a flex based widget (e.g. a column or row). We also should try to think about how it'd work in general for multi-child ROs
I think we should probably have a linter that sorts required keyword parameters first. See this screenshot of the api popup in vscode, you have to scroll around and hunt for the signature of itembuilder
@Jonah Williams Continuing from the problem of gesture subsystem of the design, here is a mental modal:
Some researchers/nvidia/etc have some "magic" such that, they can input some low resolution low fps frames, and output high resolution high fps frames. Consider this proposal as such a kind of "tween creator". In other words, originally we have janky rendering (say, 15fps). And now, we add three extra animating frames after each of the 15fps frames, to get a 60fps smooth feeling.
(Copied from updated design doc)
I've already lost track of this thread Ah feel free to ask any questions and I am willing to answer!
how this would work in a flex based widget (e.g. a column or row). We also should try to think about how it'd work in general for multi-child ROs It works well, no need to do any special treatment indeed 🙂
@dnfield Btw I have answered your comments in google doc as well
allow trivial layout changes to take priority over multi-frame changes yes, allow things like a animating indicator or a scrolling listview to take priority over normal janky heavy build/layout
if an expensive widget needed to rebuild in a different part of the tree then it would still jank since the animation builder would still need to wait its turn
If you mean that, your animation requires an expensive widget built onto a different part of tree, then yes it will jank
But will that be common? "some amount of layout shifting" - For example, a scrolling listview is very common case where we need 60fps smoothness. See the design doc, it indeed only needs a tiny change to OffsetLayer.offset
(I will update the doc probably within an hour)
I understand the argument you are trying to make but I don't find it convincing. Adding a second gesture system is not reasonable IMO. You should be working on how to eliminate that from your proposal. More time spent discussing with me is not going to help you there
Adding a second gesture system is not reasonable IMO. I am proposing to have no second gesture system 🙂
Ok I will update my proposal, reflecting there is no second gesture system at all
Forget to do that, sorry
And I am going to prototype today btw
I don't see a clear description of what your proposal is. The first information under "detailed design" are questions.
I should move it, wait a minute
I don't understand the proposal, is it that if build + layout > 1/60ms then capture a continuation of that work, submit a frame to the renderer, and finish the build/layout on the next frame?
@Jonah Williams Hi I have removed gesture system from the proposal, and also add explanations why that looks reasonable. Mainly change the "example 2 implementation" at the bottom of proposal.
"on the next frame" - if it is quite slow, maybe on the next next next frame etc indeed
"capture a continuation of that work" - Well I do not capture anything. The preemptRender is nothing but a normal function call.
We just call preemptRender, which sends layer tree to raster it, and later when it returns we continue doing layout/build
what are you rendering while that work is happening? a partial representation of the ui or the previous ui?
Previous layer tree (given that we preempt at build/layout, no modify to layer tree in current frame). But, we call preemptModifylayerTree, which modifies the layer a little bit for animations
e.g. in listview scrolling example, we call OffsetLayer.offset += 123.45 so content is moved
ahh so, the previous layer, but some effort is made it keep it animating, interesting
Yes, like that
Thanks 🙂
do you have a profile of those build + layout frames that are going over budget by any chance?
I believe it can happen, but my first thought would be that maybe something like synchronous calls that shouldn't be synchronous are happening.
I think that stuff we just said on the discord would be a good description: preempt build/layout, update the last layer, render the last frame, resume build/layout
IIRC some discussions above confirm this jank does happens (especially on low end devices), let me find a link
Around this comment: https://discord.com/channels/608014603317936148/608021234516754444/1021980287787352125
In short: bytedance people reported jank of build/layout in their real world app
I am not bytedance so maybe need to ask them for a profiling data
I also see jank in my app, but given that I am not expert in optimization and the app is not open sourced, my words is not that helpful
Thanks! I will add that
A couple of months ago I looked into build/layout performance. I tried to squeeze as much as I could out of the framework but ran out of ideas. My understanding from looking into is a lot of jank was shader compilation. The thing that might be holding back layout / build is locality which will be hard to fix. But I didn't see a lot of evidence from what I remember that it is worth the investment =T
the way flutter does build and layout is interleaved, but not fine-grained. lots of building happens, then lots of layout, then lots of building, etc. You can run out of time at any time in this process, with a hundred widgets built but not laid out, for example. now the system is in a very inconsistent state.
I guess my point is that if you want to propose a solution that fixes long (build + layout) times, we should establish that that is a worthwhile problem to fix backed up with some data.
since we haven't done paint yet, what we're painting here is just the last frame. there's no need to send it to the engine, it's already got it.
how does it know what to do?
what if the preempt happened before the progress indicator got to rebuild?
Thanks I will add a section to design doc. "shader compilation" - seems that @Jsouliang has mentioned above, they expect Impeller to solve the problem. Personally speaking I am not very sure b/c have not checked into impeller deeply. "might be holding back layout / build is locality" - could you please elaborate it a bit - Do you mean memory locality that causes page fault etc? Anyway, any form of build/layout slowness can be fixed by this proposal, no matter the cause. Even if you just heavily compute synchronously inside initState, this proposal can also fix 🙂 "I guess my point is that if you want to propose a solution that fixes long (build + layout) times, we should establish that that is a worthwhile problem to fix backed up with some data." - I agree. I wonder whether this is enough: @dnfield has gived a pointer, "A good canonical case here would be something like https://github.com/flutter/flutter/blob/master/dev/benchmarks/macrobenchmarks/lib/src/list_text_layout.dart. This ends up being janky because layout gets expensive for all that text (on a lower end phone it can easily take 20-30+ms just to layout all the text there, and the ListTile is a little deceptive because Material introduces expense - this is the kind of thing we want to figure out how to break up "automatically")."
Oh interesting, I hadn't seen the list text layout benchmark. WRT heavily computing inside initState shouldn't happen on the isolate, we should be giving users the tools to quickly offload work to a background isolate. I think flutter/dart can do better there. With the locality, yea talking about memory locality. When I ran profiles I saw numbers that couldn't be associated with a single widget. I suspect it is the recursive nature of crawling down the widget / render trees which jumps all over in memory. Also Dart doesn't have value types so every single rectangle in layout is an indirection to another location in memory. The locality thing is my pet theory, I can't prove it though 😛
also @fzyzcjy you should look into impeller, i haven't run it recently and I don't want to hype it much, but it should make a huge impact. it's almost not worth looking into jank until after it. you can turn it on with a flag.
WRT heavily computing inside initState shouldn't happen on the isolate, we should be giving users the tools to quickly offload work to a background isolate. Sure, that should be a misuse in most of the time 🙂 But indeed, sometimes it is sane to do so. For a simple example, you may have a giant tree in main isolate, and have to modify a lot of its nodes. It is hard (or impossible) to send it to another isolate, modify, and go back, otherwise copying is either too slow or even impossible for some types of objects. Anyway that is just a side remark, and my proposal is not designed to solely solve this special case
With the locality, yea talking about memory locality. When I ran profiles I saw numbers that couldn't be associated with a single widget. I suspect it is the recursive nature of crawling down the widget / render trees which jumps all over in memory. Also Dart doesn't have value types so every single rectangle in layout is an indirection to another location in memory. The locality thing is my pet theory, I can't prove it though 😛 I see. So my design should solve it 😛
IMHO impeller is purely about raster thread? (Correct me if I am wrong!) In other words, even if we have impeller, if our dart code janks, it still janks just like the old days
Sure. Then I should insert if timeout then preemptRender into the build() as well, and seems we have no problem.
Btw, no inconsistent state :) I am sending the layer tree in preemptRender, so I just accept the dart state to be completely in middle
I can't remember for sure but shader compilation may happen on the platform thread while the raster thread waits for it synchronously.
Last frame, but plus "preemptModifyLayerTree". For the scrolling listview example, we will modify the OffsetLayer.offset a bit (the amount by inertia). Then, users will see the list content shifted a little bit. So users will see list scrolling smoothly at 60fps, even if one build/layout/paint/... cycle takes, say, 10fps.
It is hard (or impossible) to send it to another isolate, modify, and go back, Yea a couple months ago I tried to come up with a way to get move semantics for sending data between isolates in constant time. I couldn't find a good proposal that the dart team liked.
See examples below, flutter framework dev (e.g. me) write down some code to cover a range of cases. Users can also write some if want very special behavior
No problem :) Progress indicator updates the UI also inside preemptModifyLayerTree, and that is why it is smooth 60fps.
Isolate.exit does send most data in constant time
but that doesn't help for long-lived isolates of course, just compute
Yeah, but the problem is I have to send data to that new isolate, which is linear. Anyway, that is just a special corner case, and my proposal does not aim to solve it as the main problem
yea, sorry to digress but it's an interesting problem
I see. Btw platform thread is not ui thread IIRC, so ui thread is still ok?
Haha I also feel it interesting
Const time transfering in both side will be quite helpful indeed
Yea, sorta, but if you are blocking the platform thread you can be blocking events which feels janky. it really depends on the embedder too what effect it can have. I'm not saying build/layout isn't a potential issue, but from my having looked into this a couple months ago I think shader compilation was by and far the biggest issue which is good the flutter team is making a big investment there
i gotta run, good luck @fzyzcjy nice meeting you
I see. Eagerly looking to see impeller be in production as well!
@gaaclarke See you 🙂 Nice meeting you
@Hixie Btw I have replied your questions on google doc (not seeing it just now)
This is the part I don't really understand.
How does the framework know this is the one that needs painting?
What if it this is the slow part?
Users manually write down "PreemptDisplayList(child:CircularProgressIndicator(preempt: false, … other argos …),)"
Then, when framework wants to preempt render an extra frame, it calls PreemptDisplayList.preemptModifyLayerTree.
What if it this is the slow part?
Then it jank. But IMHO this happens rarely, at least should be ok for progress indicator and scrolling list view - could you please provide some real world case where it will be slow?
I have replied to another comment in google doc. Does that help? If not I can elaborate
If the animation is really the slow part under that special case, then any single-threaded solution will not work IMHO. (Seems flutter does not want multithread and I agree with that, so we only consider single thread solutions)
Update: The google doc https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko/edit?usp=sharing is updated, most changes are in "touch event handling system" and "comparison"
@dnfield Yes, if I understand correctly, this case should not need to be considered separately. Since this solution does not affect the existing flow, it just calls preemptRender to submit a frame of drawing to engine at the end of the 16ms. Then continue back to the existing Layout flow
Yes it is like that
@fzyzcjy Yeah, I thought about it later, and I think one of the challenges is. In the case of video, when we go from 15FPS to 30fps, we actually know the content of each frame, so we can calculate the difference between two frames and interpolate them (IIRC some frames are diff messages). But with Flutter, we don't know what it looks like until we paint it on the next frame.
For example, in this case, when I navigate from loading to a specific page
@Nayuta I have not checked Nvidia's tech in details yet. But I guess, because of latency reasons, it will only "guess" extra frames from previous frame? (Will check this soon)
For the navigation, say the new page takes 3 frame to render and the page shift animation is 10 frame. Then, users will see 60fps smooth page shift, and during frame 0-3 they see new page is purely empty and in frame 4-10 they see new page with content
I think that a solution that requires developers to implement their own render objects to achieve this may be interesting for a package but probably won't end up in the core framework
If we require developers to think more about whether their widget/RO needs this, and how to create a custom one that will do the right thing, it's going to be very difficult for them to use. It might be very useful and a skilled developer might find very clever ways to do it, but it won't be something that will scale very well.
requires developers to implement their own render objects to achieve Well I guess yes and no.
- For anything that fit into a subtree, no need for dev to do anything except for inserting PreemptDisplayList widget to the tree
- For list view scrolling, we do need to change listview's source code a bit, but flutter devs will not even know that change
Let me think about whether I can do something like "just drop in a widget and it works" for end users 🤔
Do you think the 1. and 2. above are enough to cover most common use cases?
If so then maybe we are ok - we speed up common cases, and for rare edge cases they need to craft something
It's really hard to tell whether something fits into a subtree, particularly in a large project with many people working on it and in situations where large subtrees are getting built (route transitions)
Scrolling isn't usually the big offender here - we have mechanisms to lazily build the contents of what's getting scrolled, and changing the offset on the layer of what's already built is cheap
The problem is more like "I need to build an entire new tree of widgets in 8ms or less and I have 300+ widgets to build to get there and on my low end phone it's taking a long time"
@dnfield
It's really hard to tell whether something fits into a subtree, particularly in a large project with many people working on it and in situations where large subtrees are getting built (route transitions) Route transitions will always not fit into a leaf subtree indeed, but it is in my example 3. Scrolling isn't usually the big offender here - we have mechanisms to lazily build the contents of what's getting scrolled, and changing the offset on the layer of what's already built is cheap @Nayuta 's Keframe deals with scrolling and is popular, and I also see problems in scrolling (both in my realworld app and my experiments), so I guess it may be a problem sometimes The problem is more like "I need to build an entire new tree of widgets in 8ms or less and I have 300+ widgets to build to get there and on my low end phone it's taking a long time" i.e. page transition? (the example 3)
So maybe what I should do is answering: "Devs want to write fancy route transition animations, by themselves and without need to think hard, and at the same time, the new page is heavy and cannot be rendered in one frame"
I will come back when having a solution 🙂
I think it's a little more like "devs want to inflate a large widget tree and be confident that if it doesn't fit into frame budget it will get broken up into pieces that make sense"
(And they can't spend hours altering the logic controlling that every time they refactor or add widgets)
"devs want to inflate a large widget tree and be confident that if it doesn't fit into frame budget it will get broken up into pieces that make sense" I see. That is exactly what the proposal aims to solve. (And they can't spend hours altering the logic controlling that every time they refactor or add widgets) Totally agree 🙂 I should make the dev-facing API easier
@dnfield Well, maybe I forget to explain: Dev never need to do anything special for most of their app. The only extra thing is the part for 60fps smooth animation. Only if they want something to be very smooth, and it is not a leaf subtree (e.g. loading indicator) and not a scrolling listview and not a already-provided-in-framework page transition, they will have to write some extra code (in the current proposal).
It might help to have some example application showing the improvements
Would you mind giving a realworld example where userrs, using my currently proposed api, have to write a lot of extra code? such that I will have an example in mind to optimize
Sure, I should do that. Probably create an app w/ page transition etc
Quick update: I am now experimenting multiple subtrees (i.e. a forest). Flutter seems to allow so (e.g. examples/api/lib/widgets/framework/build_owner.0.dart example). If this works, we may have a natural API, i.e. only insert a few extra Widgets to dev's widget tree and that's all, and @dnfield's worry will be solved.
for some reason I think maybe build in one frame, layout and paint in next frame is acceptable
Agree, if that is implementable looks like also an idea, as long as the build fits in 16ms
I would like to add: this solution has less overhead than keframe. When Keframe replaces placeholder with real widget, it is driven by vsync signal to execute drawFrame process and submit to engine, so it will execute the complete build/layout/paint etc process. But build/layout/paint other than the actual widget is not necessary. In the Preempt Layout scheme, the UI thread just voluntarily submits a frame to the Engine after 16ms of detection, and then returns to the normal rendering flow without much additional overhead.
Thanks, I added it
Update: minimalist API (one widget for everything) + a prototype about framework layer.
Now, developers will only need to insert PreemptBuilder(builder: (context, child) => whatever_you_like, child: also_free_to_choose) widget, and that's all. Arbitrary builder, arbitrary child subtree, and smooth 60fps will be there for the builder. The google doc is updated to discuss this - mainly in (1) "usage examples" (2) "From preemptModifyLayerTree to PreemptBuilder" in "detailed design".
Update: minimalist API (one widget for everything) + a prototype about framework layer.
Now, developers will only need to insert PreemptBuilder(builder: (context, child) => whatever_you_like, child: also_free_to_choose) widget, and that's all. Arbitrary builder, arbitrary child subtree, and smooth 60fps will be there for the builder. The google doc is updated to discuss this - mainly in (1) "usage examples" (2) "From preemptModifyLayerTree to PreemptBuilder" in "detailed design".
@dnfield I guess your worry is now solved? 😄
Update: minimalist API (one widget for everything) + a prototype about framework layer.
Now, developers will only need to insert PreemptBuilder(builder: (context, child) => whatever_you_like, child: also_free_to_choose) widget, and that's all. Arbitrary builder, arbitrary child subtree, and smooth 60fps will be there for the builder. The google doc is updated to discuss this - mainly in (1) "usage examples" (2) "From preemptModifyLayerTree to PreemptBuilder" in "detailed design".
Code prototype:
Details
// ignore_for_file: avoid_print, prefer_const_constructors, invalid_use_of_protected_member
import 'dart:ui';
import 'package:flutter/material.dart';
import 'package:flutter/rendering.dart';
import 'package:flutter/scheduler.dart';
final secondTreePack = SecondTreePack();
// since prototype, only one [RenderAdapterInSecondTree], so do like this
final mainSubTreeLayerHandle = LayerHandle(OffsetLayer());
void main() {
debugPrintBeginFrameBanner = debugPrintEndFrameBanner = true;
secondTreePack; // touch it
mainSubTreeLayerHandle; // touch it
runApp(const MyApp());
}
class MyApp extends StatefulWidget {
const MyApp({super.key});
State<MyApp> createState() => _MyAppState();
}
class _MyAppState extends State<MyApp> {
var buildCount = 0;
Widget build(BuildContext context) {
buildCount++;
print('$runtimeType.build ($buildCount)');
if (buildCount < 5) {
Future.delayed(Duration(seconds: 1), () {
print('$runtimeType.setState after a second');
setState(() {});
});
}
return MaterialApp(
home: Scaffold(
body: _buildBody(),
),
);
}
Widget _buildBody() {
return Column(
children: [
Text('A$buildCount', style: TextStyle(fontSize: 30)),
Container(
decoration: BoxDecoration(
border: Border.all(
color: Colors.orange[(buildCount % 8 + 1) * 100]!,
width: 10,
),
),
width: 300,
height: 300,
// hack: [AdapterInMainTreeWidget] does not respect "offset" in paint
// now, so we add a RepaintBoundary to let offset==0
// hack: [AdapterInMainTreeWidget] does not respect "offset" in paint
// now, so we add a RepaintBoundary to let offset==0
child: RepaintBoundary(
child: AdapterInMainTreeWidget(
parentBuildCount: buildCount,
// child: DrawCircleWidget(parentBuildCount: buildCount),
child: Center(
child: Container(
width: 100,
height: 100,
color: Colors.pink[(buildCount % 8 + 1) * 100],
),
),
),
),
),
Text('B$buildCount', style: TextStyle(fontSize: 30)),
WindowRenderWhenLayoutWidget(parentBuildCount: buildCount),
Text('C$buildCount', style: TextStyle(fontSize: 30)),
],
);
}
}
class WindowRenderWhenLayoutWidget extends SingleChildRenderObjectWidget {
final int parentBuildCount;
const WindowRenderWhenLayoutWidget({
super.key,
required this.parentBuildCount,
super.child,
});
WindowRenderWhenLayoutRender createRenderObject(BuildContext context) =>
WindowRenderWhenLayoutRender(
parentBuildCount: parentBuildCount,
);
void updateRenderObject(
BuildContext context, WindowRenderWhenLayoutRender renderObject) {
renderObject.parentBuildCount = parentBuildCount;
}
}
class WindowRenderWhenLayoutRender extends RenderProxyBox {
WindowRenderWhenLayoutRender({
required int parentBuildCount,
RenderBox? child,
}) : _parentBuildCount = parentBuildCount,
super(child);
int get parentBuildCount => _parentBuildCount;
int _parentBuildCount;
set parentBuildCount(int value) {
if (_parentBuildCount == value) return;
_parentBuildCount = value;
print('$runtimeType markNeedsLayout because parentBuildCount changes');
markNeedsLayout();
}
void performLayout() {
// unconditionally call this, as an experiment
pseudoPreemptRender();
super.performLayout();
}
void pseudoPreemptRender() {
print('$runtimeType pseudoPreemptRender start');
// ref: https://github.com/fzyzcjy/yplusplus/issues/5780#issuecomment-1254562485
// ref: RenderView.compositeFrame
final builder = SceneBuilder();
// final recorder = PictureRecorder();
// final canvas = Canvas(recorder);
// final rect = Rect.fromLTWH(0, 0, 500, 500);
// canvas.drawRect(Rect.fromLTWH(100, 100, 50, 50.0 * parentBuildCount),
// Paint()..color = Colors.green);
// final pictureLayer = PictureLayer(rect);
// pictureLayer.picture = recorder.endRecording();
// final rootLayer = OffsetLayer();
// rootLayer.append(pictureLayer);
// final scene = rootLayer.buildScene(builder);
final binding = WidgetsFlutterBinding.ensureInitialized();
preemptModifyLayerTree(binding);
// why this layer? from RenderView.compositeFrame
final scene = binding.renderView.layer!.buildScene(builder);
print('call window.render');
window.render(scene);
scene.dispose();
print('$runtimeType pseudoPreemptRender end');
}
void preemptModifyLayerTree(WidgetsBinding binding) {
// hack, just want to prove we can change something (preemptModifyLayerTree)
// inside the preemptRender
final rootLayer = binding.renderView.layer! as TransformLayer;
rootLayer.transform =
rootLayer.transform!.multiplied(Matrix4.translationValues(0, 50, 0));
print('preemptModifyLayerTree rootLayer=$rootLayer (after)');
refreshSecondTree();
}
void refreshSecondTree() {
print('$runtimeType refreshSecondTree start');
secondTreePack.innerStatefulBuilderSetState(() {});
// NOTE reference: WidgetsBinding.drawFrame & RendererBinding.drawFrame
// https://github.com/fzyzcjy/yplusplus/issues/5778#issuecomment-1254490708
secondTreePack.buildOwner.buildScope(secondTreePack.element);
secondTreePack.pipelineOwner.flushLayout();
secondTreePack.pipelineOwner.flushCompositingBits();
secondTreePack.pipelineOwner.flushPaint();
// renderView.compositeFrame(); // this sends the bits to the GPU
// pipelineOwner.flushSemantics(); // this also sends the semantics to the OS.
secondTreePack.buildOwner.finalizeTree();
print('$runtimeType refreshSecondTree end');
}
}
class AdapterInMainTreeWidget extends SingleChildRenderObjectWidget {
final int parentBuildCount;
const AdapterInMainTreeWidget({
super.key,
required this.parentBuildCount,
super.child,
});
RenderAdapterInMainTree createRenderObject(BuildContext context) =>
RenderAdapterInMainTree(parentBuildCount: parentBuildCount);
void updateRenderObject(
BuildContext context, RenderAdapterInMainTree renderObject) {
renderObject.parentBuildCount = parentBuildCount;
}
}
class RenderAdapterInMainTree extends RenderBox
with RenderObjectWithChildMixin<RenderBox> {
RenderAdapterInMainTree({
required int parentBuildCount,
// RenderBox? child,
}) : _parentBuildCount = parentBuildCount;
// super(child);
int get parentBuildCount => _parentBuildCount;
int _parentBuildCount;
set parentBuildCount(int value) {
if (_parentBuildCount == value) return;
_parentBuildCount = value;
print('$runtimeType markNeedsLayout because parentBuildCount changes');
markNeedsLayout();
}
// should not be singleton, but we are prototyping so only one such guy
static RenderAdapterInMainTree? instance;
void attach(covariant PipelineOwner owner) {
super.attach(owner);
assert(instance == null);
instance = this;
}
void detach() {
assert(instance == this);
instance == null;
super.detach();
}
void layout(Constraints constraints, {bool parentUsesSize = false}) {
print('$runtimeType.layout called');
super.layout(constraints, parentUsesSize: parentUsesSize);
}
void performLayout() {
print('$runtimeType.performLayout start');
// NOTE
secondTreePack.rootView.configuration =
SecondTreeRootViewConfiguration(size: constraints.biggest);
print('$runtimeType.performLayout child.layout start');
child!.layout(constraints);
print('$runtimeType.performLayout child.layout end');
size = constraints.biggest;
}
// TODO correct?
bool get alwaysNeedsCompositing => true;
// static final staticPseudoRootLayerHandle = () {
// final recorder = PictureRecorder();
// final canvas = Canvas(recorder);
// final rect = Rect.fromLTWH(0, 0, 200, 200);
// canvas.drawRect(
// Rect.fromLTWH(0, 0, 50, 100), Paint()..color = Colors.green);
// final pictureLayer = PictureLayer(rect);
// pictureLayer.picture = recorder.endRecording();
// final wrapperLayer = OffsetLayer();
// wrapperLayer.append(pictureLayer);
//
// final pseudoRootLayer = TransformLayer(transform: Matrix4.identity());
// pseudoRootLayer.append(wrapperLayer);
//
// pseudoRootLayer.attach(secondTreePack.rootView);
//
// return LayerHandle(pseudoRootLayer);
// }();
void paint(PaintingContext context, Offset offset) {
assert(offset == Offset.zero,
'$runtimeType prototype has not deal with offset yet');
print('$runtimeType.paint called');
// super.paint(context, offset);
// return;
// context.canvas.drawRect(Rect.fromLTWH(0, 0, 50, 50.0 * parentBuildCount),
// Paint()..color = Colors.green);
// return;
// context.pushLayer(
// OpacityLayer(alpha: 100),
// (context, offset) {
// context.canvas.drawRect(
// Rect.fromLTWH(0, 0, 50, 50.0 * parentBuildCount),
// Paint()..color = Colors.green);
// },
// offset,
// );
// return;
// context.addLayer(PerformanceOverlayLayer(
// overlayRect: Rect.fromLTWH(offset.dx, offset.dy, size.width, size.height),
// optionsMask: 1 <<
// PerformanceOverlayOption.displayRasterizerStatistics.index |
// 1 << PerformanceOverlayOption.visualizeRasterizerStatistics.index |
// 1 << PerformanceOverlayOption.displayEngineStatistics.index |
// 1 << PerformanceOverlayOption.visualizeEngineStatistics.index,
// rasterizerThreshold: 0,
// checkerboardRasterCacheImages: true,
// checkerboardOffscreenLayers: true,
// ));
// return;
// {
// final recorder = PictureRecorder();
// final canvas = Canvas(recorder);
// final rect = Rect.fromLTWH(0, 0, 200, 200);
// canvas.drawRect(Rect.fromLTWH(0, 0, 50, 50.0 * parentBuildCount),
// Paint()..color = Colors.green);
// final pictureLayer = PictureLayer(rect);
// pictureLayer.picture = recorder.endRecording();
// final wrapperLayer = OffsetLayer();
// wrapperLayer.append(pictureLayer);
//
// // NOTE addLayer vs pushLayer
// context.addLayer(wrapperLayer);
//
// print('pictureLayer.attached=${pictureLayer.attached} '
// 'wrapperLayer.attached=${wrapperLayer.attached}');
//
// return;
// }
// {
// if (staticPseudoRootLayerHandle.layer!.attached) {
// print('pseudoRootLayer.detach');
// staticPseudoRootLayerHandle.layer!.detach();
// }
//
// print('before addLayer staticPseudoRootLayer=${staticPseudoRootLayerHandle.layer!.toStringDeep()}');
//
// context.addLayer(staticPseudoRootLayerHandle.layer!);
//
// print('after addLayer staticPseudoRootLayer=${staticPseudoRootLayerHandle.layer!.toStringDeep()}');
//
// return;
// }
// ref: RenderOpacity
// TODO this makes "second tree root layer" be *removed* from its original
// parent. shall we move it back later? o/w can be slow!
final secondTreeRootLayer = secondTreePack.rootView.layer!;
// print(
// 'just start secondTreeRootLayer=${secondTreeRootLayer.toStringDeep()}');
// HACK!!!
if (secondTreeRootLayer.attached) {
print('$runtimeType.paint detach the secondTreeRootLayer');
// TODO attach again later?
secondTreeRootLayer.detach();
}
// print(
// 'before addLayer secondTreeRootLayer=${secondTreeRootLayer.toStringDeep()}');
print('$runtimeType.paint addLayer');
// NOTE addLayer, not pushLayer!!!
context.addLayer(secondTreeRootLayer);
// context.pushLayer(secondTreeRootLayer, (context, offset) {}, offset);
print('secondTreeRootLayer.attached=${secondTreeRootLayer.attached}');
print(
'after addLayer secondTreeRootLayer=${secondTreeRootLayer.toStringDeep()}');
// ================== paint those child in main tree ===================
// NOTE do *not* have any relation w/ self's PaintingContext, as we will not paint there
{
// ref: [PaintingContext.pushLayer]
if (mainSubTreeLayerHandle.layer!.hasChildren) {
mainSubTreeLayerHandle.layer!.removeAllChildren();
}
final childContext = PaintingContext(
mainSubTreeLayerHandle.layer!, context.estimatedBounds);
child!.paint(childContext, Offset.zero);
childContext.stopRecordingIfNeeded();
}
// =====================================================================
}
// TODO handle layout!
}
class AdapterInSecondTreeWidget extends SingleChildRenderObjectWidget {
final int parentBuildCount;
const AdapterInSecondTreeWidget({
super.key,
required this.parentBuildCount,
super.child,
});
RenderAdapterInSecondTree createRenderObject(BuildContext context) =>
RenderAdapterInSecondTree(parentBuildCount: parentBuildCount);
void updateRenderObject(
BuildContext context, RenderAdapterInSecondTree renderObject) {
renderObject.parentBuildCount = parentBuildCount;
}
}
class RenderAdapterInSecondTree extends RenderBox {
RenderAdapterInSecondTree({
required int parentBuildCount,
}) : _parentBuildCount = parentBuildCount;
int get parentBuildCount => _parentBuildCount;
int _parentBuildCount;
set parentBuildCount(int value) {
if (_parentBuildCount == value) return;
_parentBuildCount = value;
print('$runtimeType markNeedsLayout because parentBuildCount changes');
markNeedsLayout();
}
// should not be singleton, but we are prototyping so only one such guy
static RenderAdapterInSecondTree? instance;
void attach(covariant PipelineOwner owner) {
super.attach(owner);
assert(instance == null);
instance = this;
}
void detach() {
assert(instance == this);
instance == null;
super.detach();
}
void layout(Constraints constraints, {bool parentUsesSize = false}) {
print('$runtimeType.layout called');
super.layout(constraints, parentUsesSize: parentUsesSize);
}
void performLayout() {
print('$runtimeType.performLayout called');
size = constraints.biggest;
}
// TODO correct?
bool get alwaysNeedsCompositing => true;
void paint(PaintingContext context, Offset offset) {
print('$runtimeType paint');
context.addLayer(mainSubTreeLayerHandle.layer!);
}
}
class SecondTreePack {
late final PipelineOwner pipelineOwner;
late final SecondTreeRootView rootView;
late final BuildOwner buildOwner;
late final RenderObjectToWidgetElement<RenderBox> element;
var innerStatefulBuilderBuildCount = 0;
late StateSetter innerStatefulBuilderSetState;
SecondTreePack() {
pipelineOwner = PipelineOwner();
rootView = pipelineOwner.rootNode = SecondTreeRootView(
configuration: SecondTreeRootViewConfiguration(size: Size.zero),
);
buildOwner = BuildOwner(
focusManager: FocusManager(),
onBuildScheduled: () =>
print('second tree BuildOwner.onBuildScheduled called'),
);
rootView.prepareInitialFrame();
final secondTreeWidget = StatefulBuilder(builder: (_, setState) {
print(
'secondTreeWidget(StatefulBuilder).builder called ($innerStatefulBuilderBuildCount)');
innerStatefulBuilderSetState = setState;
innerStatefulBuilderBuildCount++;
return Container(
width: 50 * innerStatefulBuilderBuildCount.toDouble(),
height: 100,
color: Colors.blue[(innerStatefulBuilderBuildCount * 100) % 800 + 100],
child: AdapterInSecondTreeWidget(
parentBuildCount: innerStatefulBuilderBuildCount,
),
);
});
element = RenderObjectToWidgetAdapter<RenderBox>(
container: rootView,
debugShortDescription: '[root]',
child: secondTreeWidget,
).attachToRenderTree(buildOwner);
}
}
// ref: [ViewConfiguration]
class SecondTreeRootViewConfiguration {
const SecondTreeRootViewConfiguration({
required this.size,
});
final Size size;
bool operator ==(Object other) {
if (other.runtimeType != runtimeType) {
return false;
}
return other is ViewConfiguration && other.size == size;
}
int get hashCode => size.hashCode;
String toString() => '$size';
}
class SecondTreeRootView extends RenderObject
with RenderObjectWithChildMixin<RenderBox> {
SecondTreeRootView({
RenderBox? child,
required SecondTreeRootViewConfiguration configuration,
}) : _configuration = configuration {
this.child = child;
}
// NOTE ref [RenderView.size]
/// The current layout size of the view.
Size get size => _size;
Size _size = Size.zero;
// NOTE ref [RenderView.configuration] which has size and some other things
/// The constraints used for the root layout.
SecondTreeRootViewConfiguration get configuration => _configuration;
SecondTreeRootViewConfiguration _configuration;
set configuration(SecondTreeRootViewConfiguration value) {
if (configuration == value) {
return;
}
print(
'$runtimeType set configuration(i.e. size) $_configuration -> $value');
_configuration = value;
markNeedsLayout();
}
void performLayout() {
print(
'$runtimeType performLayout configuration.size=${configuration.size}');
_size = configuration.size;
assert(child != null);
child!.layout(BoxConstraints.tight(_size));
}
// ref RenderView
void paint(PaintingContext context, Offset offset) {
// NOTE we have to temporarily remove debugActiveLayout
// b/c [SecondTreeRootView.paint] is called inside [preemptRender]
// which is inside main tree's build/layout.
// thus, if not set it to null we will see error
// https://github.com/fzyzcjy/yplusplus/issues/5783#issuecomment-1254974511
// In short, this is b/c [debugActiveLayout] is global variable instead
// of per-tree variable
final oldDebugActiveLayout = RenderObject.debugActiveLayout;
RenderObject.debugActiveLayout = null;
try {
print('$runtimeType paint child start');
context.paintChild(child!, offset);
print('$runtimeType paint child end');
} finally {
RenderObject.debugActiveLayout = oldDebugActiveLayout;
}
}
void debugAssertDoesMeetConstraints() => true;
void prepareInitialFrame() {
// ref: RenderView
scheduleInitialLayout();
scheduleInitialPaint(_updateMatricesAndCreateNewRootLayer());
}
// ref: RenderView
TransformLayer _updateMatricesAndCreateNewRootLayer() {
final rootLayer = TransformLayer(transform: Matrix4.identity());
rootLayer.attach(this);
return rootLayer;
}
// ref: RenderView
bool get isRepaintBoundary => true;
// ref: RenderView
Rect get paintBounds => Offset.zero & size;
// ref: RenderView
void performResize() {
assert(false);
}
// hack: just give non-sense value, this is prototype
Rect get semanticBounds => paintBounds;
}
class DrawCircleWidget extends LeafRenderObjectWidget {
final int parentBuildCount;
const DrawCircleWidget({
super.key,
required this.parentBuildCount,
});
RenderDrawCircle createRenderObject(BuildContext context) => RenderDrawCircle(
parentBuildCount: parentBuildCount,
);
void updateRenderObject(BuildContext context, RenderDrawCircle renderObject) {
renderObject.parentBuildCount = parentBuildCount;
}
}
class RenderDrawCircle extends RenderProxyBox {
RenderDrawCircle({
required int parentBuildCount,
RenderBox? child,
}) : _parentBuildCount = parentBuildCount,
super(child);
int get parentBuildCount => _parentBuildCount;
int _parentBuildCount;
set parentBuildCount(int value) {
if (_parentBuildCount == value) return;
_parentBuildCount = value;
print('$runtimeType markNeedsLayout because parentBuildCount changes');
markNeedsLayout();
}
void layout(Constraints constraints, {bool parentUsesSize = false}) {
print('$runtimeType performLayout');
super.layout(constraints, parentUsesSize: parentUsesSize);
}
void performLayout() {
size = constraints.biggest;
}
void paint(PaintingContext context, Offset offset) {
print('$runtimeType paint');
context.canvas
.drawCircle(Offset(50, 50), 100, Paint()..color = Colors.cyan);
}
}
Next time I may only update progress in Discord, since there are already >hundred comments there - seems everyone is there instead of in github :)
It'd help to see usage
In the example section of the google doc, updated
YourParentWidgets(
child: PreemptBuilder(
builder: (_, child) => YourFancyAnimationWhichNeeds60FPS(child: child)),
child: YourNewPageAndSoOn(),
)
)
For page transition
@dnfield
All those YourSomthing are plain old widget trees, no special
@dnfield Curious is it OK to you? Btw I am going to sleep in a minute and will reply 8hr later for later comments
A demo would help
I would rather think of an API like this would be useful
SlowSubtree(
placeholder: Center(child: CupertinoActivityIndicator())
child: MyBigScrollingPage()
)
And that child could be built over multiple frames. Main use case would be to not frame-drop during page push animation when the new page is complex. Seems like based on build_owner.0.dart example it can be possible to build arbitrary subtrees? But would need to modify the build owner in order to yield back and show the placeholder.
If someone has stackoverflow comment abilities can you please comment on this answer: https://stackoverflow.com/a/64167746/20063373 It assumes that the builder for StreamBuilder will get called 1-to-1 for the Stream events. So the code will fail depending on the timing of the Stream. It's kind of a nasty gotcha.
Filed an issue to propose the ability to make StreamBuilder lossless since the fail case is too dire imo
could you link it please?
I think if we can find a way to do multithreading nobody is against it a priori.
If I may, it sounds like your proposal boils down to "provide developers with a way to mark areas of the tree that should be updated first (doing build, layout, paint, and semantics), then, if the time runs out during a frame, send the updated layer tree that only contains those parts and not the others, then continue doing the frame". Is that right?
If so, I think the main problem with this approach is that we'd have to rerun the rarely-mentioned (because it's so cheap) "animation" step that happens before build, so that the parts of the tree that need animating early can be updated with the new time stamp, but in the current model, that requires winding down the stack frame because between the "animation" and "build" phases we run microtasks.
I also worry that it would lead to some strange effects like how to determine which parts of the tree to update and which to not. For example, suppose you have three parts to your page. Part A is marked as needing fast updates, and parts B and C other two are expensive animations.
We start frame 1, we update all the AnimationControllers, we build/layout/paint/layer/semantics the first (A), then we start on B, and we do no build and layout of B. Then we interrupt the work to send the tree to the engine with A's update. We do a new animation phase to update all the AnimationControllers again, we build/layout/paint/layer/semantics the first (A), then we resume the work on the original frame. Now we build/layout C. Finally build and layout are done so we paint B and C and do the layer tree and render it. Unfortunately, because B and C were painted after different animation phases, they'll be out of sync. It'll look like the two animations are running at a low frame rate and out of sync with each other, which IMHO is worse than today (where they're just at a low frame rate).
That's assuming we can figure out how to do the animation phase at all.
oops, sorry https://github.com/flutter/flutter/issues/112197
would be good to have a test for this.
Ahh my bad, I should have asked for a test for this one
It is still a prototype :/
I will have a more complete prototype today. Just want to confirm whether the user-facing API looks good to you or not (b/c it is an API, not an implementation detail)
If the API is still bad I need to think another
@Callum That would be great, and looks like what the modify-the-build and modify-the-layout will do (see compaison between them and my proposal, in the google doc https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko/edit#)
However, the main problem is that, seems not to exist efficient implementable solutions for those ideas (some by googlers, some by bytedancers, some by me). They have either performance problems, or logical problems IIRC. I am also looking forward to see progress on those methods!
Done (under @ch271828n account, also me)
boils down to ... Is that right?
Quite similar to my existing proposal, if it is implementable, sure.
we'd have to rerun the rarely-mentioned (because it's so cheap) "animation" step ... because between the "animation" and "build" phases we run microtasks.
Indeed I am also quite curious why it was design like that, instead of putting animation into build/layout phase. Anyway, if using the PreemptBuilder dev-facing API, dev just use a builder callback (like what we have done a million times) to drive animation, which is flexible.
Unfortunately, because B and C were painted after different animation phases, they'll be out of sync.
Sorry I cannot get it. In my proposal, they always see the same vsync signal. Indeed, using the mental model (a section in this design doc), every code should just see the plain old janky environment, except for a small portion of the code (the one inside PreemptBuilder) which knows extra info.
That's assuming we can figure out how to do the animation phase at all.
Currently I plan not to do the animation phase indeed, and just use the builder callback.
@Hixie I have tried but failed, that is why this one has no tests though others have :/ It is so deep that it is hard to construct a test, without relaxing the code visibility of the widget/renderobjects.
(I forget to ask for a test-preempt as well)
IIRC, Flutter officially supports us to using builder pattern for animations. For example, AnimatedBuilder is indeed implemented as setState per Listenable update, very similar to how we treat PreemptBuilder - build (setState?) per 60fps frame.
It's still not quite clear to me how it will work or if it's solving the problem that I'm particularly interested in (building big widget trees on slow devices)
if it's solving the problem that I'm particularly interested in (building big widget trees on slow devices) At the API level I guess yes?
YourParentWidgets(
child: PreemptBuilder(
builder: (_, child) => YourFancyAnimationWhichNeeds60FPS(child: child)),
child: ABigWidgetTreeThatYouWant(),
)
)
how it will work The code is already working (except that I have not modify engine, so it just omits later window.render to screen...), anyway I will prototype more today
Prototype: Enter-new-heavy-page, smoothly even though it takes 0.5s to build/layout
Also posted in discord and google doc
Defects in the prototype compared to the future full implementation
- The page is so heavy that even paint without the time of build and layout causes a visible jank with PreemptBuilder; in real world should not be that slow (since in real world build/layout does not take 30 frames)
- Extra frame is driven by simple
DateTime.now(instead of vsync), so it is not at its best performance - Prototype code has not been fully clean up yet
Code
https://github.com/fzyzcjy/flutter/tree/experiment-forest and https://github.com/fzyzcjy/engine/tree/experiment-smooth
Downloadable app
Video
Firstly the slow (plain old) case, then the fast (using PreemptBuilder) case. The grey circle appears when I touch the screen (by android system recorder).
https://user-images.githubusercontent.com/5236035/191970843-a9c82a38-1276-4024-8a1b-c102c9b8e22f.mp4
Prototype: Enter-new-heavy-page, smoothly even though it takes 0.5s to build/layout
Defects in the prototype compared to the future full implementation
- The page is so heavy that even paint without the time of build and layout causes a visible jank with PreemptBuilder; in real world should not be that slow (since in real world build/layout does not take 30 frames)
- Extra frame is driven by simple
DateTime.now(instead of vsync), so it is not at its best performance - Prototype code has not been fully clean up yet
Code
github.com/fzyzcjy/flutter/tree/experiment-forest and github.com/fzyzcjy/engine/tree/experiment-smooth (deliberately removed https o/w discord pop up big cards)
apk:
https://github.com/flutter/flutter/issues/101227#issuecomment-1256212759 has an attachment (the apk)
video:
Firstly the slow (plain old) case, then the fast (using PreemptBuilder) case. The grey circle appears when I touch the screen (by android system recorder).
I made the prototype 😄
/cc @Hixie @Jonah Williams @dnfield @gaaclarke @XanaHopper @Nayuta @Jsouliang @SecondFlight who had comments about the design doc (hope I remembered everyone)
I noticed that we're calling it pre-emptive rendering
but since Flutter is a reactive framework
https://reactjs.org/blog/2022/03/29/react-v18.html#gradually-adopting-concurrent-features this is what react is doing
@Piero512 Not sure which part in the link are you refer to - do you mean Fiber?
Indeed Fiber is the very beginning: https://github.com/flutter/flutter/issues/101227
But my current solution is indeed quite different from fiber (indeed little similarity)
it seems to break the specified transition duration, is it expected?
Break what?
like, if a transition was meant to be (say) 400ms then that constraint is not respected anymore
Should be 400ms
If not, it is a bug
This is prototype indeed, so please bear some bugs 🙂
hm hm i see
was asking out of curiosity indeed, cuz was wondering if it was expected or no
I see 🙂
Your app should be exactly like what you expect, except that you have extra smooth frames, when using this PreemptBuilder
is that dilating the time of the animation? What is supposed to be the duration of that transition?
It looks like a bug in my prototype. I use DateTime.now() everywhere (since this is a prototype), so it is weird that it dilates so much. It should be 0.3s.
I will debug it soon
Btw, I realized that, the video is only <30FPS (even if recording non-flutter apps), so cannot demonstrate the real case... Will change a software.
Update (2022.9.24 22:00 @ UTC+8): Given that you guys seem to be on weekend, I will not post details to avoid disturbing. But if you are interested in my today (and tomorrow)'s progress, all my progress can be seen in realtime in the github branches posted above. Have a good weekend! 🙂
Brief visual update: It runs at ~60fps, while widget build/layout needs ~500ms
Video description: (1) The slow (plain-old) case is repeated twice (2) Then the fast (using PreemptBuilder) case is done twice (3) Lastly a debug animation is shown (to be explained below).
How to verify it is 60fps: I personally use ffmpeg -i $VIDEO -vsync 0 -frame_pts true -vf drawtext=fontfile=/usr/share/fonts/truetype/freefont/FreeMonoBold.ttf:fontsize=80:text='%{pts}':fontcolor=white@0.8:x=7:y=7 ~/temp/video_frames/output_%04d.jpg to extract every frame of the video.
P.S. The last section in the video (debug animation) is used to verify the file transfer. If that part is seen janky, then it is probably a problem when transferring the video file etc, since that should definitely be 60FPS.
As usual, the code is at the GitHub branch mentioned above.
Brief visual update: It runs at ~60fps, while widget build/layout needs ~500ms
X-Posted: https://discord.com/channels/608014603317936148/608021234516754444/1023410732336939129
Video description: (1) The slow (plain-old) case is repeated twice (2) Then the fast (using PreemptBuilder) case is done twice (3) Lastly a debug animation is shown (to be explained below).
How to verify it is 60fps: I personally use ffmpeg -i $VIDEO -vsync 0 -frame_pts true -vf drawtext=fontfile=/usr/share/fonts/truetype/freefont/FreeMonoBold.ttf:fontsize=80:text='%{pts}':fontcolor=white@0.8:x=7:y=7 ~/temp/video_frames/output_%04d.jpg to extract every frame of the video.
P.S. The last section in the video (debug animation) is used to verify the file transfer. If that part is seen janky, then it is probably a problem when transferring the video file etc, since that should definitely be 60FPS.
As usual, the code is at the GitHub branch mentioned above.
https://user-images.githubusercontent.com/5236035/192124851-19bae792-ad31-4ae3-8717-8a0821038d00.mp4
https://user-images.githubusercontent.com/5236035/192124851-19bae792-ad31-4ae3-8717-8a0821038d00.mp4
VsyncWaiter schedules unneeded extra AwaitVSync callbacks for one frame
Consider the following sequence:
- ScheduleSecondaryCallback(id1, callback1)
- ScheduleSecondaryCallback(id2, callback2)
Then, without this fix, the AwaitVSync will be called twice. Then, FireCallback will be called twice at the beginning of the next frame. But we know that FireCallback calls all primary and secondary callbacks, so being called twice just waste some computation resource.
By the way, the following sequences are all ok without bugs:
- AsyncWaitForVsync
- ScheduleSecondaryCallback(id1)
- ScheduleSecondaryCallback(id2)
or
- ScheduleSecondaryCallback(id1)
- AsyncWaitForVsync
- ScheduleSecondaryCallback(id2)
List which issues are fixed by this PR. You must list at least one issue. https://github.com/flutter/flutter/issues/112439
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Brief update in recent experiments: Overhead per frame is medium=0.53ms, p95=0.81ms, p99=1.10ms, on my low-end testing device, for the "enter-new-screen" demo. (Indeed it may not be called "overhead", since those time are really needed to update data for screen)
Latest video (if you are interested :))
https://user-images.githubusercontent.com/5236035/192254354-e65a8bd2-9f49-4c5b-acdf-eda3932402f9.mp4
(This comment is also linked from https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko/edit#, i.e. Preemption for 60 FPS (PUBLICLY SHARED) (4).pdf)
Is build progress occuring during animation here? Because the effect could be replicated by just delaying the complex content build for ~500 ms (animation duration).
@moffatman
Is build progress occuring during animation here
Not completely get the question... If the question is, whether animation happens when the complex widget is being built/layouted, the answer is yes.
Because the effect could be replicated by just delaying the complex content build for ~500 ms (animation duration).
You need some extra preempt points, instead of a single sleep(500ms). For example, this should work:
build() {
for(var i=0;i<100;++i) {
Actor.instance.maybePreemptRender();
sleep(const Duration(milliseconds: 5));
}
return your widget;
}
Indeed, preempt points are auto injected via PreemptPoint, and (possibly done in the real PR) in every RenderObject.layout. So usually no need to manually write that.
By the way, your original modification does not work, because by default I do not expect a single widget.build to exceed 16ms. Anyway if that is the case just insert a few maybePreemptRender.
Good morning friends! Should I start making the PR? @Hixie @Jonah Williams @dnfield @gaaclarke @Callum @ping (googlers who had discussions about this proposal)
The design doc has been overhauled:
- Add chapter "experiments (prototype)", showing videos, code demos, FPS analysis, and overhead analysis.
- Add section "needed code change", demonstrating what framework and engine code (few) needs to be modified.
- Overhaul chapter "comparison", elaborating the differences.
I want to sincerely say thanks to Flutter. Flutter has saved me months, if not a year, by allowing me to write a single codebase and run on dual platforms (saving half of the time), to use the very productive declarative framework and hot reload, to utilize the quickly-progressing Dart language with expressive and safe type system, to easily customize the appearance as accurate as pixel level, to be able to dig down and modify engine code when I need a new feature (impossible with Web), and many more.
That is a big reason why I decided to contribute this PR (Preemption for 60 FPS) to Flutter. Flutter has saved me so much time, so now it is my turn to provide my time to Flutter to make it better. I also love open source very much, such as my flutter_rust_bridge open-source library (which could have been closed-source and only used by myself), and my previous PRs to Flutter fixing bugs, which is another driving force.
(overhauled design doc) https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko
i still don't understand how you run the animation phase in your proposal. (the discussion in the last comment in the doc)
It's unclear the mechanism that is used to resume layout. Are we really resuming or retrying? In order to resume the whole stack would have to be captured (ie a continuation).
I'm unfamiliar with how keframe works. Would this make that library obsolete? Does it handle all the cases it does?
What happens if we can't update the existing tree in the 1.667ms we have then? How do we select this number 15ms?
@Hixie Hi I add a section "How is animation implemented" just now. Hope I explain it clearly now!
(last section in the "detailed design")
ah, i see
so it only works for animations driven from tickers
this whole design seems super fragile to me
My proposed method is not sensitive to the threshold indeed. In other words, suppose you happen to pause at 15ms, then you do not necessarily need to finish UI thread work and submit to raster thread in 16.67-15=1.67ms. On the contrary, you can, e.g. finish in 2ms or 3ms or 4ms or whatever, as long as raster thread have enough time to finish his hob.
Some details can be found in the "when to call preemptRender" section
May I know what other kinds of animations exist in Flutter?
I personally have only seen those from tickers
could be anything
Hmm could you please point out where so I can try to improve?
what i mean by fragile is that there's lots of ways you could write a flutter app that violates the assumptions made here
If I understand correctly, AnimatedBuilder, AnimatedWidget, FooTransition, TweenAnimationBuilder etc all use vsync and tickers
like, for example, what if two widgets communicate via some other object, and one is in one of these interruptible subtrees and the other isn't. they'll be seeing different phases at different times
Dev only needs to pay attention to the code inside PreemptBuilder.builder, which should be a little of code (instead of a lot)
That builder is only for smooth animations
not for very huge heavy things 🙂
that's what i mean by fragile :-)
something that isn't fragile would work in many different scenarios
Sure I agree with that. But it seems to solve the smoothness at 60fps problem
Everything comes with a cost :/
Especially with the gain of "60fps, no matter how heavy your subtree is to build and layout"
well, it doesn't really. it solves the "subset of your tree at 60fps" problem :-)
The "user seeing the UI at 60fps" 🙂
sort of. parts will be 30fps :-)
anyway
is there a way to build this in a package?
parts that does not need change indeed so users never know it 🙂
what does it actually need from the core framework?
(btw why you can type that face while mine will be a yellow face)
Yes and no, the details:
i suppose you need to hook into every build/layout
The "needed code change" section
So maybe I should PR those framework changes, and then make a package? Or should I PR the whole thing?
some of these changes are sort of fundamentally opposed to flutter's design philosophy like, for example, the widget system shouldn't need to know about tickers a developer should be able to build an entirely separate animation system and just layer it on top of the framework but if we do what this proposal suggests, we're really saying that tickers are very special in a core sense
Or a way to preempt an isolate, creating a continuation that can be resumed, but that probably isn't going to happen.
yeah that seems even more fragile
Btw, not sure what exact problem will they face? The widget in main tree see one ticker per frame. The widget in second tree see many ticker ticks per frame. And they see the same microtasks and event loop run.
i dunno, it's hard to predict. that's what i mean by "fragile".
IIRC I have had some thoughts a bit similar to that before (see that github thread) also without success
I see. Just want to have one example in my mind, really cannot imagine.
If there are zero examples then just cannot know what to do/worry with it
i totally understand that my concerns are unsatisfying
It's totally OK, I just need some input to know the concerns better
such that I can try to figure out a way to solve it if it is a problem
let me see if i can think of a good example
I hope I'm not making an oot question: isn't the PreemptBuilder proposal trying to solve what Impller also aims to solve? (= remove jank on animations by pre-compiling stuff)
Well seems not. Impeller solves raster thread jank, PreemptBuilder solves build/layout jank in ui thread
One piece of feedback that is actionable is imaging what it would take to make this a plugin.
Clear, thanks!
Yes I also think it is a feasible idea. The "Needed code change" section is about "what needs to change framework/engine"
Indeed not many changes to framework/engine
(That section, plus a few "make file-private function public but do not need to change any real functionality")
@fzyzcjy so for example, suppose instead of using a Ticker i have an animation that's driven by a Timer. every time it triggers, it changes some global state. then i have many widgets that follow that global state. with your proposal, the sections inside the special widget would not animate fast.
Thanks for the example. However, IIRC "animation by timer" is discouraged?
So if a user uses a discouraged approach he gets bad results
writing a widget tree that takes more than 16ms to build is also discouraged, but we're still trying to help people who do that :-)
Well that is indeed mandatory, because:
In other words, a dev trying the best to follow Flutter suggestions still may face the build/layout-more-than-16ms problem
Such as bytedance
And there are just so many really slow devices in the world indeed
it's really important for us that flutter be something thet's predictable and easy to understand
It is just impossible to be smooth on all slow, slower, slower than slower devices (without the proposal). layout/build really needs some time, and on them it can exceed 16ms
I totally agree
for well-behaved users 😉
for everyone
because people don't necessarily know what best practices are
Hmm I see
and they will never be able to learn if they get frustrated with a system that isn't helping them when they're new
such that they just give up
totally agree
but I am not very sure will a new learner really use PreemptBuilder?
yes, because they'll google around "how to help slow app" and they'll find a youtube video that talks about it and they'll try it
or should we add some doc to PreemptBuilder saying the care needed
Ah that is quite reasonable
or use the alternative approach you mentioned above - I publish a package about it
then in the package frontpage I can just have a big warning saying "hey don't use timers" and so on
another example would be, suppose there's two RenderObjects and they each create a Layer and those Layers know they will always be used together because the RenderObjects are always used together. So they can rely on always existing as a pair and can always read the render object sizes and so on when the layer tree is walked. now suppose one of those RenderObjects is in the "expensive" part of the subtree and the other is in the "fast" part of the subtree. or suppose one is in the "expensive" part that got laid out before we interrupted layout, and the other is in the part that got laid out later.
That looks like a problem as well. But may I know how layers read RO sizes? IIRC layers do not remember ROs
they can do whatevery they want
they're just code
That sounds like a problem
For expert users who uses custom layers
another example would be GlobalKey reparenting. Suppose the section of the subtree on the "fast" branch tries to move a widget from the section of the tree in the "expensive" branch using GlobalKeys.
I see, that will definitely not work.
But IMHO no "animation" requires moving it
But I agree we should be friendly to all edge cases
fundamentally the problem is that your proposal violates some of the core assumptions of the framework
such as, that we'll always do a single build/layout/paint/layer/semantics pass per frame
I agree, it is just the least violation I can find
I refine to: per tree per pseudo or real frame. fast tree it is 60fps pseudo "frame"
I have also tried other methods and there are previous thoughts about other methods as well, but seem they have drawbacks or cannot work
(See "Comparison" chapter)
any time you violate core assumptions, you have to either very carefully think about what the new assumptions should be that we can make sure the entire model follows these new assumptions, or things will become fragile
totally agree
(it's often extremely hard to do this)
think so :/
the problem with a system like flutter's, which has many years of work already done on it, is that there's a lot of these assumptions and it's easy to just not know about some of them. e.g. i'm sure i don't know them all at this point.
totally agree about that
so should I just make a package and publish to pub.dev?
with minimal necessary modifications to framework and engine
it would be interesting to examine what the minimal changes would need to be
mainly this
the list currently in the doc is probably not minimal enough to justify doing
Let me think whether it is possible to remove some
for example, i definitely don't think we should elevate Tickers in this way
and adding a hook to every build/layout step is going to be a cost everyone would pay even if they don't use the feature
which is probably not something we'd want to do
especially when the goal is to make things faster :-)
May I know why?
we could potentially allow people to swap out the layout function with zero cost or one layer of indirection at the top of the tree
I am not expert in compiler, but will it introduce runtime cost if the hook is always null? (Will compiler throw it away)
Or maybe we can do what @gaaclarke suggests, say create a Renderer.layout(RenderObject ro) and I just class MyRenderer extends Renderer
flutter's tried to follow a "layered" approach, where there are very few "core" components that one must use. For example, you can use RenderObjects without widgets. You can use widgets without material. You don't have to use WidgetsApp if you don't want to. You don't have to use Tickers if you don't want to. etc.
if there's a way to do that that's zero cost during normal operations, that's worth considering
i'm sure lots of packages could benefit from that kind of hook
Oh I see maybe I do not explain clearly. We do notexpose all tickers. We only expose those who are created inside SingleTickerProviderStateMixin
We already let SingleTickerProviderStateMixin to read TickerMode (inherited widget) so looks like we are just doing something mimicking the existing
Then maybe I should try that class Renderer and report some metrics later
right my point is that TickerMode is just a widget. you don't have to use it. SingleTickerProviderStateMixin is just a tool, you don't have to use it. But if we say that SingleTickerProviderStateMixin now exposes a special API, we're saying it's special in a way that MyCustomSingleTickerProviderStateMixin is not, and can never be.
Sorry I do not quite get it. I am trying to modify SingleTickerProviderStateMixin in a ways that, originally it calls TickerMode (inherited widget), not it calls TickerMode + TickerRegistry (another inherited widget)
Just read an inherited widget, completely mimicking what we do to TickerMode
(code may not be exactly mimic though if merely looking at the screenshot; but digging down one or two function calls we see it is the same)
what would TickerRegistry do?
It knows all tickers in the subtree and created by TickerProviderStateMixin
so now if i want to make a new kind of Ticker, how do i register it?
Then in PreemptBuilder (e.g. in 3rd package), we just call all of those tickers's onTick
MyCustomTicker, which has no interfaces in common with Ticker
Just like how you register it with TickerMode
TickerMode doesn't know about Tickers
it just reports a boolean
and notifies you when it changes
Then what about making TickerRegistry receive an abstract class (interface) instead of the real Ticker class
Or , I know it:
Just do not let TickerRegistry know the tickers
so then what does it do?
Instead, let it (maybe w/a rename) provide addListenerWhenOtherPartsOfSystemWantsToCallAnExtraOnTick
So now the MyCustomTicker (not extend/implement Ticker) is happy
e.g. named "ExtraOnTickProvider"
basically this makes ExtraOnTickProvider a special class
May I know why it is special?
anyone who wants to participate in this model has to make sure they can express their animation logic as an ExtraOnTickProvider
Well, again, is there real examples who create CustomTicker unrelated to real ticker logic...
If so he must be an expert
And experts should understand how PreemptBuilder works 😉
as we talked about before, that's not a safe line of reasoning
non-experts will be exposed to all APIs
I totally agree we should be friendly to new users watching a youtube video
"only experts will use it" is never a valid way to design APIs
because everyone is a non-expert the first time they use an API
Well, ok :/
I just mean not sure who will really need this
Ok then let me think about other ways to expose Tickers
fwiw i think this is something we could have done from the beginning, basically instead of the animation phase which relies on scheduleMicrotask as the special magic, we would have a registry of OnTickProviders that provide the special magic
Totally agree. But seems we can never change it today? 😦
but having now gone the former route, moving to the latter route either means having extra complexity (because we have both), or means taking on the rather large task of migrating the entire ecosystem to the new mechanism
sometimes we have done that kind of thing
I can contribute
e.g. at one point State.widget was called State.config and someone (i forget who?) renamed it and basically migrated the entire ecosystem to the new name
but that was a long time ago and the ecosystem is much bigger now
that's true
(and congratulations!)
So what about this:
Change SingleTickerProviderStateMixin.createTicker . Do not ticker = Ticker(). But instead ticker = TickerProviderConfigInheritedWidget.of(context).createTicker()
In other words, we have a new TickerProviderConfigInheritedWidget
it just configures how TickerProviderStateMixin create tickers
SingleTickerProviderStateMixin.createTicker is not the only way animation triggers are created
When there is no such inherited widget, just do the old logic: new Ticker()
May I know what are the others?
Timer.periodic, for example
That just will not be supported
Streams are another
Since these logic are in my 3rd party package I guess it is not a huge problem
just write down "Stream and Timer are not supported" in README
of the 3rd package
yeah it's a lot easier if it's a package than the core framework
And also easier to upgrade as well (no need to wait 3mo for next stable flutter if someone sees a bug)
So looks like my next step is to create these 4 PRs to Flutter?
i think for each one we should carefully consider if there are potentially better ways to approach it
Sure 🙂
but that's definitely more tractable than the earlier list :-)
Feel free to raise any potential problems and I will try to fix them
i haven't studied the engine changes yet
i'm still looking at the ticker one :-)
I see, take your time 🙂
the problem you're trying to solve is, how to cause CircularProgressIndicator to tick, even though we haven't had an animation phase, right?
yes
and _CircularProgressIndicatorState uses an AnimationController with a Ticker created from a SingleTickerProviderStateMixin
yes just like that
and ticker uses SchedulerBinding.instance.scheduleFrameCallback
interesting
exactly
and the problem is that after you render your interrupted frame, you want to rerun all the newly scheduled frame callbacks, so that when you repaint this widget, it ends up advanced a little...
not all indeed
only those in second tree
that's why I make inherited widget - I put one at root of second tree
ah, even trickier
because I do not want to disturb the main tree
i certainly see why you gravitate to a way to create and/or register tickers via inherited widget
ironically it would be easier to solve if you wanted to just call the scheduled frame callbacks of every animation
I agree. But that will cause trouble since main tree will receive extra ontick
because then you could just create a new binding...
hmm
hmm looks hard to make two bindings and let things in second subtree smartly register to second binding
yeah the binding logic knows nothing about the trees
yeah i dunno how to do this efficiently. i don't think we want to change SingleTickerProviderStateMixin et al to register their tickers, that seems like a lot of cycles spent that most people would never get to benefit from. (That said, if you can find a way to hook into layout cheaply, maybe we can expose a hook for SingleTickerProviderStateMixin too?)
tell me more about the engine changes?
a lot of cycles spent that most people would never get to benefit from. (That said, if you can find a way to hook into layout cheaply, maybe we can expose a hook for SingleTickerProviderStateMixin too?) I will experiment to see whether performance regresses
So, is it enough to see benchmarks built into flutter repository? If they do not regress are we safe
i mean to be clear, performance will definitely regress if we do more work, even if we can't measure it
That's definitely true
we can't just add code that doesn't measurably affect benchmarks because if we did that 100 times then we would have moved the benchmarks
totally agree. what about this: we just have a global flag, enableTickerConfig. If it is false, do the old thing. If it is true, read Inherited Widget
And in my 3rd party package, one setup step is "set enableTickerConfig=true"
and by default false
Or, we just do not call inherited widget at all
we call a function, say: Ticker Function(BuildContext context, VoidCallback onTick) tickerCreator = (_, onTick) => Ticker(onTick)
it by default is nothing but Ticker.new, and we can set it to read the inherited widget
And I guess compilers may even inline it, if that createTheTicker is never setted?
It is in the screenshot. I can explain more if something is unclear
Indeed we are unconditionally reading inherited widget (b/c the TickerMode) whenever we create one Ticker.
And I guess reading inh widget is much much more expensive that a function call that can possibly be inlined
Though I know a little accumulates to a lot
Or, for absolutely zero overhead: Maybe enable it by bool.fromEnvironement flag. IIRC those are compile time constants, and compilers will just handle them at compile time. for example, kDebugMode ? heavy_work : cheap_work, the heavy_work seems even not in the final binary
Users of the 3rd party package will need --dart-define=enableTheTickerConfig=true or something like that. And other users have exactly zero overhead.
Hope this is a bit clearer:
i meant tell me more about why those specific changes
it sounds like what you are trying to do is allow frames to be rendered from a different callback than the one that asked for it, right?
hm, this is another one of those cases where there's some pretty fundamental assumptions built into the system that we would be breaking here, and need to think very carefully about
that has the problem of being hard to test (we would need to run all the tests for every combination of these flags, which gets exponentially expensive)
I see the problem. What about this:
bool? debugOverrideTheFlag;
bool get theFlag {
var ans;
assert(() => ans = debugOverrideTheFlag);
return ans ?? bool.fromEnvironment('the.flag');
}
It has zero overhead in release (given it is just a compile time constant). And it has testability (just debugOverrideTheFlag)
"Why" added to the doc now
I guess no? i.e. I do not violate it?
I am still calling window.render inside the BeginFrame callback, because the preemptRender is a function called from build/layout functions which is called from BeginFrame. I just call it multiple times (instead of one time).
So I guess I do not break this
but then presumably we would not call it for the next actual BeginFrame, right? since we'd have already done it
May I know what is "it"?
If "it" is "Produce", then that still calls in next BeginFrame
The logic is, at Render, when we see no continuation (this happens when one BeginFrame has two window.render), originally we just halt early
But now, when this case, we add one extra continuation via Produce().
We will finish this Produce() just a few lines below. So in next BeginFrame we need to Produce() again (for the next frame's Render)
the blue-highlighted lines are "finish the Produce continuation" logic
it=window.render
Oh, then we still call it in the next frame
Because in my proposal, one plain-old frame will have zero to many extra smooth window.render
But that plain-old frame is just there. It runs normal full pipeline
including window.render inside that pipeline
oh I see, we just skip the BeginFrame for "missed" frames that this would partially render
If a frame is missed, the whole pipeline just do not execute, and the proposed PreemptBuilder etc also do not execute
how can we know the right timestamp for the "fast" frames if we don't get the BeginFrame call?
via the last PR among the four: Get latest vsync data
Indeed only need to read "what is the latest vsync data" once per 16ms
Btw I have confirmed the time is correct in the experiment analysis section
in this (photos from a camera mp4 video), we can see the animation is of equal distance
i.e. the arrow shifts the same distance in each frame
If we have the wrong vsync info (or use something like DateTime.now), then we will see the distance not equal at all
how would the system be notified that it had changed?
Just no notification
preemptRender read it and it works well
Indeed there cannot be notifications - because UI thread is fully busy and PostTask to UI thread will not work at all
If I understand correctly, compiler explorer says this works well with zero overhead:
In the first example (with proposed Dart code), the heavyFunction is even not compiled into the binary. In the second example (just as a comparison), the heavyFunction is compiled and conditionally called.
In other words, my proposed Dart writing seems to be (1) zero overhead (2) easily testable.
For the RenderObject.layout thing
- I do see one extra function call (from RenderObject.layout to Renderer.layout)
- If that is acceptable we are done; otherwise, we may enable it conditionally via zero-overhead flag just like mentioned above, then we have exactly zero cost
Exposing hook about tickers with zero overhead
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
This PR is a part for implementing the 60fps smooth rendering (#101227).
Some discussions can be seen in Discord, such as around https://discordapp.com/channels/608014603317936148/608021234516754444/1024141682377236500.
List which issues are fixed by this PR. You must list at least one issue. https://github.com/flutter/flutter/issues/101227
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
P.S. Not sure what naming do you like, so just put a very long (temporary) variable name here :)
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Exposing hook for RenderObject.layout with zero overhead
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
Remark: This PR has less priority compared with the other two (https://github.com/flutter/engine/pull/36438, https://github.com/flutter/flutter/pull/112436), because this one can be workaround, while the other two are really mandatory to implement PreemptBuilder.
This PR is a part for implementing the 60fps smooth rendering (#101227).
Some discussions can be seen in Discord, such as around https://discordapp.com/channels/608014603317936148/608021234516754444/1024141682377236500.
List which issues are fixed by this PR. You must list at least one issue. https://github.com/flutter/flutter/issues/101227
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
P.S. Two PRs about the framework change is created: https://github.com/flutter/flutter/pull/112436
Allow render to be called multiple times for one BeginFrame
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
This PR is a part for implementing the 60fps smooth rendering (#101227).
List which issues are fixed by this PR. You must list at least one issue. https://github.com/flutter/flutter/issues/101227
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
The only change is an "if" as follows (all else are just tests)
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
By the way, the tests and code does work: If I comment out the code, the tests fail.
Ah sorry I did not see all your questions! I only see the last, and when viewing the second-last I see my avatar so wrongly think all things below have been answered...
We do not resume or retry. We just call preemptRender function as any normal function call, and just return from it. So zero cost.
I have not done thorough experiments (e.g. ListView scrolling) so cannot give a conclusion now. But it seems this package will cover all cases with less drawbacks and better performance, by solving the problem in a different approach.
@gaaclarke I replied to all your questions now in google doc (Sorry I did not see all your questions this morning... I only see the last, and when viewing the second-last I see my reply there so wrongly think all things below have been answered)
Any sliver expert: Is it expected to provide a custom SliverChildDelegate for good performance? Since with the default SliverChildBuilderDelegate, all the children will be rebuilt if the parent is rebuilt.
@Callum only the visible children, iirc, right?
Yeah that's true. For reasons, I have both pages rebuild during page pop, so the frame drop was quite noticeable. Both lists didn't need to rebuild as no change in the content, it's a good feature that was not immediately obvious.
I'd be really surprised if there is a significant framedrop from just rebuilding widgets. Do you have some sample code I could look at? I've been looking at scrolling performance issues the last couple of weeks, usually what goes wrong is folks accidentally making the entire list render (even offscreen) and that can be quite slow
The list items are quite complex paragraphs, during scrolling it can be okay to have 1-2ms build times. But if 24 of them build at once (12 on each page) during the page pop, it's going to drop frames.
what sort of device are you running on?
also, if its the same paragraph, the layout should be cached in the engine
Just looked into it, my paragraphs aren't getting cached because of my use of WidgetSpan, unless it's the exact same Widget, the paragraph gets re-laid-out.
oh, well that seems like a footgun
do you mind filling a bug on that? We should figure out how to make that work...
@fzyzcjy I believe this just needs a test (or probably just a test exemption) to land.
@goderbauer I agree. Not sure how it can be tested though, so I will ask a test-exempt.
I will do it maybe a few days later since I am working on https://github.com/flutter/flutter/issues/101227 (and on discord), and do not want to have too many things on discord in one day :)
Thanks for your reply!
Good morning/evening friends! Three (small) PRs are created yesterday, with tests and green CI: https://github.com/flutter/flutter/pull/112436, https://github.com/flutter/flutter/pull/112437, https://github.com/flutter/engine/pull/36438. May I get a little bit review 🙂
Not sure whether I should "@" some people here, maybe @dnfield @jonahwilliams @gaaclarke @flar engine experts?
May I get a code review, thanks :)
@dnfield Hi, thanks for the quick reply :)
The documentation on FlutterView.render specifies when it is safe/allowed to call render.
I will change that doc accordingly (probably after we come to a conclusion what should be done for this PR)
I'm not quite clear on how this will affect the pipeline - it seems like it will now be trivial for a dart:ui application to flood the pipeline if we remove guardrails around when you can call render. Today the contract is that the application can expect that it's time to call render because it got a call to onBeginFrame. In this world, the application calls render whenever it thinks it has been working too long and might want to give an update. But the application doesn't know about vsync and it will be very hard to reason about why render is getting called if we make this change. I don't think we should make this change. It too easily allows wasted work to happen.
Firstly, IMHO, a normal flutter app calls window.render once per frame, so no problem at all for all existing app.
Secondly, in my proposal (Preempt for 60 FPS), I do observe vsync (using VsyncAwaiter class), and only submit one window.render per vsync. Therefore, "But the application doesn't know about vsync" seems not to be the situation, and thus "it will be very hard to reason about why render is getting called" is also no problem.
That said, I do agree that, if the rasterizer thread takes too long (e.g. takes 50ms for one rasterize), it is a waste to submit a Scene per 16.6ms (but should submit per 50ms).
If this is still a problem for you, can I change as follows: Add a flag to window.render, say, window.render({bool onlyRenderOncePerBeginFrame = true}). Then the behavior will be exactly the same, except for someone who really needs this (e.g. the Preempt proposal).
In addition, given it is a so low-level API that most people will never touch, it seems reasonable to provide some flexibility to it.
@dnfield Another possibility for Dart code to understand the queue is full so it do not do anything more:
Add this 4-line function:
// return: whether it is prepared successfully. If return false, it means pipeline is full,
// and thus the user should not really compute the Scene to avoid unnecessary work.
bool Animator::PrepareExtraRender() {
if (!producer_continuation_) {
producer_continuation_ = layer_tree_pipeline_->Produce();
}
return static_cast<bool>(producer_continuation_);
}
usage:
realize_next_vsync_comes; // see the design for details https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko/edit
var prepared = window.prepareExtraRender();
if (prepared) {
ui.Scene scene = compute_the_scene();
window.render(scene);
} else {
// do not do anything since the rasterizer queue is already so full
// this mimic the behavior of Animator::BeginFrame, where we skip the current frame if it is full
}
@dnfield ... be trivial for a dart:ui application to flood the pipeline ...
IMHO the pipeline seems not to be flooded - it has depth 2. In other words, even if we call window.render(scene) a million times within a frame, only the first two scenes will be in the queue, and the rest 999998 will just be thrown away (suppose rasterizer has not processed any). So we are still safe.
For a dart:ui application, if needed, it can use the window.prepareExtraRender extra call to see whether the queue is already full, to avoid generating scene etc (just like example above).
Today the contract is that the application can expect that it's time to call render because it got a call to onBeginFrame. In this world, the application calls render whenever it thinks it has been working too long and might want to give an update.
Just as mentioned above, adding a flag like window.render({bool onlyRenderOncePerBeginFrame = true}) seems to solve the "contract" problem.
Oops sorry @chinmaygarde and @iskakaushik I just clicked the "Icons.refresh" on dnfield and do not know why github remove review requests to you...
Probably we can avoid it if we figure out the widget span has the same dimensions as last time... But If it won't or we can't figure it out the text layout may have changed.
Reviews typically happen once per week during triage meetings. I've looked at some of these PRs already though and there seems to be some missing context here. These changes don't seem quite safe on their own, and it's still not clear to me what's the bigger picture app that they fix. I think we've talked about having a sample application or benchmark that shows what you're improving - is that available (even if it requires some special patches to the engine or framework to run, that's ok)
I'll say right now though that, in their current form and without extra support, these patches are unlikely to land anytime soon.
Hi, for sample app, with video + full code + brief code + benchmark + analysis, please have a look at https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko/edit#, the new "Experiments" chapter
Thanks, I did discussed with Hixie on discord and wrongly thought that public discussion was enough. I will fill all those contents in a few hours.
I will let it not be "in their current form" by providing extra support doc now 🙂 Will tell you when finished (probably in a few hours)
By the way, those two PRs are the most important (i.e. package cannot exist without them), so if you are busy please ignore my other PRs currently
Other things need time on this thread, for example GC.
I'm saying that a single layout function might take too long and your preempt call will come too late.
https://github.com/fzyzcjy/flutter_smooth/blob/master/packages/smooth/example/lib/main.dart#L165 is a good chunk of the hard part. I'm not really clear from this document how that would automatically get inserted in meaningful places without breaking a lot of things.
Added a couple more comments.
@dnfield
What is the big picture that it fixes?
The ultimate goal is, let the app run smoothly at 60FPS, even if it has heavy subtree that is very slow to build/layout. In other words, the design doc: https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko
It already has a working demo. See the (new) "experiments" chapter, with a video, screenshots, full code, brief code, benchmark, analysis.
What's the main goal you're trying to achieve with this particular change?
In order to solve that big goal, we must let animation callbacks run at 60FPS even if the whole tree is very slow to build/layout. Otherwise, even if we refresh a subtree by 60FPS, anything like CircularProgressIndicator, FooTransition, or manual AnimationController will all never be smooth, because they do not see new timestamp at 60FPS.
Then, to fire (extra) animation callbacks, a natural solution is to work with the Tickers. Originally, Tickers are fired once per frame. But now, we also extra fire it (with proper timestamp) in each 60FPS smooth extra frame.
Lastly, to fire extra events to Tickers , we must know the existence of Tickers in the auxiliary widget subtree (no need for Tickers in the main subtree since they should not be fired at 60FPS). That is why I added a callback when Tickers are created - then I can record it such that to fire extra ticks.
Why is it doing it this way?
Why it is a compile time flag FLUTTER_ENABLE_TICKER_PROVIDER_STATE_MIXIN_CREATOR: Because Hixie is worried about performance loss (in Discord hackers-framework). Making it a compile time flag, then nobody will have any even tiny bit of performance loss, if they do not need this feature.
Why there is debugOverrideEnableTickerProviderStateMixinTickerCreator in addition to compile time flag: Because Hixie said compile time flags are hard to test. By using this debugOverride... flag we can test it easily (indeed I have written tests there).
Why a context must be passed to the callback: Because as mentioned above, I need to determine whether it is in the second subtree or main subtree.
Btw the name is temporary, just suggest any name you like :)
Why can't we used existing mechanisms to achieve the same thing?
Well, Hixie and I have tried, but cannot come up with a solution :/ Feel free to suggest solutions! Microscopic speaking, seems that cannot know a Ticker in a TickerProviderMixin is created so cannot gather them. Macroscopic speaking, did not find other ways to let it be smooth.
Why do we need this mechanism to achieve the larger goal of having incremental/progressive layout?
Hope this question is clear with above ;)
Reply done to GitHub "hooks" PR: https://github.com/flutter/flutter/pull/112436#issuecomment-1261568241
I agree theoretically. But during my experiments, I see about 39% of the UI thread time is idle, and I guess GC does not need that much time. This screenshot: https://user-images.githubusercontent.com/5236035/190553863-5a373dcb-75ba-468d-8118-66e7a393070b.png
I really appreciate your enthusiasm for this topic! But I'm still not sure I understand the big picture purpose of this method. You've explained some of the specifics about why you're guarding certain things the way you are, which isn't really what's unclear to me. What's unclear to me is why we want tickers to have an artificial way to fire an extra tick.
Just add maybePreemptRender to that single layout function. For example:
class VeryHeavySingleLayout extends RenderObject { void performLayout() { compute_heavy_things_part_1; maybePreemptRender(); compute_heavy_things_part_2; maybePreemptRender(); ... compute_heavy_things_part_5; maybePreemptRender(); } }
This is related to the engine PR concerns - it seems like we're struggling a bit to find the right way to express the concept of vsync/animation frame. The platform gives us a very clear signal, and I would like to avoid adding methods to the engine or framework to override that.
- Insert a maybePreemptRender to RenderObject.layout function seems enough, without breaking anything if I think correctly.
- I also think about another possibility, just do it the way now it is. In other words, let the user manually specify preempt points via SmoothPreemptPoint. Then one less PR to framework, and user has more flexibility.
I really appreciate your enthusiasm for this topic! Thanks! 🙂 What's unclear to me is why we want tickers to have an artificial way to fire an extra tick. Because tickers originally fire once per full pipeline, in the animation phase. But now we want it to run extra ticks in the 60fps smooth extra frame. it seems like we're struggling a bit to find the right way to express the concept of vsync/animation frame. I do respect vsync, just using a way other than "be fired by onBeginFrame" (because when we are busy running dart code, the callback can never be fired again). Shortly speaking, I let VsyncAwaiter set a thread-shared variable about the last vsync data. Then in maybePreemptRender, I read that data. (Briefly speaking) if it is a new vsync, I realize it is time to create Scene and submit via window.render. Thus I submit once per vsync and respect vsync well. The platform gives us a very clear signal, and I would like to avoid adding methods to the engine or framework to override that. I would also like to make as few changes as possible, if it is possible 🙂 Not sure what "clear signal" mean, but if it means "the vsync signal", then hope my explanaions above solve the problem - I do respect the clear signal as well in another way
Btw, I did not add replies to https://github.com/flutter/engine/pull/36438 today since yesterday already add some and some questions seem also overlap with today.
Feel free to ask if there is anything missing!
In real applications under real workloads there is more need for GC time - for example, if your application is creating a lot of objects to understand data it fetched from the network or SQLite.
If we are worried that users may submit multiple window.render inside one vsync, another possibility: We may add some code in the C++ layer (or Dart layer), such that we check current vsync status, and only Produce() if it is a new vsync that has not been produced before.
If we are worried that users may submit multiple window.render inside one vsync, another possibility: We may add some code in the C++ layer (or Dart layer), such that we check current vsync status, and only Produce() if it is a new vsync that has not been produced before.
Which patch is updating a vsync ready signal for dart:ui or the framework?
The patches I've seen so far don't seem to do that.
No patch yet, because I was thinking to submit as few as possible
If you like it I can submit one, but that may not be tiny
Spoiler: It looks like https://github.com/fzyzcjy/engine/blob/c78138e3e79abfc771449a3a8341f7fc9211066f/shell/common/vsync_waiter.cc#L187
- Set a variable ("lastVsyncInfo") when VsyncAwaiter callback is fired
- Dart can read that variable
The current possibly hard part for that potential PR: in android sdk>=29 and ios, seems that the callback of VsyncWaiter is fired on ui thread. But our Dart code is occupying the UI thread (for a long time) so that vsync callback may not be fired. For old android it does work well because it is fired in platform thread (my example is based on that)
To create the PR, I may need to move new-android and ios VsyncWaiter to platform thread as well.
I agree. But seems that we need to compare two cases:
- Without this new proposal: Suppose one frame is 100ms, then we have busy UI thread for 100ms without idle. And then frame ends so we have idle.
- With this proposal: We still be busy for 100ms (+3% overhead so indeed 103ms), and then get idle. In addition, we have 6 extra smooth frames which may generate some object.
Thus, the difference with this proposal is that, the objects we create inside extra smooth frame do give GC extra pressure. But I hope that is small - they are just animations.
Moreover, IMHO, my approach allows GC to happen, as long as it is less than 16ms stop-the-world and is not very unlucky.
For example, suppose GC happens during 17ms to 20ms. Then I can still build the layer tree and submit window.render at around 33.33ms. That stop-the-world GC does not cause any problem like visible jank. As long as we have about 0.5ms per 16.667ms, because 0.5ms is what we need (in experiments below) to produce an extra smooth frame.
On the contrary, existing methods may have jank in such cases. Because if GC runs for 3ms, they have 3ms less to compute the next scene.
I'm not suggesting you create that PR right now, but having a working patch that shows what would need to be done, with some details about why this approach is being taken would help.
For example, I'd expect your document to have a section on this explaining how it will work and what threading considerations are being made etc.
I see. I will add that (probably within a few hours) and come back when I am done.
That section is now written under "Get last vsync time information" doc section, with psuedo-code, threading concerns etc I will make a runnable code if the proposal about this vsync change looks interesting
AutomatedTestWidgetsFlutterBinding.pump provides wrong pump time stamp, probably because of forgetting the precision
The fix is just one line:
I have git blame and find the bug exist since the first version 7yr ago, and no special comments about why this is introduced so I guess it is but not feature.
IMHO the bug may be written like, the programmer wants to convert DateTime (the clock.now()) into a Duration. But then it is forgotten that both are microseconds precision instead of milliseconds precision, and the milliseconds approach is used.
List which issues are fixed by this PR. You must list at least one issue. Close #112610
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
p.s. test fails with old code, confirming that the test is effective.
Export elapseBlocking to test binding, so slow sync work can be simulated such as a slow widget build
There are needs to simulate sync heavy work, such as a slow widget build, in flutter widget tests. This method simply expose that.
As a remark, this cannot be replaced by runAsync. Consider the following example:
testWidgets('can use to simulate slow build', (WidgetTester tester) async {
final DateTime beforeTime = binding.clock.now();
await tester.pumpWidget(Builder(builder: (_) {
bool timerCalled = false;
Timer.run(() => timerCalled = true);
binding.elapseBlocking(const Duration(seconds: 1));
// if we use `delayed` instead of `elapseBlocking`, such as
// binding.delayed(const Duration(seconds: 1));
// the timer will be called here. Surely, that violates how
// a flutter widget build works
expect(timerCalled, false);
return Container();
}));
expect(binding.clock.now(), beforeTime.add(const Duration(seconds: 1)));
binding.idle();
});
As is discussed in the comments in the example, if we use delayed, timers will be fired when executing half of a build function, which is totally wrong.
List which issues are fixed by this PR. You must list at least one issue. Close https://github.com/flutter/flutter/issues/112620
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Fizzling like that would be expensive, and we'd be giving developers a button to push that actually makes things slower. We should avoid that.
@dnfield If speed is a concern, maybe the original PR is ok: For a normal usage, it only adds if (!producer_continuation_) (and that if will return false immediately). Given that explicit operator bool() const { return continuation_ != nullptr; }, this if will only check whether a pointer is nullptr, so I guess it is only a few CPU cycles (per 16667 microseconds, i.e. maybe 10000000 cycles). In addition, for a normal usage, the branch will always be false, so IMHO the cpu branch predictor will be quite correct about the prediction.
@dnfield giving developers a button to push that actually makes things slower
There seems to be another way that is not very slower:
Firstly, the onlyRenderOncePerBeginFrame should not be window.render(onlyRenderOncePerBeginFrame: true), but be window.onlyRenderOncePerBeginFrame = true; window.render(). In other words, it should be a flag that is set once. Then the code is:
...
if (!onlyRenderOncePerBeginFrame && !producer_continuation_) {
producer_continuation_ = layer_tree_pipeline_->Produce();
}
...
void SetOnlyRenderOncePerBeginFrame(bool value) { this->onlyRenderOncePerBeginFrame = value; }
class Animator { ... bool onlyRenderOncePerBeginFrame; ... }
(no need for locks, since all on UI thread.)
Then there comes the concern that if (!onlyRenderOncePerBeginFrame && !producer_continuation_) can cost CPU cycles, even when onlyRenderOncePerBeginFrame is always true (for a classical user). Firstly, for a classical user, we only pay extra cost of if(boolean) (because && is short-circuited), so only a few cycles.
Secondly, seems we can use the [[likely]] (c++20), or LIKELY (a lot of c++ library write their own version for that, e.g. see [how linux])(https://stackoverflow.com/questions/109710/how-do-the-likely-unlikely-macros-in-the-linux-kernel-work-and-what-is-their-ben) does that), to further hint compiler about this case to speed up.
@dnfield And for zero speed decrease if you like:
The Animator::PrepareExtraRender proposal seems to cause zero speed loss for a classical user. Because a classical user never calls that function, and only call Animator::Render. But Animator::Render is not modified in this proposal. For a smooth user, there does exist overhead, because need to call one extra C++ function - the PrepareExtraRender.
Quick update (still WIP, just provide some progress): I am working on the gesture system. Jonah Williams has thought that, it was bad that my old proposal did not let the pointer data packet go through Flutter's gesture system. Now, the new method just calls the classical gestureBinding.handlePointerEvent to dispatch PointerMoveEvents.
Fix logic error in markNeedsPaint
The original code comment says:
If we're the root of the render tree (probably a RenderView), then we have to paint ourselves, since nobody else can paint us. We don't add ourselves to _nodesNeedingPaint in this case, because the root is always told to paint regardless.
However, IMHO it is wrong in two aspects.
Problem 1: RenderView does not come to this branch
Firstly, for a RenderView, it will not go into this branch, but instead go into the first branch (the if (isRepaintBoundary && _wasRepaintBoundary)). This is because RenderView.isRepaintBoundary is defined to be true, which can be seen in the code.
The experiment also confirms this. Click to expand below:
Details
Add a few logs:
Code:
import 'package:flutter/foundation.dart';
import 'package:flutter/material.dart';
import 'package:flutter/rendering.dart';
import 'package:flutter/scheduler.dart';
import 'package:flutter_test/flutter_test.dart';
void main() {
testWidgets('hello', (tester) async {
debugPrintBeginFrameBanner = debugPrintEndFrameBanner = true;
final dummy = ValueNotifier(0);
await tester.pumpWidget(ValueListenableBuilder(
valueListenable: dummy,
builder: (_, dummy, __) => _DummyInner(dummy: dummy),
));
dummy.value++;
await tester.pump();
debugPrintBeginFrameBanner = debugPrintEndFrameBanner = false;
});
}
class _DummyInner extends SingleChildRenderObjectWidget {
final int dummy;
const _DummyInner({
super.key,
required this.dummy,
super.child,
});
_RenderDummy createRenderObject(BuildContext context) =>
_RenderDummy(dummy: dummy);
void updateRenderObject(BuildContext context, _RenderDummy renderObject) {
renderObject.dummy = dummy;
}
}
class _RenderDummy extends RenderProxyBox {
_RenderDummy({
required int dummy,
RenderBox? child,
}) : _dummy = dummy,
super(child);
// not mark repaint yet
int get dummy => _dummy;
int _dummy;
set dummy(int value) {
if (_dummy == value) return;
_dummy = value;
print('hi ${describeIdentity(this)} set dummy thus markNeedsPaint START');
markNeedsPaint();
print('hi ${describeIdentity(this)} set dummy thus markNeedsPaint END');
}
void paint(PaintingContext context, Offset offset) {
print('hi ${describeIdentity(this)}.paint SUPPOSE THIS IS THE REAL PAINT');
super.paint(context, offset);
}
}
output
/Volumes/MyExternal/ExternalRefCode/flutter/bin/flutter --no-color test --machine --start-paused --plain-name hello --local-engine-src-path=/Volumes/MyExternal/ExternalRefCode/engine/src --local-engine=host_debug_unopt test/hello.dart
Testing started at 09:21 ...
hi RenderParagraph#d9227.markNeedsPaint start _needsPaint=true
hi RenderPositionedBox#d9bb5.markNeedsPaint start _needsPaint=true
hi RenderView#fab3a.markNeedsPaint start _needsPaint=true
hi flushPaint PipelineOwner#89028 node=RenderView#fab3a NEEDS-PAINT _needsPaint=true owner=PipelineOwner#89028 node._layerHandle.layer!.attached=true
hi TransformLayer#64092.buildScene
hi PictureLayer#149d1._addToSceneWithRetainedRendering _needsAddToScene=true
▄▄▄▄▄▄▄▄ Frame 2 0ms ▄▄▄▄▄▄▄▄
hi _RenderDummy#a17bc.markNeedsPaint start _needsPaint=true
hi RenderView#fab3a.markNeedsPaint start _needsPaint=false
hi RenderView#fab3a.markNeedsPaint case-repaintboundary owner=PipelineOwner#89028
hi flushPaint PipelineOwner#89028 node=RenderView#fab3a NEEDS-PAINT _needsPaint=true owner=PipelineOwner#89028 node._layerHandle.layer!.attached=true
hi _RenderDummy#a17bc.paint SUPPOSE THIS IS THE REAL PAINT
hi TransformLayer#64092.buildScene
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
▄▄▄▄▄▄▄▄ Frame 3 0ms ▄▄▄▄▄▄▄▄
hi _RenderDummy#a17bc set dummy thus markNeedsPaint START
hi _RenderDummy#a17bc.markNeedsPaint start _needsPaint=false
hi _RenderDummy#a17bc.markNeedsPaint case-parent parent=RenderView#fab3a
hi RenderView#fab3a.markNeedsPaint start _needsPaint=false
hi RenderView#fab3a.markNeedsPaint case-repaintboundary owner=PipelineOwner#89028
hi _RenderDummy#a17bc set dummy thus markNeedsPaint END
hi flushPaint PipelineOwner#89028 node=RenderView#fab3a NEEDS-PAINT _needsPaint=true owner=PipelineOwner#89028 node._layerHandle.layer!.attached=true
hi _RenderDummy#a17bc.paint SUPPOSE THIS IS THE REAL PAINT
hi TransformLayer#64092.buildScene
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
hi RenderParagraph#c63b7.markNeedsPaint start _needsPaint=true
hi RenderPositionedBox#2fb99.markNeedsPaint start _needsPaint=true
hi RenderView#fab3a.markNeedsPaint start _needsPaint=false
hi RenderView#fab3a.markNeedsPaint case-repaintboundary owner=PipelineOwner#89028
hi flushPaint PipelineOwner#89028 node=RenderView#fab3a NEEDS-PAINT _needsPaint=true owner=PipelineOwner#89028 node._layerHandle.layer!.attached=true
hi TransformLayer#64092.buildScene
hi PictureLayer#18f7f._addToSceneWithRetainedRendering _needsAddToScene=true
By looking at the experiment above, we see that, the RenderView goes to the case-repaintboundary which is the first branch, instead of the third branch, so the comments seem incorrect.
Problem 2: Root is not always told to paint indeed
Theoretically, I do not find clues why "root is always told to paint" indeed. Experimentically, this is also confirmed as below.
We change the branching condition as follows, so RenderView is forced to go to the 3rd branch (the branch with comments), instead of the 1st branch.
- if (isRepaintBoundary && _wasRepaintBoundary) {
+ if (isRepaintBoundary && _wasRepaintBoundary && /*HACK!!!*/(this is! RenderView)) {
Then we run the test code same as above (only with a few more logging), and get:
Details
/Volumes/MyExternal/ExternalRefCode/flutter/bin/flutter --no-color test --machine --start-paused --plain-name hello --local-engine-src-path=/Volumes/MyExternal/ExternalRefCode/engine/src --local-engine=host_debug_unopt test/hello.dart
Testing started at 09:25 ...
hi RenderParagraph#c9872.markNeedsPaint start _needsPaint=true
hi RenderPositionedBox#f6896.markNeedsPaint start _needsPaint=true
hi RenderView#0fb85.markNeedsPaint start _needsPaint=true
hi flushPaint PipelineOwner#ce901 node=RenderView#0fb85 NEEDS-PAINT _needsPaint=true owner=PipelineOwner#ce901 node._layerHandle.layer!.attached=true
hi RenderView#0fb85.paint
hi TransformLayer#d7668.buildScene
hi PictureLayer#335b7._addToSceneWithRetainedRendering _needsAddToScene=true
▄▄▄▄▄▄▄▄ Frame 2 0ms ▄▄▄▄▄▄▄▄
hi _RenderDummy#3427b.markNeedsPaint start _needsPaint=true
hi RenderView#0fb85.markNeedsPaint start _needsPaint=false
hi RenderView#0fb85.markNeedsPaint case-else owner=PipelineOwner#ce901
hi TransformLayer#d7668.buildScene
hi PictureLayer#335b7._addToSceneWithRetainedRendering _needsAddToScene=false
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
▄▄▄▄▄▄▄▄ Frame 3 0ms ▄▄▄▄▄▄▄▄
hi _RenderDummy#3427b set dummy thus markNeedsPaint START
hi _RenderDummy#3427b.markNeedsPaint start _needsPaint=true
hi _RenderDummy#3427b set dummy thus markNeedsPaint END
hi TransformLayer#d7668.buildScene
hi PictureLayer#335b7._addToSceneWithRetainedRendering _needsAddToScene=false
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
hi RenderParagraph#66d6b.markNeedsPaint start _needsPaint=true
hi RenderPositionedBox#537a5.markNeedsPaint start _needsPaint=true
hi RenderView#0fb85.markNeedsPaint start _needsPaint=true
hi TransformLayer#d7668.buildScene
hi PictureLayer#335b7._addToSceneWithRetainedRendering _needsAddToScene=false
As we can see, RenderView.paint and RenderDummy.paint is only called once, even though we clearly call RenderDummy.markNeedsPaint. That is indeed a bug, and at least shows that the code comment is wrong - root is not always told to paint.
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.
List which issues are fixed by this PR. You must list at least one issue. close #112736
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
[WIP][Do not merge this PR] Tentative experiment to see how to fix logic error about skippedPaintingOnLayer
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.
List which issues are fixed by this PR. You must list at least one issue.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
All right, this should not be the fix
Add warning that RenderRepaintBoundary.toImage and OffsetLayer.toImage is slow
Scene.toImage has doc saying: "This is a slow operation that is performed on a background thread". However, people may use RenderRepaintBoundary.toImage and OffsetLayer.toImage and never read that comment, so they are unaware of the slowness. This PR simply adds the warning to them.
List which issues are fixed by this PR. You must list at least one issue.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Update comments that seem to contradict the code and may confuse the reader
Original comment:
... So we flatten the layer tree into a picture and use that as the thread transport mechanism.
However, looking at the whole code:
Details
Dart_Handle Picture::RasterizeToImage(sk_sp<DisplayList> display_list,
std::shared_ptr<LayerTree> layer_tree,
uint32_t width,
uint32_t height,
Dart_Handle raw_image_callback) {
if (Dart_IsNull(raw_image_callback) || !Dart_IsClosure(raw_image_callback)) {
return tonic::ToDart("Image callback was invalid");
}
if (width == 0 || height == 0) {
return tonic::ToDart("Image dimensions for scene were invalid.");
}
auto* dart_state = UIDartState::Current();
auto image_callback = std::make_unique<tonic::DartPersistentValue>(
dart_state, raw_image_callback);
auto unref_queue = dart_state->GetSkiaUnrefQueue();
auto ui_task_runner = dart_state->GetTaskRunners().GetUITaskRunner();
auto raster_task_runner = dart_state->GetTaskRunners().GetRasterTaskRunner();
auto snapshot_delegate = dart_state->GetSnapshotDelegate();
// We can't create an image on this task runner because we don't have a
// graphics context. Even if we did, it would be slow anyway. Also, this
// thread owns the sole reference to the layer tree. So we flatten the layer
// tree into a picture and use that as the thread transport mechanism.
auto picture_bounds = SkISize::Make(width, height);
auto ui_task =
// The static leak checker gets confused by the use of fml::MakeCopyable.
// NOLINTNEXTLINE(clang-analyzer-cplusplus.NewDeleteLeaks)
fml::MakeCopyable([image_callback = std::move(image_callback),
unref_queue](sk_sp<DlImage> image) mutable {
auto dart_state = image_callback->dart_state().lock();
if (!dart_state) {
// The root isolate could have died in the meantime.
return;
}
tonic::DartState::Scope scope(dart_state);
if (!image) {
tonic::DartInvoke(image_callback->Get(), {Dart_Null()});
return;
}
if (image->skia_image()) {
image =
DlImageGPU::Make({image->skia_image(), std::move(unref_queue)});
}
auto dart_image = CanvasImage::Create();
dart_image->set_image(image);
auto* raw_dart_image = tonic::ToDart(std::move(dart_image));
// All done!
tonic::DartInvoke(image_callback->Get(), {raw_dart_image});
// image_callback is associated with the Dart isolate and must be
// deleted on the UI thread.
image_callback.reset();
});
// Kick things off on the raster rask runner.
fml::TaskRunner::RunNowOrPostTask(
raster_task_runner,
[ui_task_runner, snapshot_delegate, display_list, picture_bounds, ui_task,
layer_tree = std::move(layer_tree)] {
sk_sp<DlImage> image;
if (layer_tree) {
auto display_list = layer_tree->Flatten(
SkRect::MakeWH(picture_bounds.width(), picture_bounds.height()),
snapshot_delegate->GetTextureRegistry(),
snapshot_delegate->GetGrContext());
image = snapshot_delegate->MakeRasterSnapshot(display_list,
picture_bounds);
} else {
image = snapshot_delegate->MakeRasterSnapshot(display_list,
picture_bounds);
}
fml::TaskRunner::RunNowOrPostTask(
ui_task_runner, [ui_task, image]() { ui_task(image); });
});
return Dart_Null();
}
It seems that, the layer_tree is directly moved into raster_task_runner callbacks. Then, inside the raster thread, layer_tree->Flatten is called and it is converted to a DisplayList. In other words, the "thread transport mechanism" seems to be the layer_tree (ui -> raster thread) and DlImage (raster -> ui thread), instead of the "flatten the layer tree into a picture and use that" (the flattened layer tree, i.e. the picture).
List which issues are fixed by this PR. You must list at least one issue.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
This only updates comments so seems no need for tests
@jonahwilliams Hi thanks for the reply.
- If it is slow, IMHO the users need to know it, otherwise users may abuse it because they may think it is just a normal function.
- If it is fast, then we need to remove the original comment (because it is outdated).
- As for impl specific, if it is slow but in the future it becomes fast (hopefully!), seems that we can update comments at that time.
Anyway this is just a small doc change and it does not matter whether it is changed or not for myself (since I already know it is slow and should be careful). I have spent some making this PR simply because I hope other users works correctly with the api :)
Minor change type nullability
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.
List which issues are fixed by this PR. You must list at least one issue.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
You're discounting the time it takes to actually make a native call from Dart.
On top of that, we should not expose an API that might or might not do something and developers have no good way to reason about whether they're really supposed to call it or not.
This proposal is fundamentally changing the invariants around render/onBeginFrame, but it's not providing any way for developers to know if they're using the new invariants correctly or not. Even if the new potentially useless API is relatively cheap, it adds up when developers (and packages they use) start doing it multiple times per frame. And those developers/packages will have no way to know whether they're doing it correctly or not, so it will definitely get misused.
Why, for example, shouldn't the framework just call render and schedule a new frame when its hit its potential limit?
@dnfield Hi thanks for the reply.
You're discounting the time it takes to actually make a native call from Dart.
Originally I thought that is small just like a normal function call... Ok now I learn it.
On top of that, we should not expose an API that might or might not do something and developers have no good way to reason about whether they're really supposed to call it or not.
Indeed they have a way to reason: look at time or vsync info. They should not submit twice inside one vsync interval.
This proposal is fundamentally changing the invariants around render/onBeginFrame, but it's not providing any way for developers to know if they're using the new invariants correctly or not.
I am not sure, if I expose the vsync info and ensure only one call is made per vsync interval (16.67ms), does this satisfy your requirement?
Even if the new potentially useless API is relatively cheap, it adds up when developers (and packages they use) start doing it multiple times per frame.
Again, as mentioned above, dev should not call it multiple times per frame. A naive dev may use DateTime.now() - last_vsync_time > 15ms etc to check, and a more sophisticated way may be read the vsync info (exposed from engine) to really ensure we never call twice per vsync interval.
And those developers/packages will have no way to know whether they're doing it correctly or not, so it will definitely get misused.
Then maybe we should return bool to indicate whether it is really scheduled. If they see a lot of false they are doing it wrong (call too many times that are useless).
Why, for example, shouldn't the framework just call render and schedule a new frame when its hit its potential limit?
Because of the fundamental design of the preempt proposal (https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko/edit), mainly "The flow chart" section.
Indeed, window.render is called per vsync interval (16.67ms). The main difference from classical code is that, it may be called multiple times per onBeginFrame (when onBeginFrame is super slow).
Devices do not always have 60fps vsync - sometimes it's 90 or 120 or more or less (at one point we had a customer looking at 240hz devices, and it's likely there are customers out there looking at 30hz use cases). There is no way currently in dart:ui to know what the current refresh rate is, and on some platforms it's not even possible to implement because the vendors don't provide an API for it (e.g. some Android vendors), and it can change from frame to frame.
In other words, a developer must not assume that 16.67ms is the right interval for a frame in all circumstances. And the query of DateTime.now is almost certain to not match the actual vsync start time, so if you assume you have roughly 16ms from onBeginFrame you might actually overshoot vsync.
Devices do not always have 60fps vsync - sometimes it's 90 or 120 or more or less (at one point we had a customer looking at 240hz devices, and it's likely there are customers out there looking at 30hz use cases). There is no way currently in dart:ui to know what the current refresh rate is, and on some platforms it's not even possible to implement because the vendors don't provide an API for it (e.g. some Android vendors), and it can change from frame to frame.
Definitely! That's why I also propose to expose vsync-related information to the dev. Last week you asked me to describe it and it was at "Get last vsync time information" section of google doc.
And the query of DateTime.now is almost certain to not match the actual vsync start time, so if you assume you have roughly 16ms from onBeginFrame you might actually overshoot vsync.
Totally agree. Indeed in my (previous) implementation, I let the C++ side expose the timeStamp (i.e. vsync target time we provide to dart tonBeginFrame) both a "Duration" and a "DateTime-compatible time". If you like I can add that back (deleted it b/c want to make PR small).
P.S. I am also considering relaxing when to start a onBeginFrame which seems to reduce unnecessary idle time and improve performance. That may be related to the big picture you are concerned - how vsync and code are interacted. I will add it to design doc and reply here maybe in an hour.
@dnfield Here it goes: "Relax onBeginFrame starting criterion" section in https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko/edit
I think it would be easier to start with a patch that exposes more about vsync timings to the developer, because that will be critical to whether this one makes sense.
Thanks, I will do that.
Expose vsync information to developer
This PR tries to expose vsync information to the developer, so they can know when it is proper to call the more un-restricted window.render proposed in #36438.
Currently only the API is shown, because IMHO the implementation details is unrelated to thoughts about #36438, and the API (and therefore implementations) are subject to changes. I will continue working on it, once the API is approved.
For detailed design about this API and its implementation, please have a look at "Get last vsync time information" section of https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko/edit.
Sample usage:
final info = SchedulerBinding.instance.lastVsyncInfo();
List of work:
- code the (draft) Dart API
- discuss whether the API is acceptable
- implement the C++ part on SDK<=29 Android
- implement the C++ part on new android, ios, and other platforms
- create a wrapper function in
flutter/flutterrepo, probably inSchedulerBinding
List which issues are fixed by this PR. You must list at least one issue.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
@dnfield Hi, PR is here: https://github.com/flutter/engine/pull/36607
Only the API is there currently, because IMHO the implementation details is unrelated to thoughts about this issue, and the API (and therefore implementations) are subject to changes.
| 00:15 +36: /b/s/w/ir/x/t/flutter_customer_testing.flutter_packages.RNUBSQ/tests/packages/animations/test/open_container_test.dart: Container closes - Fade (by default)
| ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
| The following TestFailure was thrown running a test:
| Expected: 1.0 (±1e-10)
| Actual: <0.9999833333333332>
| Which: 0.9999833333333332 is not in the range of 1.0 (±1e-10).
|
| When the exception was thrown, this was the stack:
| #4 main.<anonymous closure> (file:///b/s/w/ir/x/t/flutter_customer_testing.flutter_packages.RNUBSQ/tests/packages/animations/test/open_container_test.dart:273:7)
| <asynchronous suspension>
| <asynchronous suspension>
| (elided one frame from package:stack_trace)
|
| This was caught by the test expectation on the following line:
| file:///b/s/w/ir/x/t/flutter_customer_testing.flutter_packages.RNUBSQ/tests/packages/animations/test/open_container_test.dart line 273
| The test description was:
| Container closes - Fade (by default)
| ════════════════════════════════════════════════════════════════════════════════════════════════════
|
| 00:15 +37 -1: /b/s/w/ir/x/t/flutter_customer_testing.flutter_packages.RNUBSQ/tests/packages/animations/test/fade_scale_transition_test.dart: FadeScaleTransitionConfiguration builds a new route
| 00:15 +37 -1: /b/s/w/ir/x/t/flutter_customer_testing.flutter_packages.RNUBSQ/tests/packages/animations/test/open_container_test.dart: Container closes - Fade (by default) [E]
| Test failed. See exception logs above.
| The test description was: Container closes - Fade (by default)
@pdblasi-google I guess maybe need to update the custom testing configurations to point to the latest tests?
@fzyzcjy Yup, you called it. Apologies, I forgot to point the flutter/tests repo to the latest tests. PR is up for that now, I'll ping here when it goes through.
Golden file changes have been found for this pull request. Click here to view and triage (e.g. because this is an intentional change).
If you are still iterating on this change and are not ready to resolve the images on the Flutter Gold dashboard, consider marking this PR as a draft pull request above. You will still be able to view image results on the dashboard, commenting will be silenced, and the check will not try to resolve itself until marked ready for review.
For more guidance, visit Writing a golden file test for package:flutter.
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Changes reported for pull request #112609 at sha 02ebd54c3343d0c3aaabcba423b5db13b2bfaadb
Thanks and 🎉 !
@fzyzcjy Did you have a chance to look into testing this or applying for the test exception?
Any reason this is still marked as Draft?
This will need a testing exception.
I agree that this comment alone is not particularly useful. To be useful it would need more context so developers can actually make an informed decision of whether they want to use this or not. Let's close this for now.
This patch is missing a lot of context information. To quote Dan from the other patch:
This patch needs a lot more context. What's the main goal you're trying to achieve with this particular change? Why is it doing it this way? Why can't we used existing mechanisms to achieve the same thing? Why do we need this mechanism to achieve the larger goal of having incremental/progressive layout?
Regardless, this doesn't seem like a great API to provide a single static callback that gets called on every layout. What if multiple implementations are trying to set this? Also, this gets called for the layout of every single RenderObject. This doesn't sound great for performance. I know, the title claims it has no overhead, but I find that hard to believe. What's the basis for that claim?
Any reason this is still marked as Draft?
No, I just forgot it :)
This will need a testing exception.
I think so, thanks
test-exempt: API refactor
@Hixie So shall I relax the code visibility of the widgets and render objects (i.e. change from private to public) in order to have a test? If so I will add it.
@goderbauer Ah I forgot it completely. Here is the exemption request a few seconds ago: https://discordapp.com/channels/608014603317936148/608018585025118217/1029170238190784704
Please ignore this PR for now, since in the https://github.com/fzyzcjy/flutter_smooth (i.e. impl of https://docs.google.com/document/d/1FuNcBvAPghUyjeqQCOYxSt6lGDAQ1YxsNlOvrUx0Gko/edit#), I am trying to use manual widgets as a workaround. So skip it is you are busy :)
This doesn't sound great for performance. I know, the title claims it has no overhead, but I find that hard to believe. What's the basis for that claim?
I mean zero overhead when it is disabled (which IIRC is what hixie(?) cares about a lot).
Compiler explorer says it is zero overhead b/c the compiler just correctly understands it and eliminate the dead code: https://discord.com/channels/608014603317936148/608021234516754444/1024141725024923688
I am still working on the "Preemption for 60FPS", i.e. flutter_smooth, currently. Just having this (very rough) idea and want to share it here https://github.com/flutter/flutter/issues/113281
(Spoiler: It tries to solve the hot update problem)
test-exempt: code refactor with no semantic change
Quick update: ListView scrolling at 60FPS with heavy build/layout
Highlights:
- It is 60FPS (check via splitting video into frames, and by my script to examine timeline tracing data; not checked this demo video though; you can find the script in my repo)
- The list shifting is (roughly) uniform speed (up to error from OS pointer events) (check via script to examine timeline tracing data; again script is in my repo)
- The system uses
gestureBinding.handlePointerEventto dispatchPointerMoveEvents
Experiment setup: Slow build/layout when new item comes in. Full code can be seen in https://github.com/fzyzcjy/flutter_smooth.
May still contain (a lot of) bugs, since it is still WIP :)
Video (firstly raw case, then use-flutter_smooth case):
https://user-images.githubusercontent.com/5236035/195363841-240fa44c-c471-412e-9c3d-3314cf6ed8ea.mp4
Sample screenshots from tracing and my script:
Hi guys, quick update:
ListView scrolling at 60FPS with heavy build/layout
Highlights:
- It is 60FPS (check via splitting video into frames, and by my script to examine timeline tracing data; not checked this demo video though; you can find the script in my repo)
- The list shifting is (roughly) uniform speed (up to error from OS pointer events) (check via script to examine timeline tracing data; again script is in my repo)
- The system uses
gestureBinding.handlePointerEventto dispatchPointerMoveEvents
Experiment setup: Slow build/layout when new item comes in. Full code can be seen in https://github.com/fzyzcjy/flutter_smooth.
May still contain (a lot of) bugs, since it is still WIP 🙂
Video (click to see):
https://github.com/flutter/flutter/issues/101227#issuecomment-1276239303
@fzyzcjy @goderbauer this is a breaking change. Can a migration guide be written on how developers can migrate their code with this change? I'm not sure what's needed on my end as a developer, and my animation tests are now very flaky.
@CaseyHillers Hi,
Can a migration guide be written on how developers can migrate their code with this change? ...and my animation tests are now very flaky.
Could you please share some flaky test minimal reproducible samples? IMHO this should not make any problem so want to see reproductions in order to know what happens
Here's an example now that is flaky:
await tester.pumpWidget(myAnimatedWidget);
await tester.pumpAndSettle();
await tester.sendSelectEvent();
await tester.pumpFrames(scene, Duration(milliseconds: 100));
await expectLater(
find.byType(MyAnimatedWidget),
matchesGoldenFile(
'animated_widget'));
The resulting goldens are changing. When I change pumpFrames to microseconds, I am still seeing the same flakiness. I'm unsure if it's because the earlier clocks are still in millisecond mode.
@CaseyHillers Weird. Some possible ideas:
- Do you have anything that depends on e.g. a real clock? If so, it will be flaky. (I guess no)
- Could you please change
MyAnimatedWidgetto something likeAnimatedBuilder(builder: (_, value) => Text('the value is: $value'). Then, when the golden is changing, we can know what value indeed it is having.
I'm unsure if it's because the earlier clocks are still in millisecond mode.
Do you mean the new golden (i.e. after the PR) are different from old golden, and the new golden is itself stable? If so, looks like it is possible. Indeed the animation controller is fed with a changed animation time.
My animated widgets are just an animated builder that has a custom animation controller. @goderbauer or @pdblasi-google can help with reproducing the issue here.
My understanding is I need to change every possible clock to be in microseconds instead of milliseconds. This seems to be a breaking change for any other customers, and I'm having a difficult time tracking all the various clocks in my codebase.
I haven't found the discord threads, but I wonder if this is something that should be in the framework. There could be a tester field added for high precision or this can be added directly to your package. I assume most Flutter tests aren't needing high precision, and this is going to cause a lot of pain once it's in beta/stable.
@CaseyHillers
I need to change every possible clock to be in microseconds instead of milliseconds
Btw I am curious why a clock can in milliseconds in codebase - clock package, Duration, DateTime, etc are all in microseconds.
I haven't found the discord threads, but I wonder if this is something that should be in the framework. There could be a tester field added for high precision or this can be added directly to your package. I assume most Flutter tests aren't needing high precision, and this is going to cause a lot of pain once it's in beta/stable.
I agree that an alternative solution is to use a bool flag to enable high-accuracy (I have done that indeed - https://github.com/flutter/flutter/pull/112609/commits/e0b5882b87bc8fe025d55d626ec663c8587522b7).
I agree that an alternative solution is to use a bool flag to enable high-accuracy (I have done that indeed - https://github.com/flutter/flutter/commit/e0b5882b87bc8fe025d55d626ec663c8587522b7).
Thanks, that sounds like it would work for me! Are there plans to upstream that change?
@CaseyHillers I am ok with that, just not sure what other reviewers think? (Since that was my original design and later a reviewer suggests me to change to what is current merged.) If you guys are OK I will PR it.
Reland AutomatedTestWidgetsFlutterBinding.pump provides wrong pump time stamp, probably because of forgetting the precision, via optional flag
This relands https://github.com/flutter/flutter/pull/112609, but with a flag that is off by default. The reason why it is designed like this can be found in discussions around https://github.com/flutter/flutter/pull/112609#issuecomment-1278889389.
Close #112610
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
@CaseyHillers Here is the PR: https://github.com/flutter/flutter/pull/113433
@fzyzcjy @CaseyHillers @goderbauer
I still think that this change should be made without the boolean flag. The underlying issue with the goldens is that pumpFrames defines interval with microsecond precision, but the AutomatedTestWidgetsFlutterBinding didn't support microsecond precision. @Piinks and I went over a couple of golden changes in the flutter repos test as well and accepted the changes to those goldens as they were a change to a correct state (which is why the goldens are current hanging for this PR).
The secondary issue that makes this difficult to break cleanly is that LiveWidgetsFlutterBinding does support microsecond precision, so we can't just update pumpFrames' interval parameter to be a clean 16 milliseconds by default, as that would break other tests.
To start with, the first paragraph in the breaking changes process says:
Sometimes, however, doing this is necessary for the greater good. We want our APIs to be intuitive; if being backwards-compatible requires making an API into something that we would never have designed that way unless forced to by circumstances, then we should instead break the API and make it good.
Adding a boolean to make the correct behavior happen is not an API that we would have designed on purpose. It's also not a change that we can easily guide people to using, as there's no "new API" we can drive them towards with deprecations or data driven fixes. It'd be something we introduce for a period of time, hope people read the blog, then end up running into the same "breaking change" issues when we eventually remove the boolean or default it to true.
From there, digging into the process, the preferred process is the three step process:
- Add new API and opt in to the new API
- Remove the old API
- Remove the opt in
I think the only way we can get that to happen is to introduce a new version of pumpFrames that would support the correct behavior, deprecate the current pumpFrames, then eventually remove pumpFrames. Here's what I'd propose:
- Fix
AutomatedTestWidgetsFlutterBindingwithout the flag - Introduce a new method with the exact contents that
pumpFramescurrently has:
pumpFramesFor(
Widget target,
Duration duration, [
Duration interval = const Duration(milliseconds: 16, microseconds: 683),
])
- Update
pumpFramesto pass through to the newpumpFramesFormethod:- Check
if (binding is AutomatedTestWidgetsFlutterBinding) - If it is, then truncate the microseconds off of
intervalbefore passing through to maintain the current incorrect behavior
- Check
My biggest concern with this approach is that pumpFramesFor isn't as clean a name as just pumpFrames, but it's the best I can come up with. Names aside, introducing a new method and deprecating the old is the only way I can think of to keep the existing behavior and actually be able to drive users to the new api before landing the correct behavior.
This is an artifact of what the code used to do, but it has since been refactored to not do that :)
I guess so :) That is why I update it - otherwise it will mislead future readers
@pdblasi-google @CaseyHillers @goderbauer I agree with both sides of the opinion, both looks very reasonable to me. So just ping me when googlers reach a conclusion that what I should do!
Fix wrong VSYNC event
Firstly, we know VSYNC event in timeline is special - chrome://tracing tool will show "zebra" colors (gray and white) at every VSYNC. Therefore, it is critical to let this event have correct timing, otherwise every user is doing reasoning with the wrong vsync time.
In the image below, the "a" shows the new VSYNC with this PR, while the "b" shows the old VSYNC interval before this PR. As we can see, the left side of "a" and "b" does not coincide. In other words, before this PR (where we have "b" as the VSYNC and there is no "a"), we consider the wrong time as the vsync time.
The cause is quite simple: Before this PR, the left edge of "VSYNC" event is (for example) the call time of VsyncWaiterAndroid::OnVsyncFromJava. However, there exist "frame delay" (e.g. frameDelayNanos argument in OnVsyncFromJava), so the real vsync time should minus that delay.
As a side remark, in the image below the difference is not very much, but in real scenarios, I have seen once in a while it has large differences. Then you know, the visualization goes wild, and it took me some time before I realized, it is not a bug in code anywhere, but a bug of the VSYNC event time.
Close #113475
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Allow disable report timing in profile build since it takes not-negligible amount of time
Flutter does say the time cost is "less than 0.1ms every 1 second to report the timings measured on iPhone6S". However, not every mobile phone is as high-end as iPhone6S. For example, on my testing device (TRT-AL00, indeed not the lowest-end device!), I measured that it takes about 20-30ms per second. Then we have a problem. When having https://github.com/fzyzcjy/flutter_smooth, we know a big janky frame (say, takes 200ms) will never let user really feel janky, but instead user will see the app being 60FPS smooth. However, this is based on the assumption that misc work such as report timings should not block the UI thread for a continuous period of time - which is not true if report timings happens. After the 200ms janky frame, we see about 6ms of report timing. Among with other things such as dispatch touch events, they easily take up more than ~16ms and we get one jank. Then flutter_smooth is no longer smooth due to the jank.
Except for the case of flutter_smooth, IMHO this PR is also useful for normal Flutter users. It takes 2-3% of CPU time, which is not negligible and may be measured. In addition, this is not a critical feature. Surely, when this is disabled, the DevTool will not show the frame ui/rasterizer time at all. However, not everyone needs to read that timing data, since they may either do not open DevTool, or use the tracing timeline instead (which contains more than enough information to know the frame timing data). Therefore, it looks reasonable to at least give users a chance (i.e. a flag) to disable it.
The code is deliberately written by reading a const bool environment variable. Therefore, it has completely zero overhead. I have confirmed that by using compiler explorer before - https://discordapp.com/channels/608014603317936148/608021234516754444/1024141682377236500.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Expose NotifyIdle from RuntimeController to Dart, allowing flutter_smooth to get 60FPS, even if GC needs to run for 14ms per 16.67ms
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
The PR simply exposes RuntimeController::NotifyIdle to the dart layer.
It is necessary for https://github.com/fzyzcjy/flutter_smooth because of the following commonly seen scenario: Suppose we are in a janky frame (say it takes 200ms). Then, NotifyIdle is never called at all, because it is usually called at the end of DrawFrame (indeed, more exactly, in the Animator::AwaitVSync). Then, during the 200ms, garbage accumulates, and at one time the young generation is full, then Dart VM must stop the world and make a GC. From my experiments, such GC can even take 20ms on my testing device. Stop the world for 20ms - then we must miss one frames, causing non-60FPS. Even if the stop-the-world GC is fast, say, 5ms, it can still cause a jank. For example, when it happens at 97.5ms-102.5ms, then the preemptRender which should originally be done near 98-100ms can only be done at 105ms, so it calls window.render too late, thus the rasterizer may fail to rasterize the frame before the 116.67ms vsync, so there is a jank. (If needed, I can draw a figure).
However, with this PR, there is no such problem at all. The flutter_smooth will call NotifyIdle immediately after each and every preemptRender, with a deadline of roughly 14ms (16.67ms minus a few ms). By doing this, there are two benefits. Firstly, since the heap is not that full, GC can finish its work sooner instead of the 20ms bad case when the heap is really full. This avoids the 20ms-long-GC problem above. Secondly, since we actively tell Dart VM that it can start a GC at this time, GC can run for a time duration as long as ~14ms without causing any jank. This is contrary to the discussion above, where even a 5ms GC can cause a frame jank. As for why it can run 14ms without causing trouble, it is because, suppose we start it at 100ms and it runs 14ms, then we are now at 114ms, and we start preemptRender. Since preemptRender is really fast (e.g. 2ms), we will submit window.render at 116ms. In other words, we submit window.render with sufficient time left for rasterizer to finish its job - as long as rasterizer finishes its job before 133.33ms, no jank will happen.
Therefore, the title is explained well: It allows flutter_smooth to get 60FPS, even if GC needs to run for 14ms per 16.67ms. (That extreme GC will not happen in real world, I just want to say this proposal works even for that.)
I have already done that for my engine branch and ran experiments on flutter_smooth. It works pretty well - originally I observe GC-caused janks and then they disappear after this fix. If you are interested I can present some data.
As for tests: Have not found a way to add tests for this very simple calling, may need a test exempt.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Tests can actually get to the private classes, they just can't use the private class name. You can cast the render object to dynamic and can call things on it blindly.
@Hixie Thanks, but the problem is I cannot construct a _CupertinoDialogRenderWidget object. All its current usages does not cause this bug (but future usages may do), while this is a logical bug indeed (since by definition updateRenderObject should update the fields).
So... what should I do?
Ah, I see. Yeah, that's unfortunate. Probably fine to skip the test for now then.
test-exempt: not technically changing actual behaviour.
Fix 1-char typo
Well, just 1-char typo that I come across...
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.
List which issues are fixed by this PR. You must list at least one issue.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
This pull request was opened against a branch other than main. Since Flutter pull requests should not normally be opened against branches other than main, I have changed the base to main. If this was intended, you may modify the base back to master. See the Release Process for information about how other branches get updated.
Reviewers: Use caution before merging pull requests to branches other than main, unless this is an intentional hotfix/cherrypick.
Eliminate duplicated code when dealing with pointer data
Hope the PR is self-explanatory :) If needed I can explain it.
List which issues are fixed by this PR. You must list at least one issue.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
@fzyzcjy
If you're alright with it, I'd like to grab the issue from you to wrap it up. We'll need to make some changes internally and do some extra documentation to release this due to the internal test failures. If you'd like, I'll guide you through the extra docs and just handle the internal stuff myself, but I think it'll be easier with just one person working to get this landed.
@pdblasi-google Sure :) so shall I close this issue now?
This PR yes. I'll still work off of the original issue so we can keep the wonderful history of this surprisingly complex issue! 😛
@pdblasi-google Looking forward to see it landed!
Speed up pointer data packet dispatching by roughly 2x when multiple packets come
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
Benefits for Flutter (without considering flutter_smooth)
Multiple pointer data packets often arrive in one vsync interval. Currently, each of them requires a PostTask, C++-to-Dart-call, etc. However, when there is already one pending PostTask and a second data packet arrives, we can optimize it - no need to schedule a second PostTask and C++-to-Dart call, but instead utilize the pending PostTask and submit more data inside one call.
How much speed up does it give: Consider the following screenshot (It happens after a long janky frame, but serves pretty well for us to compute numbers because it contains a lot of items - the average measure error will be much smaller). As we can see,
- total wall time: ~8.2ms
- total Dart time (measured by the
_handlePointerDataPackettime, which is the 4th purple row. I made an extra Timeline event to measure that): ~3.2ms
Therefore, if we merge multiple into one, we can get roughly 2x speed up, because it removes those idle periods between them, as well as some of the big overhead between Engine::DispatchPointerDataPacket and the real Dart code execution.
Concrete cases when this happens:
- When it janks, this PR helps speed up. For example, many Android devices provide two data packets per vsync interval. So suppose somehow the UI thread took 30ms to compute a frame, then this approach will merge 3-4 packet deliver into one.
- In some devices, speedup due to this PR happens in each and every time. For example, below is a tracing on a test phone. As you can see, it delivers 4 pointer data packets per frame. Consider what will happen when UI thread needs (e.g.) 15ms to compute (unlike what is in the screenshot which is very lightweight workload indeed). Then, the first 3 out of 4 packets will be able to be merged by this PR.
Benefits for flutter_smooth
The analysis is similar to above. However, since there are a lot of big janky frames in flutter_smooth, it is common to see a dozen of pointer event dispatching after that long janky frame. Therefore, this PR makes that part much much faster.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing. --- Note: I will fix CI later after hearing some review feedbacks, because after review feedback the code itself may change a lot, so I do not want to waste time to fix to-be-thrown code :)
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Make deadline of NotifyIdle configurable, allowing flutter_smooth to get 60FPS, even if GC needs to run for 14ms per 16.67ms
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
This PR is similar to https://github.com/flutter/engine/pull/36797. However, it addresses another portion of the GC-caused-jank problem.
Consider the following case: For each frame, UI thread needs to run for 16.00ms. Then:
Without this PR and without flutter_smooth: We know NotifyIdle will be called after the frame ends (more specifically, at AwaitVSync), and the "deadline" argument of NotifyIdle is set to "next_vsync_time - current_time". In other words, it is 16.67-16=0.67ms in our scenario. When DartVM receives this NotifyIdle call, it estimates how long a young GC needs, and realize it needs more than 0.67ms, so it do not call any young GC here. Therefore, garbage starts to accumulate. Finally, at one time, (young) GC must happen because the heap is full. At that time, Dart VM will stop the world for (e.g.) 10ms. Given that the UI thread needs 16.00ms to compute the content of one frame, the 10ms stop-the-world means it must miss at least one deadline. Thus, it janks whenever GC comes.
With this PR and flutter_smooth: No such problem at all. Let's consider one specific frame. Suppose the UI thread runs from 0.00-16.00ms and finished computing the content. Then, when calling NotifyIdle, I will deliberately set the "deadline" to be "next_vsync_time - current_time + 14ms". In other words, DartVM is now notified that, it has 14.67ms (instead of 0.67ms as before). Given this loose deadline, Dart VM happily executes a young GC (when it feels needed) using (e.g.) 10ms. Now we are at 26.00ms and the next frame begins. Given that we are using flutter_smooth, we can easily deliver an extra smooth frame when needed near 33.33ms, even though the plain-old frame needs 16.00ms to compute. Therefore, GC is triggered at proper time that does not cause any jank. And since NotifyIdle is triggered per 16.67ms with sufficient deadline (>14ms deadline duration), Dart VM will do GC at these period, so there will be no GC mentioned in the previous case which happens at random location causing UI to jank.
In conslusion, this PR allows flutter_smooth to get 60FPS, even if GC needs to run for (e.g.) 14ms per 16.67ms.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on ------ I deliberately do not add any tests yet, because the test are trivial and I want to listen to some feedbacks first (e.g. changing the PR). After feedbacks I will definitely add tests, no worries :) writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Fix jank and large-jumping frame by controlling rasterizer ending time
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
Consider the problem - what will happen, when the computation latency becomes lower temporarily? Looks like it is a good thing, since faster means better; many FPS monitors also do not think this is a problem. Spoiler: It is a bad thing - pay a jank.
Detailed analysis is as follows. To begin with, let us define "latency" as the number of frames it takes from starting drawing frame to ending rasterization. Now, what happens when latency temporarily drops to 1 for one or some frames, while it is 2 in other frames? This is separted to two parts: latency decrease (2->1) and increase (1->2).
The decrease itself does not introduce jank, but causes a uncomfortable "jumping" feeling from the user (will be discussed in the "linearlity" section later). For example, say frame a (0.00-16.67ms) has latency 2, frame b (16.67-33.33ms) has latency 2, and frame c (33.33-50.00ms) has latency 1. Then, at 33.33ms, content of frame a is displayed. However, at 50.00ms, both the content from frame b and frame c wants to be displayed to screen, so frame b will never be shown and only frame c is shown. If it is a linear moving animation with 1px per millisecond, we will see offset being 0 (frame a) at 33.33ms and offset being 33.33px (frame c) at 50.00ms, while we know all other frames will introduce an offset of 16.67px per frame. Thus a big jump happens.
As for the increase (1->2), it will introduce one jank. Suppose frame 0.00-16.67ms has latency 1, and 16.67-33.33ms has latency 2. Then, the rasterizer will provide new content to screen only at 16.67ms and 50.00ms, not at 33.33ms, and there is a jank.
Similar analysis holds for any latency change. For example, "1->2->1" latency change will cause a jank and then a uncomfortable big-jump.
Does this happen in real world? Yes, and quite frequently! I do observe it a lot of times in my tracing timeline. For example, the UI+rasterizer time may be near 16.67ms with fluctuation, then we do see a lot of 1->2 / 2->1 latency change. As another example, sometimes a frame may be much faster or slower to compute.
That is what this PR solves. Let's discuss by concrete numbers. Suppose latency is always 2 for a lot of frames, and suddenly in this frame latency drop to 1. Then, this PR will delay the rasterizer ending by sleeping (or can be changed to signaling or whatever you like). It will sleep (shortly speaking) to the next vsync, such that after the sleep, this frame has latency 2. No worries if the sleep happens to be a bit longer - it is still latency 2 if that happens.
Related: https://cjycode.com/flutter_smooth/benchmark/pitfall/latency-change
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests. --- I will add tests and refine code and enhance strategy etc after some code review - since review may request changing the code :)
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
flutter_smooth package is out now :)
https://github.com/fzyzcjy/flutter_smooth
(forgot to mention it here yesterday...)
Maybe another typo (2 char only)
Hope there exist a more lightweight way to fix typos...
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.
List which issues are fixed by this PR. You must list at least one issue.
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
My bad, it is correct :)
Fix janks caused by await vsync in classical Flutter
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
This fixes the jank happened in classical Flutter, even without the existence of flutter_smooth
I will add tests and refine code etc after some code review - since review may request changing the code :)
This works pretty well in flutter_smooth, see https://github.com/fzyzcjy/engine/blob/flutter-smooth/shell/common/animator.cc for full code.
During experiments, I observe a phenomenon: Even when the UI thread finishes everything before the deadline (vsync) a few milliseconds, the next frame is scheduled one vsync later, causing one jank. For example, UI thread may run from 0-15ms, but the next frame starts from 33.33ms instead of the correct 16.67ms.
An example screenshot can be seen at the end of this proposal. I added a timeline event, Animator::AwaitVSync, so we can clearly see when vsync await is called. (This screenshot has roughly 3ms space; but more frequently, I see this bug when there is about 0.5-2ms space.)
Therefore, this PR tries to fix this problem. The main idea is that, when detecting we are very near the next vsync, we do not wait at all, but instead directly start the next frame.
zoom in:
further zoom in:
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests. -- see above
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Remove (3N-1) jank and big-jump when N rasterization misses deadline
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
This fixes the jank happened in classical Flutter, even without the existence of flutter_smooth
This PR works with https://github.com/flutter/engine/pull/36438
This optimization holds for both classical Flutter and flutter_smooth - indeed the figure below is for classical Flutter.
In experiments, I do see rasterization takes longer time once in a while, instead of having the exact same duration. Experiments show that, a portion of rasterization ends a little bit later than the deadline (the vsync), while all others meet the deadline.
The following figure demonstrates the case. Given that this code change is unrelated to flutter_smooth, the scenario assumes UI is fast and no flutter_smooth exist at all. If using flutter_smooth, things are similar indeed. The first row is the case without code change to animator.cc, and the second row is the case with (1) this change (2) plus the https://github.com/flutter/engine/pull/36837 change.
Consider the frame starting at time 1. In the first row, when the rasterization misses the deadline a little bit (seen in time 2-3), there is nothing new to be shown to the screen, so time 2-3 yields a jank. This is inevitable and also holds for the second row - indeed the only jank in the second row.
Now consider the frame starting at time 2. It yields a big jump in classical Flutter, because the scene "1" (rasterized at about time 3.1) never has a chance to be shown to the screen. The second row does not have the problem because of the deliberate sleep.
Then comes the frame starting at time 3. In classical Flutter, the Animator::BeginFrame early returns, and thus no Dart pipeline is run, because it detects the pipeline is full. The pipeline is full because it is occupied with both the frame around 1-3.1 and the frame around 2-3.9. However, we are too pessimisitic about this - even though the pipeline is full at the beginning of BeginFrame, it may not be full at the end when we really need to call Animator::Render and enqueue a real scene to rasterizer. Thus, the classical Flutter (row 1) voluntarily give up a whole frame causing a jank, while the proposed solution runs the normal pipeline and produce a new scene.
Next is the frame starting at time 4, which we again assume its rasterization misses the deadline a little bit. All frames starting at this one indeed mimics the analysis above, so we do not repeate here. The interesting thing is that, the proposed solution no longer yields a jank anymore.
So, if we count the numbers, there are 3N janks in the first row (where N is the number of slightly missing deadline), and only 1 jank in the second row.
The drawback is that, the latency is increased by one frame, until the end of current frame chain (such as when animation finally finishes). However, when scrolling or touching, this seems better than having a large annoying jump in the UI - which is directly perceptible by human eyes easily. My test mobile phone has intrinsic (i.e. OS/hardware constraints) touch event latency of about 100ms, so adding 16ms to it looks almost non-distinguishable. Of course, if someone is developing a game, having low latency may be more important.
The same analysis also holds for any "latency changes from 2 to 3 to 2" scenario. For example, the "latency being 3" may last for more than one frame (contrary to the figure), with flutter_smooth.
As a remark, flutter_smooth is indeed implicitly doing something similar when in the middle of a plain jank frame. As we know, when a preempt render is about to start (analogy to "when Animator::BeginFrame is called"), we never skip it if pipeline is full (analyogy to the code change to BeginFrame). This works well in experiments.
P.S. Indeed, this is not something caused by flutter_smooth (since it is rasterizer slowness instead of build/layout slowness), but I have found a way trying to improve it.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests. -- see above
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Add peekPointerDataPacket to get pointer data packets earlier
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
This is needed by flutter_smooth in order to handle events. More specifically, when we are in the middle of a long jank frame, flutter_smooth will do preempt render. When doing preempt render, it needs to read and dispatch the pointer events, otherwise the UI will not respond to user fingers at all.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests. -- see above
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Provide fallback vsync target time for window.render
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
This works together with a few other PRs: https://github.com/flutter/engine/pull/36438 (to support multi render), https://github.com/flutter/engine/pull/36837 (which consumes vsync tagret time).
In order to make https://github.com/flutter/engine/pull/36837 and other Flutter logic work, we need to provide the correct vsync target time. However, currently the fallback target time is filled as the current time, which is surely incorrect and caused bugs for things like https://github.com/flutter/engine/pull/36837.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests. -- see above
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Fix incorrect newline in pull request template
Just one char fix :)
Before this fix, it looks like the following on github, with a bug newline:
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Fix errors when using multiple build/pipeline owners
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
Close https://github.com/flutter/flutter/issues/114002
We all know that, Flutter allows us to create our own BuildOwner and PipelineOwner, and here is even an official example: https://github.com/flutter/flutter/blob/master/examples/api/lib/widgets/framework/build_owner.0.dart. The doc also agrees with that. For example:
You can create other pipeline owners to manage off-screen objects, which can flush their pipelines independently of the on-screen render objects. (https://api.flutter.dev/flutter/rendering/PipelineOwner-class.html)
And
Additional build owners can be built to manage off-screen widget trees. (https://api.flutter.dev/flutter/widgets/BuildOwner-class.html)
Therefore, theoretically, we should be able to happily use our own BuildOwner and PipelineOwner anywhere freely. However, it has a bug as follows: If I call pipelineOwner.flushPaint(); (and sibling methods) inside the layout phase of the main PipelineOwner, then I get an assertion error in debug mode.
The root cause is that, even though the self-managed PipelineOwner is isolated from the flutter-managed PipelineOwner, the debug variable RenderObject.debugActiveLayout is shared. Therefore, when calling flushPaint on self-manged PipelineOwner within a call of flushLayout of flutter-managed PipelineOwner, the assertions get confused and wrongly throws.
My hack can be seen in https://github.com/fzyzcjy/flutter_smooth/blob/0c5db0ff270aa0c8cff28ea19055999627a8df6d/packages/smooth/lib/src/infra/auxiliary_tree_pack.dart#L214. Copy it here for completeness:
...
_temporarilyRemoveDebugActiveLayout(() {
pipelineOwner.flushPaint();
});
...
void _temporarilyRemoveDebugActiveLayout(VoidCallback f) {
// NOTE we have to temporarily remove debugActiveLayout
// b/c [SecondTreeRootView.paint] is called inside [preemptRender]
// which is inside main tree's build/layout.
// thus, if not set it to null we will see error
// https://github.com/fzyzcjy/yplusplus/issues/5783#issuecomment-1254974511
// In short, this is b/c [debugActiveLayout] is global variable instead
// of per-tree variable
// and also
// https://github.com/fzyzcjy/yplusplus/issues/5793#issuecomment-1256095858
final oldDebugActiveLayout = RenderObject.debugActiveLayout;
RenderObject.debugActiveLayout = null;
try {
f();
} finally {
RenderObject.debugActiveLayout = oldDebugActiveLayout;
}
}
However, for this to work, the debugActiveLayout setter must be public.
The most naive solution is to make it public, but that may violate encapsulation. Thus, in the proposed PR, I create a method to wrap that.
Reproduction code and output
Details
import 'package:flutter/foundation.dart';
import 'package:flutter/material.dart';
import 'package:flutter/rendering.dart';
import 'package:flutter_test/flutter_test.dart';
void main() {
testWidgets('When call PipelineOwner.flushPaint inside another PipelineOwner.flushLayout', (tester) async {
int onPerformLayoutCount = 0;
await tester.pumpWidget(_SpyLayoutBuilder(onPerformLayout: () {
onPerformLayoutCount++;
const Widget widget = ColoredBox(color: Colors.green, child: SizedBox(width: 100, height: 100));
// mimic https://github.com/flutter/flutter/blob/master/examples/api/lib/widgets/framework/build_owner.0.dart
final PipelineOwner pipelineOwner = PipelineOwner();
final MeasurementView rootView = pipelineOwner.rootNode = MeasurementView();
final BuildOwner buildOwner = BuildOwner(focusManager: FocusManager());
final RenderObjectToWidgetElement<RenderBox> element = RenderObjectToWidgetAdapter<RenderBox>(
container: rootView,
debugShortDescription: '[root]',
child: widget,
).attachToRenderTree(buildOwner);
rootView.scheduleInitialLayout();
rootView.scheduleInitialPaint(TransformLayer(transform: Matrix4.identity())..attach(rootView));
buildOwner.buildScope(element);
pipelineOwner.flushLayout();
pipelineOwner.flushPaint();
}));
expect(onPerformLayoutCount, 1);
});
}
class MeasurementView extends RenderBox with RenderObjectWithChildMixin<RenderBox> {
void performLayout() {
assert(child != null);
child!.layout(const BoxConstraints(), parentUsesSize: true);
size = child!.size;
}
void paint(PaintingContext context, Offset offset) {
print('hi ${describeIdentity(this)}.paint');
context.paintChild(child!, offset);
}
bool get isRepaintBoundary => true;
Rect get paintBounds => Offset.zero & size;
void debugAssertDoesMeetConstraints() => true;
}
class _SpyLayoutBuilder extends SingleChildRenderObjectWidget {
final VoidCallback onPerformLayout;
const _SpyLayoutBuilder({required this.onPerformLayout});
_RenderSpyLayoutBuilder createRenderObject(BuildContext context) => _RenderSpyLayoutBuilder(
onPerformLayout: onPerformLayout,
);
void updateRenderObject(BuildContext context, _RenderSpyLayoutBuilder renderObject) {
renderObject.onPerformLayout = onPerformLayout;
}
}
class _RenderSpyLayoutBuilder extends RenderProxyBox {
_RenderSpyLayoutBuilder({
required this.onPerformLayout,
RenderBox? child,
}) : super(child);
VoidCallback onPerformLayout;
void performLayout() {
super.performLayout();
onPerformLayout();
}
}
yields
Details
00:08 +0: When call PipelineOwner.flushPaint inside another PipelineOwner.flushLayout
══╡ EXCEPTION CAUGHT BY RENDERING LIBRARY ╞═════════════════════════════════════════════════════════
The following assertion was thrown during performLayout():
RenderBox.size accessed beyond the scope of resize, layout, or permitted parent access. RenderBox
can always access its own size, otherwise, the only object that is allowed to read RenderBox.size is
its parent, if they have said they will. It you hit this assert trying to access a child's size,
pass "parentUsesSize: true" to that child's layout().
'package:flutter/src/rendering/box.dart':
Failed assertion: line 2009 pos 13: 'debugDoingThisResize || debugDoingThisLayout ||
_computingThisDryLayout ||
(RenderObject.debugActiveLayout == parent && size._canBeUsedByParent)'
Either the assertion indicates an error in the framework itself, or we should provide substantially
more information in this error message to help you determine and fix the underlying cause.
In either case, please report this assertion by filing a bug on GitHub:
https://github.com/flutter/flutter/issues/new?template=2_bug.md
The relevant error-causing widget was:
_SpyLayoutBuilder
_SpyLayoutBuilder:file:///Users/tom/Main/yplusplus/frontend/yplusplus/test/a.dart:9:29
When the exception was thrown, this was the stack:
#2 RenderBox.size.<anonymous closure> (package:flutter/src/rendering/box.dart:2009:13)
#3 RenderBox.size (package:flutter/src/rendering/box.dart:2022:6)
#4 MeasurementView.paintBounds (file:///Users/tom/Main/yplusplus/frontend/yplusplus/test/a.dart:53:41)
#5 PaintingContext._repaintCompositedChild (package:flutter/src/rendering/object.dart:154:56)
#6 PaintingContext.repaintCompositedChild (package:flutter/src/rendering/object.dart:98:5)
#7 PipelineOwner.flushPaint (package:flutter/src/rendering/object.dart:1116:31)
#8 main.<anonymous closure>.<anonymous closure> (file:///Users/tom/Main/yplusplus/frontend/yplusplus/test/a.dart:28:21)
#9 _RenderSpyLayoutBuilder.performLayout (file:///Users/tom/Main/yplusplus/frontend/yplusplus/test/a.dart:86:20)
#10 RenderObject.layout (package:flutter/src/rendering/object.dart:2135:7)
#11 RenderBox.layout (package:flutter/src/rendering/box.dart:2418:11)
#12 RenderView.performLayout (package:flutter/src/rendering/view.dart:170:14)
#13 RenderObject._layoutWithoutResize (package:flutter/src/rendering/object.dart:1973:7)
#14 PipelineOwner.flushLayout (package:flutter/src/rendering/object.dart:999:18)
#15 AutomatedTestWidgetsFlutterBinding.drawFrame (package:flutter_test/src/binding.dart:1194:23)
#16 RendererBinding._handlePersistentFrameCallback (package:flutter/src/rendering/binding.dart:378:5)
#17 SchedulerBinding._invokeFrameCallback (package:flutter/src/scheduler/binding.dart:1175:15)
#18 SchedulerBinding.handleDrawFrame (package:flutter/src/scheduler/binding.dart:1104:9)
#19 AutomatedTestWidgetsFlutterBinding.pump.<anonymous closure> (package:flutter_test/src/binding.dart:1057:9)
#22 TestAsyncUtils.guard (package:flutter_test/src/test_async_utils.dart:71:41)
#23 AutomatedTestWidgetsFlutterBinding.pump (package:flutter_test/src/binding.dart:1043:27)
#24 WidgetTester.pumpWidget.<anonymous closure> (package:flutter_test/src/widget_tester.dart:554:22)
#27 TestAsyncUtils.guard (package:flutter_test/src/test_async_utils.dart:71:41)
#28 WidgetTester.pumpWidget (package:flutter_test/src/widget_tester.dart:551:27)
#29 main.<anonymous closure> (file:///Users/tom/Main/yplusplus/frontend/yplusplus/test/a.dart:9:18)
#30 testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:171:29)
<asynchronous suspension>
<asynchronous suspension>
(elided 7 frames from class _AssertionError, dart:async, and package:stack_trace)
The following RenderObject was being processed when the exception was fired: _RenderSpyLayoutBuilder#f19a8:
creator: _SpyLayoutBuilder ← [root]
parentData: <none>
constraints: BoxConstraints(w=800.0, h=600.0)
size: Size(800.0, 600.0)
This RenderObject has no descendants.
════════════════════════════════════════════════════════════════════════════════════════════════════
00:08 +0 -1: When call PipelineOwner.flushPaint inside another PipelineOwner.flushLayout [E]
Test failed. See exception logs above.
The test description was: When call PipelineOwner.flushPaint inside another PipelineOwner.flushLayout
To run this test again: /Users/tom/fvm/versions/3.3.5/bin/cache/dart-sdk/bin/dart test /Users/tom/Main/yplusplus/frontend/yplusplus/test/a.dart -p vm --plain-name 'When call PipelineOwner.flushPaint inside another PipelineOwner.flushLayout'
00:08 +0 -1: Some tests failed.
Performance overhead
Using compiler explorer, we can see that it does not generate worse assembly (as long as we use the prefer-inline pragma)
https://godbolt.org/z/EoznoWex7
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt. -- see above
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Expose Ticker.startTime so users know the start time and the absolute time
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
Currently, the user of Ticker only knows the elapsed time. However, it looks reasonable to allow the user to know when the ticker thinks it starts ticking.
If this does not wanted to be widely used, maybe we can mark it as @protected or @visibleForTesting or @experimental.
As for where it is needed inside flutter_smooth, it is utilized to know the relative time between a few Tickers as well as the system. In other words, when Ticker A says it elapsed 1 second, flutter_smooth needs to know the startTime such that it knows what "1second" means in absolute time. Detailed code can be seen in https://github.com/fzyzcjy/flutter_smooth/blob/0c5db0ff270aa0c8cff28ea19055999627a8df6d/packages/smooth/lib/src/drop_in/list_view/shift.dart#L352-L356
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Fix bug thattimeDilation is not reset, causing subsequent test errors, and add verifications to ensure such problem does not exist in the future
Some tests set the time dilation to be non-one, but is not reset after the test ends. Thus, every test after it will see very weird time. I find this bug because got trapped in https://github.com/flutter/flutter/pull/113828.
The added line, timeDilation = 1.0; // restore time dilation, or it will affect other tests, is copied from image_stream_test.dart's 'timeDilation affects animation frame timers' test.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Allow GestureBinding subclasses to know hitTest information
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
This PR allows the subclasses of GestureBinding to read the hitTest information. It is directly needed in flutter_smooth, because flutter_smooth has extra call to dispatchEvent and only execute those who are in auxiliary tree (and omit those in the main tree) during preempt render. I can copy-and-paste the content of dispatchEvent to mimic the behavior, but there is one missing piece: the hitTest information. By adding this PR, it can work.
An alternative solution, which the flutter-smooth branch is currently using (but more hacky), may be to add a filter to dispatchEvent. Then, we can utilize the filter to skip those RenderObjects in the main tree.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt. -- see above
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Enable a frame to be scheduled immediately
- I will finish code details, refine code, add tests, make tests pass, etc, after a code review that thinks the rough idea is acceptable. It is because, from my past experience, reviews may request changing a lot. If the general idea is to be changed, all detailed implementation efforts are wasted :)
- The PR has an already-working counterpart, and it produces ~60FPS smooth experimental results. The benchmark results and detailed analysis is in chapter https://cjycode.com/flutter_smooth/benchmark/. All the source code is in https://github.com/fzyzcjy/engine/tree/flutter-smooth and https://github.com/fzyzcjy/flutter/tree/flutter-smooth.
- Possibly useful as a context to this PR, there is a whole chapter discussing the internals - how flutter_smooth is implemented. (Link: https://cjycode.com/flutter_smooth/design/)
This PR depends on the merging of https://github.com/flutter/engine/pull/36911.
This PR is needed by flutter_smooth, because of the "Brake" mechanism discussed in https://github.com/fzyzcjy/flutter_smooth/blob/feat%2Fdoc-insight/website/docs/design/infra/brake/intro.md (TODO@fzyzcjy: post website link when it is published). In short, for that mechanism to work without jank, a frame must be able to be started immediately instead of waiting for the next vsync (otherwise we must have a jank).
More specifically, let's analyze the figure in the Brake mechanism. The red arrow points where we need this PR. If this PR is not there, the frame cannot start at time=2.6, but have to start at time=3. Then, even though we have preempt render mechanism, we are not able to produce scene and rasterizer quick enough before time=4, so we will have a jank.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests. -- see above
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Fix wasted memory caused by debug fields - 16 bytes per object (when adding that should-be-removed field crosses double-word alignment)
Close #113940
Theoretical analysis
Consider the following example code. Code with "case A" occupies 2.5GB memory, while "case B" is 4.0GB memory.
Details
class C {
int? a;
int? b;
// int get computed => a.hashCode; // case A
int get computed => a.hashCode ^ b.hashCode; // case B
}
Future<void> main() async {
final arr = <C>[];
for (var i = 0; i < 100000000; ++i) {
arr.add(C()
..a = 42
..b = 100);
}
print('hash=${Object.hashAll(arr.map((e) => e.computed))}');
print('sleep...');
await Future<void>.delayed(const Duration(seconds: 10000));
}
Therefore, we know that, a field does not occupy memory if there is no read/write to it. For example, the field b in case A is never read, so Dart compiler seems so smart that it knows it can be eliminated and no memory is allocated for that. By the way, b is written even in case A, but seems memory is not allocated as long as it is not read. This agrees with common sense (but since I am not compiler expert, feel free to correct me if I am wrong!).
For fields inside Flutter that is merely used for debug, they are usually accessed inside an assert(() { ... }()); block. That is great, because if each and every field access are inside assert, those code will be eliminated in release build, and by discussions above, we will not pay memory for those debug fields.
However, AnimationController.debugLabel does not seem to follow this. Inside AnimationController.toStringDetails, it uses debugLabel field without being inside a assert block. Therefore, IMHO, we will be paying the memory of debugLabel even if we never use that in runtime.
As for why toStringDetails is guaranteed to be called, it is simple: Animation.toString calls that toStringDetails, and we know toString will not be tree shaken out.
Experimental analysis
Setup
Use these code:
Details
Need to add two dummy fields to AnimationController, such that it crosses the double-word alignment. (Yes, this PR does not affect memory for today's AnimationController, but who can guarantee it never have two more fields or two less fields, which this PR will reduce 16 bytes).
class AnimationController {
int? dummy1;
int? dummy2;
... old code ...
}
Use this code
import 'package:flutter/material.dart';
import 'package:flutter/scheduler.dart';
void main() => runApp(const MyApp());
class MyApp extends StatefulWidget {
const MyApp({Key? key}) : super(key: key);
State<MyApp> createState() => _MyAppState();
}
class _MyAppState extends State<MyApp> {
final arr = <AnimationController>[];
void initState() {
super.initState();
for (var i = 0; i < 2000000; ++i) arr.add(AnimationController(vsync: const TestVSync()));
print('${arr.first.dummy1} ${arr.first.dummy2}'); // ensure the dummy fields are not removed
arr.first
..duration = const Duration(seconds: 1)
..forward();
}
Widget build(BuildContext context) => Container();
}
class TestVSync implements TickerProvider {
const TestVSync();
Ticker createTicker(TickerCallback onTick) => Ticker(onTick);
}
Operations
flutter build apk --extra-gen-snapshot-options='--print-object-layout-to=object_layout.json'and look at json to know memory layoutflutter run --profileto know memory consumption at runtime
Results
Without PR
96 bytes per object
Details
{
"class": "AnimationController",
"size": 96,
"fields": [
{
"field": "dummy1",
"offset": 20
},
{
"field": "dummy2",
"offset": 24
},
{
"field": "lowerBound",
"offset": 28
},
{
"field": "upperBound",
"offset": 36
},
{
"field": "debugLabel",
"offset": 44
},
{
"field": "animationBehavior",
"offset": 48
},
{
"field": "duration",
"offset": 52
},
{
"field": "reverseDuration",
"offset": 56
},
{
"field": "_ticker",
"offset": 60
},
{
"field": "_simulation",
"offset": 64
},
{
"field": "_value",
"offset": 68
},
{
"field": "_direction",
"offset": 72
},
{
"field": "_status",
"offset": 76
},
{
"field": "_lastReportedStatus",
"offset": 80
}
]
},
and
With this PR
80 bytes per object
Details
{
"class": "AnimationController",
"size": 80,
"fields": [
{
"field": "dummy1",
"offset": 20
},
{
"field": "dummy2",
"offset": 24
},
{
"field": "lowerBound",
"offset": 28
},
{
"field": "upperBound",
"offset": 36
},
{
"field": "animationBehavior",
"offset": 44
},
{
"field": "duration",
"offset": 48
},
{
"field": "reverseDuration",
"offset": 52
},
{
"field": "_ticker",
"offset": 56
},
{
"field": "_simulation",
"offset": 60
},
{
"field": "_value",
"offset": 64
},
{
"field": "_direction",
"offset": 68
},
{
"field": "_status",
"offset": 72
},
{
"field": "_lastReportedStatus",
"offset": 76
}
]
},
and
Conclusion
16B reduce per object.
Without PR
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
This pull request has been changed to a draft. The currently pending flutter-gold status will not be able to resolve until a new commit is pushed or the change is marked ready for review again.
For more guidance, visit Writing a golden file test for package:flutter.
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
You'll need either a test or a test exception.
@jonahwilliams Thanks, sent https://discord.com/channels/608014603317936148/608018585025118217/1034242780857376799
test-exemption: optimization
auto label is removed for flutter/flutter, pr: 113927, due to - Please get at least one approved review if you are already a member or two member reviews if you are not a member before re-applying this label. Reviewers: If you left a comment approving, please use the "approve" review action instead.
auto label is removed for flutter/flutter, pr: 113927, due to Validations Fail.
Let _debugCheckNotUsedAsOldLayer provide hashcode in addition to runtime type
I see this assert fail when working on my own app today (unrelated to flutter_smooth; work on stable channel). With identity hashcode, we can surely know more about which layer violates.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - [
- x] I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Fix addToScene documentation
Its return type is void, but doc says "return the engine layer", so I guess the doc is outdated.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
[WIP] Fix Layer ... was previously used as oldLayer, caused by LeaderLayer addToScene bug
Close https://github.com/flutter/flutter/issues/113995 Please see the issue for a bug reproduction. Here I will discuss how it is solved and what caused it.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Layer ... was previously used as oldLayer assertion error in debug mode, and page being blank in release mode, caused by LeaderLayer addToScene bug
Close https://github.com/flutter/flutter/issues/113995 Please see the issue for a bug reproduction. Here I will discuss how it is solved and what caused it.
The real world bug
My app has a part of it not rendered (i.e. page blank) sometimes, which is very weird. After I manage to reproduce it in debug environment, the error is:
Details
======== Exception caught by scheduler library =====================================================
The following assertion was thrown during a scheduler callback:
Layer ClipRectEngineLayer was previously used as oldLayer.
Once a layer is used as oldLayer, it may not be used again. Instead, after calling one of the SceneBuilder.push* methods and passing an oldLayer to it, use the layer returned by the method as oldLayer in subsequent frames.
'dart:ui/compositing.dart':
Failed assertion: line 88 pos 9: '<optimized out>'
Either the assertion indicates an error in the framework itself, or we should provide substantially more information in this error message to help you determine and fix the underlying cause.
In either case, please report this assertion by filing a bug on GitHub:
https://github.com/flutter/flutter/issues/new?template=2_bug.md
When the exception was thrown, this was the stack:
#2 _EngineLayerWrapper._debugCheckNotUsedAsOldLayer (dart:ui/compositing.dart:88:9)
#3 SceneBuilder.addRetained.<anonymous closure>.recursivelyCheckChildrenUsedOnce (dart:ui/compositing.dart:656:21)
#4 List.forEach (dart:core-patch/growable_array.dart:416:8)
#5 SceneBuilder.addRetained.<anonymous closure>.recursivelyCheckChildrenUsedOnce (dart:ui/compositing.dart:662:18)
#6 SceneBuilder.addRetained.<anonymous closure> (dart:ui/compositing.dart:665:7)
#7 SceneBuilder.addRetained (dart:ui/compositing.dart:668:6)
#8 Layer._addToSceneWithRetainedRendering (package:flutter/src/rendering/layer.dart:643:15)
#9 ContainerLayer.addChildrenToScene (package:flutter/src/rendering/layer.dart:1241:13)
#10 OffsetLayer.addToScene (package:flutter/src/rendering/layer.dart:1378:5)
#11 Layer._addToSceneWithRetainedRendering (package:flutter/src/rendering/layer.dart:646:5)
#12 ContainerLayer.addChildrenToScene (package:flutter/src/rendering/layer.dart:1241:13)
#13 OffsetLayer.addToScene (package:flutter/src/rendering/layer.dart:1378:5)
#14 Layer._addToSceneWithRetainedRendering (package:flutter/src/rendering/layer.dart:646:5)
#15 ContainerLayer.addChildrenToScene (package:flutter/src/rendering/layer.dart:1241:13)
#16 EnhancedLeaderLayer.addToScene (package:flutter_portal/src/enhanced_composited_transform/flutter_src/rendering_layer.dart:159:5)
#17 Layer._addToSceneWithRetainedRendering (package:flutter/src/rendering/layer.dart:646:5)
#18 ContainerLayer.addChildrenToScene (package:flutter/src/rendering/layer.dart:1241:13)
#19 EnhancedLeaderLayer.addToScene (package:flutter_portal/src/enhanced_composited_transform/flutter_src/rendering_layer.dart:159:5)
#20 Layer._addToSceneWithRetainedRendering (package:flutter/src/rendering/layer.dart:646:5)
#21 ContainerLayer.addChildrenToScene (package:flutter/src/rendering/layer.dart:1241:13)
#22 EnhancedLeaderLayer.addToScene (package:flutter_portal/src/enhanced_composited_transform/flutter_src/rendering_layer.dart:159:5)
#23 Layer._addToSceneWithRetainedRendering (package:flutter/src/rendering/layer.dart:646:5)
#24 ContainerLayer.addChildrenToScene (package:flutter/src/rendering/layer.dart:1241:13)
#25 EnhancedLeaderLayer.addToScene (package:flutter_portal/src/enhanced_composited_transform/flutter_src/rendering_layer.dart:159:5)
#26 Layer._addToSceneWithRetainedRendering (package:flutter/src/rendering/layer.dart:646:5)
#27 ContainerLayer.addChildrenToScene (package:flutter/src/rendering/layer.dart:1241:13)
#28 OffsetLayer.addToScene (package:flutter/src/rendering/layer.dart:1378:5)
#29 Layer._addToSceneWithRetainedRendering (package:flutter/src/rendering/layer.dart:646:5)
#30 ContainerLayer.addChildrenToScene (package:flutter/src/rendering/layer.dart:1241:13)
#31 TransformLayer.addToScene (package:flutter/src/rendering/layer.dart:1834:5)
#32 ContainerLayer.buildScene (package:flutter/src/rendering/layer.dart:1054:5)
#33 RenderView.compositeFrame (package:flutter/src/rendering/view.dart:231:37)
#34 RendererBinding.drawFrame (package:flutter/src/rendering/binding.dart:517:18)
#35 WidgetsBinding.drawFrame (package:flutter/src/widgets/binding.dart:884:13)
#36 RendererBinding._handlePersistentFrameCallback (package:flutter/src/rendering/binding.dart:378:5)
#37 SchedulerBinding._invokeFrameCallback (package:flutter/src/scheduler/binding.dart:1175:15)
#38 SchedulerBinding.handleDrawFrame (package:flutter/src/scheduler/binding.dart:1104:9)
#39 SchedulerBinding._handleDrawFrame (package:flutter/src/scheduler/binding.dart:1015:5)
#40 _invoke (dart:ui/hooks.dart:148:13)
#41 PlatformDispatcher._drawFrame (dart:ui/platform_dispatcher.dart:318:5)
#42 _drawFrame (dart:ui/hooks.dart:115:31)
(elided 2 frames from class _AssertionError)
====================================================================================================
The minimal reproduction
(I will omit how I find out the root cause. If you are interested I can write down.)
Please see https://github.com/flutter/flutter/issues/113995 with minimal reproduction code
Why that causes bug
The first pumpWidget creates initial tree, and I deliberately set padding to non-zero, such that LeaderLayer.paint will see non-zero Offset, and thus will pushTransform.
In the second pumpWidget, I deliberately set padding to zero. Then, LeaderLayer.paint will not call pushTransform. Correct implementation should set engineLayer to null, but the old code just leave that variable unchanged.
Then comes the third pumpWidget. I change the color inside a RepaintBoundary which is the sibling of LeaderLayer (CompositedTransformTarget). This is carefully constructed (notice the sibling and the RepaintBoundary) to reproduce the following behavior in real-world complicated app: We should (1) ensure LeaderLayer. _addToSceneWithRetainedRendering is called, and (2) ensure LeaderLayer. _needsAddToScene = false. For example, if we do not add that RepaintBoundary, it will not construct the case, because CompositedTransformTarget's RO will repaint and thus _needsAddToScene becomes true.
Then, interesting thing happens. Look at the code:
void _addToSceneWithRetainedRendering(ui.SceneBuilder builder) {
if (!_needsAddToScene && _engineLayer != null) {
builder.addRetained(_engineLayer!);
return;
}
...
We have constructed a case, such that the if is true. Notice that, if we fix the bug (just as what we do in the PR), the engineLayer will be null so if condition will not be true. Thus, buggy code will call LeaderLayer.addRetained, while correct code will not.
Then the error happens. Notice that, this LeaderLayer._engineLayer indeed is the layer constructed in the second pumpWidget instead of the third (i.e. it is stale from last frame). Therefore, this EngineLayer's children are all from the stale previous frame. For example, the ClipRectLayer from previous frame (second pumpWidget). Since it has already been used as oldLayer in RenderClipRect, that old stale ClipRectLayer should never be used. However we are now using it in addRetained. Therefore, no wonder when addRetained is checking all subtree ensuring _debugCheckNotUsedAsOldLayer, it fails the assertions.
Why is the solution valid
With analysis above, we can clearly see this solves the bug. In addition, looking at ClipRectLayer etc, they also have such "engineLayer = null" logic, so even if only mimicking those we should do this.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Hmm it gets auto closed after I rename branch. Anyway please see https://github.com/flutter/flutter/pull/113998
Introduce debugWithActiveLayoutCleared to avoid duplicated code
This is indeed a part of https://github.com/flutter/flutter/pull/113817. However, that one actually does two things in one PR: refactor internal code, and exposes an API. Thus, in order to follow the spirit mentioned in Flutter - one PR for only one thing - I make this refactor a separate PR, which should be easier to review.
Performance difference
As is discussed in https://github.com/flutter/flutter/pull/113817#issue-1417868505 via compiler explorer, the generate assembly is not worse as long as we use that prefer-inline annotation.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
TestWidgetsFlutterBinding._verifyInvariants could be the right place to check for this.
@Piinks Yes I think so, @goderbauer mentions the _verifyInvariants - the one I have seen a ton of times reporting I changed somthing (e.g. fake screen resolution) but do not change it back
I will update soon.
Anyway this is not a big problem, so I will close it now since have a lot of other more important PRs remaining for me to improve :)
The CI failure, "infra failed", does not look like a problem of my code indeed
CI passes
Looks like this was changed in https://github.com/flutter/flutter/pull/36402, but we never updated the doc (cc @yjbanov).
Please ignore this PR for now
I am going to close this then for now to get it off the queue. Feel free to reopen when you can provide more context.
In isolation, the API this exposes doesn't make a lot of sense to me and it feels hacky.
If you chose to continue working on this PR, please also take another look at the flutter style guide, especially the sections around API documentation: https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo#documentation-dartdocs-javadocs-etc.
I'm still not quite clear on why this is a solution that should live in the framework.
Tickers have no means of interrupting work more often. Tickers have no special knowledge of when vsync happens. This seems to introduce a number of problems that will be difficult for developers to reason about. I think this could be solved in a package instead. I'm going to close this for now but let's discuss more on discord if you like.
Without this, how do you measure frame timings in profile mode?
Looks like this was changed in https://github.com/flutter/flutter/pull/36402, but we never updated the doc (cc @yjbanov).
I agree. That's why the PR happens :)
Sure, thanks!
@dnfield Without this, how do you measure frame timings in profile mode?
I personally by the timeline data. You know, look at the GPURasterizer::Draw rectangle inside timeline, etc.
p.s. partially related https://github.com/flutter/devtools/issues/4522
@goderbauer I find adding this a bit dubios. The environment flag will be really hard to discover.
Hmm why are other env flags in flutter easy to discover, by adding doc or something else? I can do the same, then this is no problem.
Also, profile mode exists so you can get these kind of performance metrics out of your app.
As mentioned above, these perf metrics are already exposed via the timeline data, such as by looking at GPURasterizer::Draw.
If you don't want that, there's always release mode...
But we need to do profiling to get profile data, don't we :)
Rephrase the problem: We are providing redundant data (i.e. report timing, even if timeline tracing already has the data), and that redundancy is causing measurable speed drop compared with release mode. Then we are biasing the profiling result.
So, if this is not to be merged, maybe we should create another PR to the doc site, with something like: "Please do not believe the speed in profile mode. It will be measureably slower than release mode." But IMHO users will not like that sentence.
Thank you, I will reconsider the API and discuss on discord.
Look like I can remove the environment variable.
In Dart, if we have a normal variable that only has one constant value, then the field will be removed. https://github.com/dart-lang/sdk/issues/50287#issuecomment-1289027245 "Also AOT compiler is capable of removing unused fields of various kinds (the field is also effectively unused if it always contains the same constant value or is only written into, but never read)."
The experiment confirms this:
https://godbolt.org/z/Thd91YcPj
I will update the code shortly.
Fix incorrectly named "debug" prefix
Close #111874
Please read discussions there for why this PR is made :)
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Avoid future bugs about wrong manipulation of engineLayer inside addToScene
In https://github.com/flutter/flutter/pull/113998, I have fixed the bug caused by wrongly using engineLayer. But that bug is so time consuming to locate, so I write this PR to add assertions so we will not add such bugs in the future anymore.
Question: In order to insert assertions, I have to wrap the addToScene function. IMHO the best thing is addToScene vs performAddToScene (like layout vs performLayout so everyone is familiar). However, that is a breaking change. I am not a googler so not sure whether you like it or not?
IMHO the breaking change may be acceptable, because (1) it is merely a rename so simple to migrate (2) very few people write their own Layer class so this will affect few people.
(Currently, I temporarily create a addToSceneWrapped method in order to see whether the code works - but I guess this should not be the final name)
This assertion does work, because https://github.com/flutter/flutter/pull/114124/checks?check_run_id=9132589021 reports the following error, which is exactly the error that #113998 finds and fixes.
══╡ EXCEPTION CAUGHT BY SCHEDULER LIBRARY ╞═════════════════════════════════════════════════════════
The following assertion was thrown during a scheduler callback:
When addToScene previously configures the engineLayer, it should either update it in current
addToScene, or set it to null explicitly. Otherwise, Flutter framework may utilize that already
out-of-date engineLayer and thus cause problems. However, it is observed that
previousEngineLayer=TransformEngineLayer#38532153 while engineLayer=TransformEngineLayer#38532153.
This originates in LeaderLayer#204442042.
'package:flutter/src/rendering/layer.dart':
Failed assertion: line 673 pos 9: 'previousEngineLayer == null || previousEngineLayer !=
engineLayer'
Either the assertion indicates an error in the framework itself, or we should provide substantially
more information in this error message to help you determine and fix the underlying cause.
In either case, please report this assertion by filing a bug on GitHub:
https://github.com/flutter/flutter/issues/new?template=2_bug.md
When the exception was thrown, this was the stack:
#2 Layer.addToSceneWrapped.<anonymous closure> (package:flutter/src/rendering/layer.dart:673:9)
#3 Layer.addToSceneWrapped (package:flutter/src/rendering/layer.dart:683:6)
#4 Layer._addToSceneWithRetainedRendering (package:flutter/src/rendering/layer.dart:701:5)
#5 ContainerLayer.addChildrenToScene (package:flutter/src/rendering/layer.dart:1311:13)
#6 OffsetLayer.addToScene (package:flutter/src/rendering/layer.dart:1448:5)
#7 Layer.addToSceneWrapped (package:flutter/src/rendering/layer.dart:669:5)
#8 Layer._addToSceneWithRetainedRendering (package:flutter/src/rendering/layer.dart:701:5)
#9 ContainerLayer.addChildrenToScene (package:flutter/src/rendering/layer.dart:1311:13)
#10 OffsetLayer.addToScene (package:flutter/src/rendering/layer.dart:1448:5)
#11 Layer.addToSceneWrapped (package:flutter/src/rendering/layer.dart:669:5)
#12 Layer._addToSceneWithRetainedRendering (package:flutter/src/rendering/layer.dart:701:5)
#13 ContainerLayer.addChildrenToScene (package:flutter/src/rendering/layer.dart:1311:13)
#14 TransformLayer.addToScene (package:flutter/src/rendering/layer.dart:1941:5)
#15 Layer.addToSceneWrapped (package:flutter/src/rendering/layer.dart:669:5)
#16 ContainerLayer.buildScene (package:flutter/src/rendering/layer.dart:1124:5)
#17 RenderView.compositeFrame (package:flutter/src/rendering/view.dart:231:37)
#18 AutomatedTestWidgetsFlutterBinding.drawFrame (package:flutter_test/src/binding.dart:1257:26)
#19 RendererBinding._handlePersistentFrameCallback (package:flutter/src/rendering/binding.dart:375:5)
#20 SchedulerBinding._invokeFrameCallback (package:flutter/src/scheduler/binding.dart:1275:15)
#21 SchedulerBinding.handleDrawFrame (package:flutter/src/scheduler/binding.dart:1204:9)
#22 AutomatedTestWidgetsFlutterBinding.pump.<anonymous closure> (package:flutter_test/src/binding.dart:1096:9)
#25 TestAsyncUtils.guard (package:flutter_test/src/test_async_utils.dart:71:41)
#26 AutomatedTestWidgetsFlutterBinding.pump (package:flutter_test/src/binding.dart:1082:27)
#27 WidgetTester.pump.<anonymous closure> (package:flutter_test/src/widget_tester.dart:618:53)
#30 TestAsyncUtils.guard (package:flutter_test/src/test_async_utils.dart:71:41)
#31 WidgetTester.pump (package:flutter_test/src/widget_tester.dart:618:27)
#32 main.<anonymous closure> (file:///b/s/w/ir/x/w/flutter/packages/flutter/test/widgets/autocomplete_test.dart:491:18)
<asynchronous suspension>
<asynchronous suspension>
(elided 7 frames from class _AssertionError, dart:async, and package:stack_trace)
════════════════════════════════════════════════════════════════════════════════════════════════════
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
@dnfield Without this, how do you measure frame timings in profile mode?
I personally by the timeline data. You know, look at the GPURasterizer::Draw rectangle inside timeline, etc.
p.s. partially related flutter/devtools#4522
This method is significantly cheaper than the timeline.
auto label is removed for flutter/flutter, pr: 113998, due to - Please get at least one approved review if you are already a member or two member reviews if you are not a member before re-applying this label. Reviewers: If you left a comment approving, please use the "approve" review action instead.
auto label is removed for flutter/flutter, pr: 113998, due to Validations Fail.
cc @dnfield
Let's close this one out to get it off review queues until we have more consensus around the approach.
I'm going to close this to remove it from review queues. As I'm mentioning in some other PRs, let's discuss further on discord.
auto label is removed for flutter/engine, pr: 36822, due to - Please get at least one approved review if you are already a member or two member reviews if you are not a member before re-applying this label. Reviewers: If you left a comment approving, please use the "approve" review action instead.
auto label is removed for flutter/engine, pr: 36822, due to Validations Fail.
I started to review this, but then realized this seems to be at odds with some similar work that was done for iOS - see pointer_data_dispatcher.h and in particular the SmoothPointerDataDispatcher.
I think you're on to something here, but I wonder if we should be looking at making the SmoothPointerDataDispatcher work on more than just iOS at this point (as in, perhaps some Android phones are now doing what iOS was doing w.r.t. pointer events and screen vsync being different). This approach does not currently seem to be compatible with that, and I have some concerns about thread safety around it (you're capturing this and posting it to a different task runner in an unsafe manner).
I would suggest filing a specific bug for this issue with more details about which platform(s) you're observing this behavior on and how what you need is different from the SmoothPointerDataDispatcher (if it's different at all). If you're on Android, it would probably be worth doing an experiment where the smooth dispatcher is enabled there.
This approach is not passing tests and doesn't seem to be ready for review. It would need substantial reworking to be ready for review. I'm going to close this to get it off of review queues.
This seems very similar to https://github.com/flutter/engine/pull/29276
Similar concerns as to that one: we need some motivating benchmarks that clearly show what benefits would be gained here, and we need to see that changes to this won't negatively impact existing benchmarks/users.
This removes backpressure from the rasterizer and will make the application run hotter than necessary in a lot of cases. It'd be nice to improve things here, but we can't completely remove that backpressure safely.
This is not the only thing that would need peeking,a nd creating this kind of "pull" based model is a major architectural change we'd need more time to reason about. I'm going to close this PR to get it off of review queues for now - tests are not passing and it is not clear that this API is the desirable one to expose to developers to achieve the larger goals you're discussing.
Also re-kicked the infra failure :)
This would ideally be tested on the framework side, to make sure the output matches what we expect from other messages (we have a special matcher that knows how to normalize this kind of hash code).
That can't be in this PR, though.
test-exempt: test needs to be in another repo
Same comment as on the linked PR applies.
In particular, it's really hard to reason about how to use this parameter correctly. A lot of the work you're doing seems to be geared towards ignoring/overriding the vsync that the system gives us instead of cooperating with it more.
In particular, vsync is an important source of backpressure that we must respect. We cannot just keep shoving frames at the system compositor, we will overwhelm it and it will start dropping more frames than necessary/run into long GPU stalls. This will be really bad on older hardware, and still be bad on newer hardware.
Closing to remove from review queues.
auto label is removed for flutter/engine, pr: 36985, due to - Please get at least one approved review if you are already a member or two member reviews if you are not a member before re-applying this label. Reviewers: If you left a comment approving, please use the "approve" review action instead.
auto label is removed for flutter/engine, pr: 36985, due to Validations Fail.
@iskakaushik can you provide a secondary review here?
Sorry, this should probably be closed. There is usage of this API internally in google that needs some reworking before this can be updated. Thank you for the effort!
You are welcome!
You are welcome!
Hmm it still fails. I have seen this test failing across my PRs recently without reasons. let me bump ci
@dnfield This method is significantly cheaper than the timeline.
I originally thought so. But interestingly, seems no in a few reasons:
- timeline is usually there (or even cannot be disabled? not find a flag to disable), while report timings is an extra thing
- timeline makes app slower evenly, while report timings is a big burst. thus, flutter_smooth can handle timline slowness very easily (and still gets 60fps and smooth user feeling), while report timings will cause a jank (because it itself takes some long time and during that time flutter_smooth cannot perform any job to submit extra frames).
For disable tracing: Not see any flag to disable IMHO
flutter run --help | grep trace
--trace-startup Trace application startup, then exit, saving the trace to a file. By default, this will be saved in the "build" directory. If the FLUTTER_TEST_OUTPUTS_DIR environment variable is set, the file will be written there instead.
--endless-trace-buffer Enable tracing to an infinite buffer, instead of a ring buffer. This is useful when recording large traces. To use an endless buffer to record startup traces, combine this with "--trace-startup".
--trace-systrace Enable tracing to the system tracer. This is only useful on platforms where such a tracer is available (Android, iOS, macOS and Fuchsia).
--trace-skia Enable tracing of Skia code. This is useful when debugging the raster thread (formerly known as the GPU thread). By default, Flutter will not log Skia code, as it introduces significant overhead that may affect recorded performance metrics in a misleading way.
--[no-]await-first-frame-when-tracing Whether to wait for the first frame when tracing startup ("--trace-startup"), or just dump the trace as soon as the application is running. The first frame is detected by looking for a Timeline event with the name "Rasterized first useful frame". By default, the widgets library's binding takes care of sending this event.
--[no-]hot Run with support for hot reloading. Only available for debug mode. Not available with "--trace-startup".
@dnfield Sure.
Btw if you guys think the cost paid by CHECK is ok, I will change DCHECK to CHECK (pretty easy to change)
All right...
Thanks, I will discuss on discord about this
Thanks for the reply, I will ask on discord later
thanks, will discuss on discord further
but then realized this seems to be at odds with some similar work that was done for iOS - see pointer_data_dispatcher.h and in particular the SmoothPointerDataDispatcher.
Looking at https://github.com/flutter/engine/blob/6368dee0d78f4345e6cfdc4541754acb0891d845/shell/common/pointer_data_dispatcher.h#L157, seems that SmoothPointerDataDispatcher is just delaying one packet. On the other hand, this PR tries to merge multiple packets into one bigger packet, so we do not pay extra cost of PostTask, call Dart, etc.
then realized this seems to be at odds with some similar work that was done for iOS - see pointer_data_dispatcher.h and in particular the SmoothPointerDataDispatcher.
That one seems to be queueing
(you're capturing this and posting it to a different task runner in an unsafe manner).
This is just minor details and I will fix them definitely
I would suggest filing a specific bug for this issue with more details about which platform(s) you're observing this behavior on and how what you need is different from the SmoothPointerDataDispatcher (if it's different at all). If you're on Android, it would probably be worth doing an experiment where the smooth dispatcher is enabled there.
Thanks I will do that
It'd be really helpful if you could share some measurements (including device/platform/runtime mode).
CHECK will cause a crash at runtime. DCHECK is probably fine here to make tests fail.
Thanks for pointing out the related PR.
Similar concerns as to that one: we need some motivating benchmarks that clearly show what benefits would be gained here, and we need to see that changes to this won't negatively impact existing benchmarks/users.
Sure. I will try to make one.
we need to see that changes to this won't negatively impact existing benchmarks/users.
May I know how to do this? does not see benchmark data on CI IMHO
flutter-flutter-perf.skia.org and flutter-engine-perf.skia.org has benchmark data
@dnfield Btw I have replied to your review comments (not sure whether you can see that so also reply here)
@dnfield Thanks, may I know some doc, or how to see perf benchmark of a PR? Click "help" gives http://go/perf-user-doc which cannot be opened (google internal only?)
@dnfield Let me find an old screenshot from my past experiment (https://github.com/fzyzcjy/yplusplus/issues/6124#issuecomment-1272830057)
after fix
@dnfield I agree. Will try to figure out some methods.
Ok I will discuss the pull model later maybe on discord
Btw tests are deliberately not passed since I want to have a quick rough review first before working into details
@dnfield I see
We cannot just keep shoving frames at the system compositor, we will overwhelm it and it will start dropping more frames than necessary/run into long GPU stalls. This will be really bad on older hardware, and still be bad on newer hardware.
May I know a bit more details?
By the way, this PR does not shoving too many frames. Indeed, it has one and exactly one rasterization ending (i.e. submit data to OS) in each vsync interval. Thus, it behaves exactly the same as a super-smooth app, which also provide one and exactly one data to system per vsync interval.
I see, will try to work towards the target
What happens when an application decicdes to just repeatedly call this method? How does it know that it's only called it once per vsync?
@chunhtai sure, done
@dnfield I see your point: If repeatedly call this method, and at the same time the whole UI thread pipeline is much faster than 16ms, then we end up running multiple UI thread pipeline inside one vsync interval. In other words, multi window.render per vsync interval. I admit is a waste - but it is the user who is doing the wrong thing ;) Just like, users can put a ton of Opacity and see rasterizer jank, or they can run sync operations on ui thread and observe ui jank, etc. Nobody can stop them from doing the wrong thing and observe bad outcome.
@cbracken done https://discord.com/channels/608014603317936148/608018585025118217/1035696084963577957
Btw, is it polite to ask for test exemption directly when creating this PR or I should wait for a few days?
test-exemption: code refactor with no semantic change
Btw, is it polite to ask for test exemption directly when creating this PR or I should wait for a few days?
If you think the PR should be exempted, I don't think there is an inappropriate grace period to ask for NTE as far as I know.
@chunhtai I see, thank you
auto label is removed for flutter/flutter, pr: 114117, due to - The status or check suite Google testing has failed. Please fix the issues identified (or deflake) before re-applying this label.
Minor code cleanup: remove redundant return
Find this when reading source code. Maybe next time it can be caught by a linter :)
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
test-exemption: code refactor with no semantic change
That said, it would be great to file an issue on the linter to ask for a lint to catch this kind of thing.
auto label is removed for flutter/flutter, pr: 114290, due to - The status or check suite Google testing has failed. Please fix the issues identified (or deflake) before re-applying this label.
That said, it would be great to file an issue on the linter to ask for a lint to catch this kind of thing.
Totally agree, here it is: https://github.com/dart-lang/linter/issues/3804
Tiny fix about outdated message
The message is in _ensureOutputIsNotJsonRpcError, and that method is called both in runSkia and runRasterizer. Thus, it is not reasonable to say "... skia output ..." because it can also be a rasterizer output.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
auto label is removed for flutter/flutter, pr: 114290, due to - The status or check suite Google testing has failed. Please fix the issues identified (or deflake) before re-applying this label.
This seems to be some sort of infrastructure issue. I've overridden the "Google testing" status check and this PR should be ok to land. Googlers, please see b/256753114 for more details.
Incorrect rendering of SnapshotWidget
Close https://github.com/flutter/flutter/issues/114398
Problem description
In short, the SnapshotWidget is rendered differently when enabled vs disabled.
Reproduction:
Details
import 'dart:ui' as ui;
import 'package:flutter/material.dart';
import 'package:flutter/rendering.dart';
import 'package:flutter_test/flutter_test.dart';
void main() {
final binding = TestWidgetsFlutterBinding.ensureInitialized();
testWidgets('SnapshotWidget should have same result when enabled', (tester) async {
binding.window
..physicalSizeTestValue = const Size(10, 10)
..devicePixelRatioTestValue = 1;
addTearDown(() => binding.window
..clearPhysicalSizeTestValue()
..clearDevicePixelRatioTestValue());
final controller = SnapshotController(allowSnapshotting: false);
await tester.pumpWidget(MaterialApp(
debugShowCheckedModeBanner: false,
home: Container(
color: Colors.black,
padding: const EdgeInsets.only(right: 0.6, bottom: 0.6),
child: SnapshotWidget(
controller: controller,
child: Container(
margin: const EdgeInsets.only(right: 0.4, bottom: 0.4),
color: Colors.blue,
),
),
),
));
final imageWhenDisabled = await _captureImage(tester.element(find.byType(MaterialApp)));
controller.allowSnapshotting = true;
await tester.pump();
final imageWhenEnabled = await _captureImage(tester.element(find.byType(MaterialApp)));
await expectLater(imageWhenEnabled, matchesReferenceImage(imageWhenDisabled));
});
}
Future<ui.Image> _captureImage(Element element) {
assert(element.renderObject != null);
RenderObject renderObject = element.renderObject!;
while (!renderObject.isRepaintBoundary) {
renderObject = renderObject.parent! as RenderObject;
}
assert(!renderObject.debugNeedsPaint);
final OffsetLayer layer = renderObject.debugLayer! as OffsetLayer;
return layer.toImage(renderObject.paintBounds);
}
If you dump the images, will see:
imageWhenDisabled
imageWhenEnabled
How PR solves it
It seems that this is caused by integer rounding. For example, suppose our SnapshotWidget (and thus its Layer) is 9.4 pixels, then after toImageSync will get a 10x10 (or 9x9?) ui.Image instead of a 9.4x9.4 image - since image size must be integer. Then, next time we paint this ui.Image, the original code will paint the 10x10 rectangle area into the 9.4x9.4 Canvas, thus causing resizing of the content. The PR will remember the real size is 9.4 instead of 10, so next time when painting the ui.Image we will paint the 9.4x9.4 rectangle area into the 9.4x9.4 canvas, so no resize of content.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Can you update this PR with the latest master to make sure what is actually changed by this?
Done merging (wait for ci though, but the idea is clear)
@jonahwilliams done
Sorry for the trouble, but you'll also have to rebase this to the latest master to make the "ci.yaml validation" check happy.
Done
Refactor usages of physicalSizeTestValue to simplify code and improve DX
Just a small refactor. Using:
set physicalSizeCurrentTestValue(ui.Size value) {
physicalSizeTestValue = value;
addTearDown(clearPhysicalSizeTestValue);
}
We can make setting physical size test values a bit better in the following two aspects:
- Code is less duplicated. Originally need to specify set value + clear value, now only need one call
- Writing new tests are less error-prone, especially for new learners of Flutter, thus increasing DX. When I firstly learn Flutter, I often wrongly write down set test value calls, without the clear value calls. Then the tests works pretty well when isolated, but you know, it fails weirdly when run sequentially because the later tests have wrong physical size. As a new learner of Flutter (years ago), it took me some time before realizing it is this bug. Thus, it would be great to avoid the possibility of such problem from the beginning.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Fix error when resetting configurations in tear down phase
(This is WIP, since I want to see whether this change will make regression test fail or not) ready for review
Consider this simple example
import 'package:flutter/scheduler.dart';
import 'package:flutter_test/flutter_test.dart';
void main() {
testWidgets('addTearDown should work', (tester) async { // <-- THIS FAILS
timeDilation = 2;
addTearDown(() => timeDilation = 1);
});
testWidgets('directly reset should work', (tester) async { // <-- this is ok
timeDilation = 2;
timeDilation = 1;
});
}
It yields:
Details
/Volumes/MyExternal/ExternalRefCode/flutter/bin/flutter --no-color test --machine --start-paused test/a.dart
Testing started at 09:50 ...
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown running a test:
The timeDilation was changed and not reset by the test.
When the exception was thrown, this was the stack:
#0 SchedulerBinding.debugAssertNoTimeDilation.<anonymous closure> (package:flutter/src/scheduler/binding.dart:662:9)
#1 SchedulerBinding.debugAssertNoTimeDilation (package:flutter/src/scheduler/binding.dart:665:6)
#2 TestWidgetsFlutterBinding._verifyInvariants (package:flutter_test/src/binding.dart:968:12)
#3 AutomatedTestWidgetsFlutterBinding._verifyInvariants (package:flutter_test/src/binding.dart:1433:11)
#4 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:952:7)
<asynchronous suspension>
The test description was:
addTearDown should work
════════════════════════════════════════════════════════════════════════════════════════════════════
Test failed. See exception logs above.
The test description was: addTearDown should work
In other words, we do not allow resetting configurations in addTearDown (or tearDown). Instead, we must do it at the end of the closure.
IMHO resetting things in addTearDown/tearDown is a commonly seen practice. For example, window.physicalSize can be reset in a tear-down function, and even Flutter test code inside the framework does so a lot of times. A quick search also shows that, such as this example, resetting in tear down holds even for other tests such as Python.
By disallowing so, Flutter devs may have a bit more friction when learning Flutter, since they may firstly write down code that follows common practice, realizing it does not work, and change it.
It is also inconsistent with other parts of the Flutter. As mentioned above, window.physicalSize can be reset in a tear-down function, but things like timeDilation cannot.
The PR can also make code a bit simpler. Originally, whenever writing setup code (e.g. timeDilation=2), we must put tear down code at the end of the function, and wrap with a try-finally. But with the PR, it can be put near the setup code, so it is a bit cleaner for code readers. By the way, this is also a bit like the "defer" keyword in go - something like configure(); defer reset(); other_functions() will let the reset be executed last.
In some cases, this seems to simplify code a lot. For example, in https://github.com/flutter/flutter/pull/113830#discussion_r1005887028, I have to introduce a weird timeDilation reset that seems to have no pairing timeDilation modification (and cause confusion of readers - even code reviewers). With this PR, the reset will be put to the next line of modification, so it is clear.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
ping :) @caseyhillers, since
If you still see failures, feel free to ping a Googler for help. You're welcome to ping me on any PRs (my GitHub is @caseyhillers) and I'll take a look. There's an internal switch to mark it as passing as if rebasing isn't working, it likely indicates a separate PR is causing the failures. https://discord.com/channels/608014603317936148/608018585025118217/1037051780443414628
ping :) @CaseyHillers
You can ignore this failure for now. Google Testing can only run if a Flutter hacker approves your PR. Internally, it says there's no LGTMs, and it marks the status as failed (there's an internal tracking bug for making this status show pending instead of failing)
@CaseyHillers Thanks, get it
FYI @dnfield / @goderbauer as secondary review
Tiny improvement of RouteSettings display
When name is null, originally it prints something like: RouteSettings("null", null). But we know "null" looks like a string instead of a real null. So I change to RouteSettings(null, null) when that is null.
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Tiny code cleanup: remove unnecessary comparisons
The flags themselves are normal variables, so seems no need to check equality before assigning.
This PR is so small that I dare not ask for a test exemption or a review :P (Anyway I will continue working on flutter_smooth and those big PRs in a few days when I am not that busy)
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Ensure methods like dispose are not async by accident
Tiny PR, just ensure nobody will accidentially write sth like Future<void> didChangeDependencies() or Future<void> dispose
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Clarify what AnimatedBuilder really speeds up when using child
When I was reading AnimatedBuilder doc (when not that familiar w/ Flutter), I wrongly have the following conclusion: Suppose I have the following, and MyWidget constructs a huge tree:
return AnimatedBuilder(
builder: () => AnimatedOpacity(child: Padding(padding: ..., child: MyWidget())),
);
vs
return AnimatedBuilder(
builder: (child) => AnimatedOpacity(child: child),
child: Padding(padding: ..., child: MyWidget()),
);
Then, reading the doc, I think this will be very little performance boost when using this child. It is because,
If your builder function contains a subtree that does not depend on the animation, it's more efficient to build that subtree once instead of rebuilding it on every animation tick.
So I thought: Yes, I do have a subtree (Padding + MyWidget) that does not change, and this trick can make them built once instead of many times. But it is very fast, because it just creates two objects! (Here, I understood the "a subtree" in the doc as the Padding+MyWidget two objects, while in reality we know it should mean the whole subtree that MyWidget and its descedent widgets build.)
Thus, I add a few lines to clarify this. Hope future flutter new learners do not get confused as me!
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
Secondly, I think having two similarly named setters, one that cleans up after itself and one that doesn't, is still pretty unintuitive for developers. There is still a lot of potential for making a mistake, say by accidentally setting physicalSizeTestValue instead of physicalSizeCurrentTestValue and expecting it to automatically do clearPhysicalSizeTestValue.
What about renaming it to: physicalSizeTestValue vs physicalSizeTestValueAutoClear? Then nobody can use it by mistake since the latter says "auto clear"
ThemeData does not respect ColorScheme for TabBar, CheckBox, etc
Close https://github.com/flutter/flutter/issues/114601
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
- I signed the CLA.
- I listed at least one issue that this PR fixes in the description above.
- I updated/added relevant documentation (doc comments with
///). - I added new tests to check the change I am making, or this PR is test-exempt.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
@dnfield
(content of this comment is moved to: https://github.com/flutter/engine/pull/37341)
(content is also moved to #37341)
Details
@dnfield and we need to see that changes to this won't negatively impact existing benchmarks/users
Could you please give some hints? I guess I have no permission to execute skia perf as it seems only run on master or other flutter branches. Thus, maybe I can do nothing except for creating a PR and see skia perf after it is merged...
Fix jank and large-jumping frame by controlling rasterizer ending time (V2)
Fix janks caused by await vsync in classical Flutter (V2)
Previous: https://github.com/flutter/engine/pull/36911
(Since the previous one is already closed and I have no permission to reopen, I guess I should create a new issue such that it can be put in the review queue)
As @dnfield points out in https://github.com/flutter/engine/pull/36911#issuecomment-1294089965:
we need some motivating benchmarks that clearly show what benefits would be gained here, and we need to see that changes to this won't negatively impact existing benchmarks/users.
Thus, reproduction is shown in the section below. As for existing benchmarks, could you please give some hints? I guess I have no permission to execute skia perf as it seems only run on master or other flutter branches. Thus, maybe I can do nothing except for creating a PR and see skia perf after it is merged...
Reproduction
Setup and analysis
In the benchmark, I deliberately make a dead loop to mimic the case when ui thread runs for smaller than but very close to 16.67ms. This is quite hacky and may not work on your device, so please modify the hardcoded time in the code if you cannot reproduce it. But hopefully this is ok as a "motivating" benchmark you mentioned :)
I also realize this seems hard to reproduce if the await vsync is called per 16ms. Instead, when I make ui thread deliberately janky (e.g. ~100ms), so that await vsync is not called for ~100ms, this is reproduced. Not sure why, but anyway regardless of why this happens, the PR can solve it.
In the screenshots below, I add red vertical lines manually by mimicking the VSYNC interval (this is the correct interval that I have fixed previously, so can use it). As we can see, the AwaitVsync is called before the vsync (which is imaginary and not shown in figure), but we miss it for a whole 16.67ms.
Data
Code: https://github.com/fzyzcjy/flutter/tree/feat/await-vsync-directly-call + standard engine build (because this is reproduction not bugfix)
Run it: /path/to/flutter drive --profile -t test_driver/run_app.dart --driver test_driver/near_full_tim_perf_test.dart
Sample result timeline json: near_full_time_perf.timeline.json.zip
Sample screenshot:
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
...I would suggest filing a specific bug
Hi @iskakaushik may I know what previous work you have done? Thanks
So... What should I do now? I have already replied to all questions and it has been more than a week :)
@dnfield What about changing like this:
Remove (3N-1) jank and big-jump when N rasterization misses deadline (V2)
This PR is just a rough sketch. If it looks OK, I will polish (e.g. add tests, refine code etc)
Previous: https://github.com/flutter/engine/pull/36912
Background: The original proposal completely removes back-pressure. Thus, suppose a rasterzation takes 1000ms, then we will be running ui thread unnecessarily for 60 times.
Proposed new change: Maybe we should conditionally remove back-pressure. More specifically, when we see PipelineFull, we only ignore it and continue running ui thread under the following condition: The current ongoing rasterization has not been running for a long time. This condition holds for the figure in my previous comment, so the main scenario that this PR wants to solve is happy. On the other hand, for the "1000ms-long-rasterization" case, we will only waste one single ui pipeline computation, instead of 60 computations, so the waste does not look like a lot. I agree one waste is indeed waste, but it is the cost to pay (user battery consumes a tiny bit more) if we want to make the UI that user feels completely smooth at 60FPS (which is one of the goals of Flutter IMHO).
Remark: Code comments also describe this as well.
We don't want to expose this kind of internal of the VM to users.
Hmm NotifyIdle seems very generic - there seems no VM internals, but only "hey I am idle for a period of time". It is not something like NotifyDartToDoYoungGC which is internals :) But instead, it is a well-defined interface so dev can tell flutter it is free.
Perhaps it would be more appropriate to expose a different API around whether an animation is occurring currently, although I think @iskakaushik has already done some work on that.
Sorry but I do not quite get it. In flutter_smooth (detailed design: https://cjycode.com/flutter_smooth/design/), a janky frame can last for a long time (for convenience of description, suppose 100ms). Then, during this whole period, the Dart in UI thread will be busy, so it is impossible to execute any C++ code unless Dart code explicitly do so. With this PR, flutter_smooth will explictly call notifyIdle with an explicit deadline of ~14ms (much longer than classical Flutter). This is a bit like the pull vs push model problem.
Same problems as the other NotifyIdle PR - these are details we don't want to expose to users.
Same confusion as in https://github.com/flutter/engine/pull/36797. Still think "dev tells system it can be idle for a period of time" is not something internal but a reasonable thing.
NotifyIdle might not do anything, or it might do a lot more than the user expects.
Totally agree - I have checked Dart NotifyIdle source code and saw it uses something like history data to guess how long GC will happen now, but it can run for longer or shorter time.
As for this specific case of GC: If it does not do anything, there is surely no problem. If it do a lot more, it is still no problem. This is because, a much longer execution means that, even if we do not NotifyIdle, GC will happen somewhere in the future with at least this time duration (because garbage only accumulates).
It is also like what we do with sleep. Dart (and all lang) have a sleep function and can specify how long to sleep. But as you have pointed out earlier, the thread may not be woken up at that time because of OS schedule. In other words, sleep may sleep a lot more than the user expects. Or, even consider this: Each and every line of Dart code (indeed Java/... as well), will have the possibility that, it takes much much longer time to finish! For example, needs 10ms to finish a single i++. This is just fact, not something that I make up - because we have the stop-the-world GC. When STW happens, the code just stuck. Furthermore, we know operating systems which Flutter targets schedule threads with preemption. Thus, it is totally possible that any line of code executes much longer than the user expects even if using C/C++/Rust/etc. So, shall we ban sleep from users if the same logic holds, or ban any language with GC, or ban any operating system that is not real-time ;) (Surely I am joking!)
In all honesty, it would be nice to get rid of it entirely since it is hard to reason about and seems to come out wrong very frequently. If users can change this arbitrarily it will make it much harder to reason about what's going on in an application.
Sorry but do not get it.
May I know a bit more details why "wrong very frequently"? IMHO it just "notifies it is idle" and the only way to be wrong is the user forgets it can run for a much longer time - then my discussions about sleep and STW GC holds.
And I also wonder why "much harder to reason about what's going on in an application"? IMHO it is shown in the timeline, so it is quite clear what is going on. And this is indeed a Dart VM specific thing, so it is even not that related to Flutter framework and engine.
https://github.com/flutter/engine/pull/36921#issuecomment-1294116970
Same comment as on the linked PR applies.
I have made a v2 of that PR: https://github.com/flutter/engine/pull/37341
In particular, it's really hard to reason about how to use this parameter correctly. A lot of the work you're doing seems to be geared towards ignoring/overriding the vsync that the system gives us instead of cooperating with it more.
I agree, it is overriding the system vsync. However, this is used for flutter_smooth, which really has to override some things in the engine...
Does it help if we mark it as @experimental? Then we can freely change it in the future without worrying potential users excluding flutter_smooth.
P.S. Why this is needed: https://cjycode.com/flutter_smooth/design/infra/brake/
In particular, vsync is an important source of backpressure that we must respect. We cannot just keep shoving frames at the system compositor, we will overwhelm it and it will start dropping more frames than necessary/run into long GPU stalls. This will be really bad on older hardware, and still be bad on newer hardware.
Replied above - IMHO seems not a problem?
allowing the developer to specify the vsync time is a big footgun, the developer is very likely to get this wrong and overwhelm the system.
If this API is for normal dev then I totally agree. So same as in the other reply - shall we mark it as "not suitable for normal dev" and mainly for flutter_smooth (and others who knows deeply about flutter)?
The direction of this information has to go in the opposite direction: we need to improve or make it easier for developers to understand the target vsync time and how much time they have left to do work rather than letting developers tell the system what they think vsync should be.
That's perfect for the standard flutter, but IMHO may be impossible for flutter_smooth (e.g. https://cjycode.com/flutter_smooth/design/infra/preempt/idea). In flutter_smooth, there are just tons of long janky frame (e.g. say one frame takes 100ms), and flutter_smooth trigger a lot of extra window.render.
Related: https://github.com/flutter/engine/pull/36438 - one onBeginFrame with multiple window.render.
It's not clear to me how this is any different from observing the Duration you get in onBeginFrame.
In flutter_smooth, one janky frame can be (e.g.) 100ms, and we need to submit window.render per vsync interval. When is the next vsync interval? This question is answered by this PR.
In other words, in the 100ms janky frame, onBeginFrame only provides first vsync target time, while we need to know the rest.
As I'm mentioning in some other PRs, let's discuss further on discord.
Sure, will discuss when you have time (guess you are on weekends so reply here firstly)
I'm still not quite clear on why this is a solution that should live in the framework. Tickers have no means of interrupting work more often. Tickers have no special knowledge of when vsync happens. This seems to introduce a number of problems that will be difficult for developers to reason about. I think this could be solved in a package instead.
Totally agree. However, I cannot find out a method to let it live in a package while making flutter_smooth implementable. flutter_smooth need to support AnimationController and everything built on it (CircularProgressIndicator, SlideTransition, ...), so it seems impossible to create a 3rd party package. Otherwise, at least users cannot use any builtin widget such as CircularProgressIndicator and higher-level widgets, as long as a widget directly or indirectly depends on AnimationController/TickerProviderMixin. Users also cannot use most of the other 3rd party packages, because they use Flutter's AnimationController/TickerProviderMixin as well.
I'm going to close this for now but let's discuss more on discord if you like.
(Firstly reply here before discussing on Discord since I guess you are on weekends)
Part of the work to make this benchmark would be coming up with an at least somewhat realistic case where UI thread work goes to just over vsync budget. Yes, you can construct a benchmark that does exactly that, but what's an example of a real application that actually behaves that way on a real device? IME, it's more common to see an application that far exceeds budget on some frames and then is under budget on many other frames, rather than one that is consistently just slightly over budget. In fact, if you had one that was consistently over budget, you should instead look to optimize your consistent workload to be less - it's an application problem rather than an engine one. But right now, we don't have good ways to fix the problem where some frames over budget despite the fact that you're not doing anything all that unreasonble - for example, oyu're showing a screen full of text that can't all do its initial layout within frame budget on your target device.
You should be able to run the devicelab benchmarks locally and do experiments before and after -see https://github.com/flutter/flutter/blob/master/dev/devicelab/README.md
I'm closing this PR for the same reasons as the previous PR right now - it's not in a state that is ready for review, and much of this content might be more appropriate for an issue right now.
No, conditionally removing back-pressure is still bad: it will cause other problems for sure, since we're conditionally letting the CPU get ahead of the GPU. It'll just be harder to figure out when that's happening and fix it.
@dnfield
Yes, you can construct a benchmark that does exactly that, but what's an example of a real application that actually behaves that way on a real device? IME, it's more common to see an application that far exceeds budget on some frames and then is under budget on many other frames, rather than one that is consistently just slightly over budget.
IMHO it is more like a math/statistics problem than a programming problem. Using my observation from previous issue (this problem happens as long as 2ms before deadline), and a vsync interval is 16.67ms. Then, for a very rough estimation, suppose the real time used is uniform from shortest time to longest time possible - this is very rough, but it somehow makes sense because an app targets from highest-end to lowest-end devices. Then, we have 2/16.67 = 12% probability. Surely that does not sound huge, but it does happen and affect user feeling. Indeed, my flutter_smooth's goal is 60FPS (on 60FPS machine) and this single problem drops 7.2FPS if using the above very rough analysis - and in reality I roughly see more drops.
So, answering "it's more common to see..." - surely yes, but Flutter is a high-performance framework and wants to be 60FPS, instead of a framework that allows jank from time to time ;)
In fact, if you had one that was consistently over budget, you should instead look to optimize your consistent workload to be less - it's an application problem rather than an engine one
By the way, this bug is not only about "over budget" - if ui thread runs over 16.67ms (on 60hz machine) one can never get 60FPS (but can still be 60FPS with flutter_smooth - that's another story). In addition, a frame that is computed quicker than 16.67ms in one phone may be slower than 16.67ms on another lower end phone.
But right now, we don't have good ways to fix the problem where some frames over budget despite the fact that you're not doing anything all that unreasonble - for example, oyu're showing a screen full of text that can't all do its initial layout within frame budget on your target device.
Indeed we have - flutter_smooth :) That example is indeed the first example that I have solved when prototyping (btw thanks for providing that example as a canonical test base!). (As long as all PRs are merged after discussions and modifications)
You should be able to run the devicelab benchmarks locally and do experiments before and after -see flutter/flutter@master/dev/devicelab/README.md I'm closing this PR for the same reasons as the previous PR right now - it's not in a state that is ready for review, and much of this content might be more appropriate for an issue right now.
I see, thanks (originally I used macrobenchmark folder directly)
@dnfield
it will cause other problems for sure, since we're conditionally letting the CPU get ahead of the GPU. It'll just be harder to figure out when that's happening and fix it.
May I know a bit more about "other problems"? Because without knowing more, I have no idea what they are thus how to solve it.
I like the goal of this (easier tests, less likely to make mistakes), but I have a couple of concerns with this as implemented:
- Does this hide a problem, rather than addressing it?
- It's pretty common that
addTearDownandtearDowncalls are missed as you pointed out in your description. This PR makes it easier to not miss those calls for a few specific cases, but it doesn't really address why those calls are missed or drive users to write better tests in other cases.
- It's pretty common that
- Does adding these methods here set a standard that the testing framework will be expected to stick to elsewhere?
- How do we determine which properties or fields should/shouldn't have versions that auto clear?
- How should adding those auto clear versions be prioritized?
- If/when those versions break, should it be considered a breaking change and should the breaking change process be followed for just the auto clear version, or for auto clear and standard versions?
- Will this cause unintuitive behavior in other use cases?
- Specifically I'm thinking of someone trying to use this for setting up a
groupof tests. Since this makes use ofaddTearDown, it'll only behave as expected for individual tests. If someone tries to use the "AutoClear" version in a group, it'll throw an exception. - In combination with number 2, would any "AutoClear" versions also need an "AutoClearGroup" version?
- Specifically I'm thinking of someone trying to use this for setting up a
Personally, I'd lean towards pushing the convenience method part to a separate package where the expectations for support, maintenance, and coverage won't be as high. It'd also allow for more community contributions for different conveniences and extensions to be added without needing to go through as much process as this repo requires. As for the fixes to our own tests, it's a great catch and should be included, but IMO without using the convenience method.
@Renzo-Olivares @justinmc done :)
@pdblasi-google
Does this hide a problem, rather than addressing it? It's pretty common that addTearDown and tearDown calls are missed as you pointed out in your description. This PR makes it easier to not miss those calls for a few specific cases, but it doesn't really address why those calls are missed or drive users to write better tests in other cases.
So what do you think is the reason "why those calls are missed"? I indeed am not sure - maybe because they just forget?
Does adding these methods here set a standard that the testing framework will be expected to stick to elsewhere? How do we determine which properties or fields should/shouldn't have versions that auto clear?
For fields that have a set somethingTestValue and clearSomethingTestValue I guess?
How should adding those auto clear versions be prioritized?
IMHO this is quite trivial to implement (if you like I can submit more)
If/when those versions break, should it be considered a breaking change and should the breaking change process be followed for just the auto clear version, or for auto clear and standard versions?
Sorry, not sure about the question
Will this cause unintuitive behavior in other use cases? Specifically I'm thinking of someone trying to use this for setting up a group of tests. Since this makes use of addTearDown, it'll only behave as expected for individual tests. If someone tries to use the "AutoClear" version in a group, it'll throw an exception. In combination with number 2, would any "AutoClear" versions also need an "AutoClearGroup" version?
Luckily, not a problem! If you like I can put the following code which automatically uses addTearDown vs tearDown and is used internally in my app:
void autoSetUpTearDownOrAddTearDownSync(
void Function() setUpBody,
void Function() tearDownBody,
) {
// ref: test_api :: test_structure.dart :: addTearDown
// `Invoker.current==null` will lead to a message `addTearDown() may only be called within a test.`
// so use this to determine whether in test
if (Invoker.current == null) {
setUp(setUpBody);
tearDown(tearDownBody);
} else {
setUpBody();
addTearDown(tearDownBody);
}
}
@fzyzcjy
Personally, I think those calls are missed because they aren't made obvious to users through documentation and examples. As such, I'm not sure hiding those calls behind convenience methods promotes better usage of those methods.
Adding the convenience methods also increases our API surface, which increases our maintenance burden, and sets up expectations for other parts of the testing framework that don't actually solve the problem of those methods being missed, but instead makes it even less likely that users will run into an example using them correctly.
Long story short, I don't think the convenience methods are worth the long term cost of introducing that pattern to the framework. Hence suggesting creating a package for them, where the maintenance costs and coverage expectations are going to be lower.
It's also worth noting that there's work at the moment to move from a single window to multiple views (to support desktop better), which may be a great opportunity to improve on these particular fields' APIs from the ground up. I believe @goderbauer is driving that initiative, though he's out until next Tuesday.
@pdblasi-google
Personally, I think those calls are missed because they aren't made obvious to users through documentation and examples.
Not sure how other people do, but I am not this case personally. I have this simply because I forgot to call the clear :/ I do vaguely remember the clear, and if you test me in a questionaire I will remember it, but when programming I just set the value - doing the minimal necessary work - while forgetting the clear.
Hence suggesting creating a package for them
Then what should we do for the flutter framework itself? Or shall we mark them @internal so flutter framework can use it?
It's also worth noting that there's work at the moment to move from a single window to multiple views (to support desktop better), which may be a great opportunity to improve on these particular fields' APIs from the ground up. I believe @goderbauer is driving that initiative, though he's out until next Tuesday.
Looks interesting!
If the convenience methods are moved to a package, I would suggest just not using them in the flutter repo and updating the tests here to use the addTearDown methods where appropriate to clear the values. That way, those tests can serve as an example of how to use the standard addTearDown pattern.
Then the tests are not DRY ;)
I mean, if you want to get the tests really DRY, then the setup/teardown would be moved out to the group level and done either once for the entire group, or set up to happen before and after each test with the group level methods.
Since we're working on a public API though, we don't necessarily want to be DRY on everything. We have to balance trying to keep our internal stuff as clean as we can with not making the API too specific to our internal use cases.
then the setup/teardown would be moved out to the group level and done either once for the entire group
Well, those tests are not in the same file, so IMHO they cannot be solved by group.
Since we're working on a public API though, we don't necessarily want to be DRY on everything. We have to balance trying to keep our internal stuff as clean as we can with not making the API too specific to our internal use cases.
I see. Btw, it is not "too specific to our internal use cases" as it is used everyday by external users
Yes! I think there are a number of things (sorry, I've been in meetings)
Changed the channel name.
So there are a few things that I think are needed: we need to make sure we don't make it "easy" for developers to ignore backpressure from the GPU
We need to make sure that we're not relying on the OS to schedule us exactly when we'd like if we choose to deschedule, and that we're not spending significant amounts of time spinning to avoid descheduling
It's okay
Definitely
Btw I have replied to your questions in most PRs previously
(just a reminder in case github does not send notification)
we don't make it "easy" for developers to ignore backpressure from the GPU So continuing from https://github.com/flutter/engine/pull/37343 - may I know a bit more about why conditionally removing back-pressure is bad?
We need to make sure that we're not relying on the OS to schedule us exactly when we'd like if we choose to deschedule, and that we're not spending significant amounts of time spinning to avoid descheduling Hope my thoughts here help a little bit: https://github.com/flutter/engine/pull/37341#issuecomment-1304649243
The purpose of GPU backpressure is to help make sure we don't end up outrunning the GPU on the CPU, which will lead to jank because the GPU can't give us buffers fast enough. We've had issues with this for example even when we tried to use extensions on Android wher it would tell the GPU to drop all but the latest buffer submitted per vsync (led to jittering sometimes which looked worse/as bad as jank).
we don't end up outrunning the GPU on the CPU, which will lead to jank because the GPU can't give us buffers fast enough. A bit confused... Let's say GPU is super janky and takes 100ms + CPU outrunning it and compute 6 scenes (though my proposal will not be this bad). Then, IMHO the main problem is CPU waste battery, because the scene is thrown away. However, seems that the jank will be exactly the same, no matter whether CPU computes 6 scenes or 0 scenes, because the bottleneck is GPU?
The problem is more like: the GPU has 3 or 4 buffers to work with. If you keep asking it to give you buffers to fill when it's busy with them, you're taking time from it doing the work it needs to do to show pixels on the screen, and because you're distracting it from the work it needs to be doing now you're making everything worse for subsequent frames (until you finally stop and let it finish its work queue).
Ah interesting insight!
So you could write a program that just continually asks the GPU to hand you a buffer to draw into, but you'll overwhelm the GPU and won't be able to draw as many frames. The solution is to listen to vsync and do your best to only ask for/submit a buffer once per vsync.
asks the GPU to hand you a buffer to draw into May I know where the code in flutter does this? Is it happened during the rasterization, or happen when ui thread creates a scene tree? (I guess former?)
If the former, then imho, whether ui thread is computing extra scenes is not a problem.
because ui thread does not consume GPU buffers, and also do not disturb rasterizer from doing anything fancy
It happens implicitly when the raster task runner asks for a surface
And we tell the rasterizer to do that when the animator is done getting a frame from the framework
when the animator is done getting a frame from the framework so, a
window.render?
i.e. Animator::Render
yes
Then imho it is no problem, at least for https://github.com/flutter/engine/pull/37343
because imho we just do extra onBeginFrame, not extra window.render
hmm wait a bit
my reasoning is: in this pr, what I did is, BeginFrame will run ui thread pipeline, even if it sees rasterizer queue is full. when ui thread finishes the pipeline, it calls window.render thus Animator::Render. then, if rasterizer queue is still full, it will drop the Scene (ensured by pipeline.h implementation); if not full, it will enqueue the Scene. Thus, IMHO from rasterizer's eyes it is not different.
once you call onBeginFrame, the framework will eventually call window.render right?
sure, but that window.render may be useless
that will cause a waste, but no harm to rasterizer
a waste because the work ui thread has done (the Scene) is not used
(if you are worried about this waste I can explain more - shortly speaking it may be worthwhile; but anyway we are discussing rasterizer and gpu now so I do not want to distract)
If we have a benchmark that shows improvement from this it's probably worth exploring. Right now I'm a bit skewed towards saying this will be bad because it will consume more CPU/power, and on lower end devices that will hurt a lot
The other thing to consider with that is what happens when you succeed: does the new frame that gets added to the pipeline look like it "fits" well with the previous and next frames, or does it get rendered in an unexpected vsync and end up looking slightly off/jittery
will be bad because it will consume more CPU/power, and on lower end devices that will hurt a lot That happens only if it is really wasted. However, looking at figure in https://github.com/flutter/engine/pull/36912 - which is the main case the proposal is to solve - ui thread work is not wasted at all
If we have a benchmark that shows improvement from this it's probably worth exploring Is it OK to be a benchmark like https://github.com/flutter/engine/pull/37341, i.e. a one with tons of deliberately blank loops?
Sure, but this is not the typical problem that I see when working on real applications on lower end devices 🙂
Btw what is the typical problems?
The more typical problem is a frame that just blows way through frame budget (e.g. 40-50ms) to build.
Indeed I aim at 60FPS (not 55FPS etc) so each corner case I face needs to be solved 🙂
That's just https://github.com/fzyzcjy/flutter_smooth solves! 😄
So this benchmark: https://github.com/flutter/flutter/blob/master/dev/benchmarks/macrobenchmarks/lib/src/list_text_layout.dart
Indeed, I do not realize PR 36912 is even a problem, before I have finished the main part of flutter_smooth
In other words: flutter_smooth solves the main type of problem, then the problems that are less likely to happen now become main problems that makes me about 50-55FPS but not ~60FPS
That draws something like a contacts list. On a low end Android device, building that scene can easily take 27-30ms, mostly in text layout
IIRC my earliest prototype of flutter smooth have already solved that 🙂
(at that time it is not called flutter_smooth)
I've yet to see a full, runnable solution that solves it though. My recollection (which may be wrong) is that the proposals tended to have a lot of comments like "fix this later"
Ah, sure, I should make a full runnable app now (but with custom framework and engine code)
But I already have sth similar: https://github.com/fzyzcjy/flutter_smooth/blob/bf4f69884b2b05147875d7c6ee1b28b52e5c34eb/packages/smooth/example/lib/example_page_transition/sub_page.dart#L68
Smooth animation (material page route) when entering a page, and that page is really heavy to build (using ComplexWidget to simulate, can be seen in code)
is that sound interesting to you? or I need to make an exact reproduction again for list_text_layout
I am not convinced that it is actually the same. And even if it is the same problem, it would need to be adopted pretty widely before we could land changes to the engine that only make sense if you're using it.
I'd like to see what it looks like to use your solution to fix the list_text_layout to be faster
(even if it requires a custom engine or framework build to achieve)
(forget to reply this) It looks well IMHO. But if you are having a rasterization that is longer than 16ms, then it is inevitable to have one jank feeling. see the main figure of PR for more details. And, as for flutter_smooth (note: the PR also holds for classical flutter), an extra doc page: https://cjycode.com/flutter_smooth/benchmark/analyze/linearity/ (I made up a term called linearity, which is what you mention as jittery I guess)
A bit confused: Do you mean flutter_smooth should be adopted widely before flutter engine can change?
But imho most people (including me) will never use something that is not in flutter stable for production app
But the stars (>700) may show that many are interested in it
So I said that a patch you opened in the engine will fail to help some common sources of jank and actually potentially make them worse on low end devices. If I understand, you're saying those common sources go away with flutter_smooth so that's not a problem, and you're moving on to other sources of jank once you solve those common problems.
as well as reddit popularity https://www.reddit.com/r/FlutterDev/comments/y6um51/be_60fps_smooth_no_matter_how_janky_your_app/
4th popular in this year
Put more concretely: flutter has some very large customers who don't use flutter_smooth. We can't make changes to the engine that will make their applications worse. If flutter_smooth helps those applications, we should try to integrate it into the framework
That's very cool 🙂
and even get to GitHub trending main board twice (I really did not expect that - you know github trending main board contains all languages, so almost impossible for a Flutter/Dart project to be there)
https://github.com/search?q=flutter_smooth&type=code can see some spiders get that trending data - 10.30 and 10.21
imho maybe not "make them worse".
If I understand, you're saying those common sources go away with flutter_smooth so that's not a problem, and you're moving on to other sources of jank once you solve those common problems. Yes
definitely, it is also very interesting to be integrated to flutter
so... what makes it worse if having the PR on low end device?
except that I do not provide a benchmark yet (since you requested just a few minutes ago 🙂 )
the rasterizer problem seems not to be a problem as said above
or I understand wrongly
Doing more work will make things worse
I see
If you're on a slow device and it's taking a lot of time to build every frame
taking a lot of time to build every frame yes, that happens for users without flutter_smooth
There are many more users who don't have flutter_smooth than do 🙂 FWIW, internally at google applications are built relatively close to head of master
(usually about a week or so behind it)
you guys are so brave 🙂
even not built with beta?
We'll get yelled at by all those customers when their benchmarks go south, and telling them "oh well if you use flutter_smooth it'll be fixed" won't be good enough
definitely
It's a little more complicated than what I'm saying but no, for the most part they're newer than beta
so I guess the main priority is like this:
- maybe we can firstly PR for a partial-flutter_smooth, i.e. a version for only maybe 50FPS not 60FPS (because all those edge cases are not solved). then the PR merged into flutter, and people can try it. then, if many people find it quite helpful etc, maybe we can consider the rest prs or integrate into framework?
- or alternatively, shall we just consider integrating flutter_smooth to framework now? so many PRs that solely wants to make flutter expose some low-level things are never a problem now
For future readers: This is discussed in https://discord.com/channels/608014603317936148/1039667632342835250
yeah, making incremental improvements is a good way to go.
And having some compelling benchmarks would help ideally without requiring major surgery on the application side to achieve)
totally agree
Btw flutter_smooth does have (hopefully) compelling bench: (1) 3-second video in https://github.com/fzyzcjy/flutter_smooth (2) a chapter in https://cjycode.com/flutter_smooth/benchmark/
so, one of the main blocker before partial-flutter_smooth can land, maybe: https://github.com/flutter/engine/pull/36438
also, pull-based model (e.g. https://github.com/flutter/engine/pull/36438) - otherwise ListView scrolling cannot be smooth because we have no way to read touch events, though your "list of text" can be
I think ideally we have a solution that doesn't require calling render more than once per onBeginFrame
I have tried to think about it, but that seems fundamental for flutter_smooth
whole design details: https://cjycode.com/flutter_smooth/design/
so we may not have ideal things 😐 but considering the benefits of flutter_smooth, I hope it is worthwhile
(benefits, including future benefits if it is merged into flutter)
I have to agree with @pdblasi-google, I'm not certain that this is better than the existing API, so I don't think we should make this change. Instead, here's what I think we can do about this problem:
- Clearing should happen immediately after setting. This makes it obvious to future readers of the test that these values need to be cleared, and it makes it less likely the test will end without the values being cleared. I think this could be done in this PR or in a new PR.
Old way:
tester.binding.window.devicePixelRatioTestValue = 1.2;
tester.binding.window.physicalSizeTestValue = const Size(250, 300);
... // Rest of test
tester.binding.window.clearPhysicalSizeTestValue();
tester.binding.window.clearDevicePixelRatioTestValue();
Should be changed to:
tester.binding.window.devicePixelRatioTestValue = 1.2;
addTearDown(clearDevicePixelRatioTestValue);
tester.binding.window.physicalSizeTestValue = const Size(250, 300);
addTearDown(clearPhysicalSizeTestValue);
... // Rest of test
devicePixelRatioTestValueAutoClearandphysicalSizeTestValueAutoClearcould be released as a Pub package if desired.Like @pdblasi-google said, ideas about autoclear and other patterns that avoid this problem should be directed at @goderbauer's work to move from TestWindow to TestView, since we have the opportunity to start fresh here and change the API. He has a design doc, which it doesn't specifically mention this about testing so it might be worth bringing it up in a comment there.
@christopherfujino Sure, will do this soon
Hmm I see. Then maybe I will refactor like what you said.
P.S Did a comment there just now.
All right, I will try to squeeze some time to make a benchmark later :)
@Piinks Sure, will do it later. Btw may I get some hints about how to add test for this feature? Not very familiar with registerBoolServiceExtension etc
auto label is removed for flutter/flutter, pr: 114400, due to - The status or check suite Mac tool_tests_commands has failed. Please fix the issues identified (or deflake) before re-applying this label.
@fzyzcjy could you update to ToT so we can land this?
@jonahwilliams sure, done (waiting for ci now)
I am going to close this as it doesn't add any new information to the doc.
(This is WIP, since I want to see whether this change will make regression test fail or not)
Do you still have further plans for this? Or did it achieve its goal and can be closed?
Oh it is ready for review
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
@fzyzcjy Can you rebase this to trigger the Google testing check again? Looks like it got stuck...
@goderbauer done
This is internal state that nobody should depend on. If you have a use case for this, I suggest filing an issue that describes what you want to achieve instead of describing a solution.
Ok, I will explain why this is needed in a separate issue later.
I don't think we should be exposing random implementation details like that. I would suggest filing an issue discussing the problem you're trying to solve without tying it to a particular solution. We can then see if this fixes that particular problem.
You can use systracing instead of timeline by doing --trace-systrace. That is less expensive than timeline based tracing.
Unfortunately, I'm having trouble from the images telling where the extra time is being spent. I'm also confused why the traces are looking at pointer data packet events when this bug is about frame timings.
I think we should try to look at ways to make the frame timings measurements cheaper on representative hardware rather than disabling them. Timeline adds more overhead on every frame. Maybe you could file an issue with some of your findings?
Have no bandwidth now, will reply to review soon, thanks :)
I see, same as before, I will file soon when having time. Thanks!
Unfortunately, I'm having trouble from the images telling where the extra time is being spent. I'm also confused why the traces are looking at pointer data packet events when this bug is about frame timings.
Ah sorry! I must paste the wrong image here - at the time I wrote the reply I may be thinking about my another PR about pointer events.
I think we should try to look at ways to make the frame timings measurements cheaper on representative hardware rather than disabling them. Timeline adds more overhead on every frame. Maybe you could file an issue with some of your findings?
Sure. I will do that soon (have no time now so possibly within a few days)
Thanks :) I will modify and update it soon (no time now though)
Followup on moving the invariant verifications into an addTearDown (came to me at about 3AM last night...).
I think if we move the invariant verifications into an addTearDown, we may also need to move the storing of those variables (for checking the end state) that get checked into a setUp call. I think as it stands right now, if someone were to use setUp to change an invariant, it'll have a couple problems:
a) It won't validate that the invariant is reset after the test, since it's actually set before the test
b) If (after this issue is fixed) the invariant is reset via an addTearDown or tearDown, then the verifications will fail because they'll have the changed value saved to check against, since it was changed before the test.
Good point! I will check that later.
auto label is removed for flutter/flutter, pr: 114481, due to - Please get at least one approved review if you are already a member or two member reviews if you are not a member before re-applying this label. Reviewers: If you left a comment approving, please use the "approve" review action instead.
auto label is removed for flutter/flutter, pr: 114481, due to Validations Fail.
Hm, I am not really sure how to test this. Have you asked Hixie on Discord if it qualifies for a no test exemption?
@Piinks Sure, I will ask soon ask sent
As discussed earlier in Discord, maybe I will focus on the main part of flutter_smooth now (quite busy recently), and restart this PR after the critical PRs of flutter_smooth has been merged.
(For future readers of this thread: Recent progress of the critical PRs are not only in flutter repositories and discord, but also somewhere such as https://github.com/fzyzcjy/flutter_smooth/issues/173)
As discussed in Discord with @dnfield previously, I will firstly focus on the PRs that are critical to the main part of flutter_smooth, and deal with these PRs (which improves performance roughly several FPS per PR) later.
(For future readers of this thread: Recent progress of the critical PRs are not only in flutter repositories and discord, but also somewhere such as https://github.com/fzyzcjy/flutter_smooth/issues/173)
test-exempt: code refactor with no semantic change
Reland fix wrong VSYNC event
Reland #36775 Previously reverted by https://github.com/flutter/engine/pull/37589 Close https://github.com/flutter/flutter/issues/115161
Pre-launch Checklist
- I read the Contributor Guide and followed the process outlined there for submitting PRs.
- I read the Tree Hygiene wiki page, which explains my responsibilities.
- I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
- I listed at least one issue that this PR fixes in the description above.
- I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
- I updated/added relevant documentation (doc comments with
///). - I signed the CLA.
- All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel on Discord.
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).
If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?
Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.
Issue created describing this: https://github.com/flutter/flutter/issues/115899
The requested issue is created and described the details: https://github.com/flutter/flutter/issues/115901
Well, tearDown does not seem to work. I guess it is because, our addTearDown(verifyInvariants) are called before this user-added tearDown. Thus, even if user resets value in this tearDown, our verifications run before that so is not aware of it.
However, IMHO this PR is still useful, because even if tearDown is not supported, it adds support for addTearDown, which is very frequently used.
tests:
group('tearDown does not work yet', () {
tearDown(() {
timeDilation = 1;
});
testWidgets('main test inside group', (WidgetTester tester) async {
timeDilation = 2;
});
});
yields
package:flutter/src/scheduler/binding.dart 662:9 SchedulerBinding.debugAssertNoTimeDilation.<fn>
package:flutter/src/scheduler/binding.dart 665:6 SchedulerBinding.debugAssertNoTimeDilation
package:flutter_test/src/binding.dart 971:12 TestWidgetsFlutterBinding._verifyInvariants
package:flutter_test/src/binding.dart 1436:11 AutomatedTestWidgetsFlutterBinding._verifyInvariants
package:flutter_test/src/binding.dart 950:9 TestWidgetsFlutterBinding._runTestBody.<fn>
===== asynchronous gap ===========================
dart:async _CustomZone.registerUnaryCallback
package:flutter_test/src/binding.dart 941:9 TestWidgetsFlutterBinding._runTestBody.<fn>
The timeDilation was changed and not reset by the test.
Hmm CI fails. I will firstly revert now
Correct the unit of file size from "kb" (maybe "kilo bits") to "KB"
The error message when enabling "invert oversized images" tells us how many extra space the image uses. That is quite helpful! However, when I read
...an additional 1234kb...I thought that means kilo bits (since in network programmingkbandkBare definitely different and should be taken with care). However, after reading the source code, I suddenly realized that it is kilo bytes. Thus, this PR is here to ensure that people do not misunderstand the unit.List which issues are fixed by this PR. You must list at least one issue. Fixes https://github.com/flutter/flutter/issues/76032
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy. N/A
Pre-launch Checklist
///).If you need help, consider asking for advice on the #hackers-new channel on Discord.