VS Arduino
Visual Micro >> Ideas, Suggestions and Related Tools >> ESP8266 Arduino rebuilds loads of files that weren't changed
https://www.visualmicro.com/forums/YaBB.pl?num=1565275024

Message started by dok on Aug 8th, 2019 at 2:37pm

Title: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by dok on Aug 8th, 2019 at 2:37pm
Each time, without any code changes whatsover, "Build project" (control-shift-B).
Every other build rebuilds loads of unchanged sources - wastes a lot of developer time, 29s in this case.
https://www.visualmicro.com/forums/YaBB.pl?action=downloadfile;file=Documents.zip ( 35 KB | 2 Downloads )

Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by Visual Micro on Aug 8th, 2019 at 2:48pm
I can't reproduce the problem. Ae you making any changes at all between builds?

Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by Visual Micro on Aug 8th, 2019 at 3:04pm
Please close the ide and open regedit and rename hkey_current_user\software\visual micro.

Then restart the ide add the arduino 1.6/1.8 program location to IDE Locations. Then open your project and try this again.

This will ensure you are using the visual micro default cache settings.

Thanks

Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by dok on Aug 8th, 2019 at 3:13pm
I hope this doesn't lose my license?

Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by dok on Aug 8th, 2019 at 3:20pm
First time, "Rebuild Solution":

Code (]Compiling 'blinkbuttonandweb' for 'LOLIN(WEMOS) D1 R2 & mini'
Creating BIN file "C:\Users\dok\AppData\Local\Temp\VMBuilds\blinkbuttonandweb\esp8266com_d1_mini\Release/blinkbuttonandweb.ino.bin" using "C:\Users\dok\AppData\Local\Temp\VMBuilds\blinkbuttonandweb\esp8266com_d1_mini\Release/blinkbuttonandweb.ino.elf"
Program size: 320,324 bytes (used 31% of a 1,044,464 byte maximum) (56.89 secs)
Minimum Memory Usage: 28568 bytes (35% of a 81920 byte maximum)
[/code):



Second time, no changes, control-shift-B:
[code]Compiling 'blinkbuttonandweb' for 'LOLIN(WEMOS) D1 R2 & mini'
Creating BIN file "C:\Users\dok\AppData\Local\Temp\VMBuilds\blinkbuttonandweb\esp8266com_d1_mini\Release/blinkbuttonandweb.ino.bin" using "C:\Users\dok\AppData\Local\Temp\VMBuilds\blinkbuttonandweb\esp8266com_d1_mini\Release/blinkbuttonandweb.ino.elf"
Program size: 320,324 bytes (used 31% of a 1,044,464 byte maximum) (2.51 secs)
Minimum Memory Usage: 28568 bytes (35% of a 81920 byte maximum)


Third time, no changes, control-shift-B:

Code (]Compiling 'blinkbuttonandweb' for 'LOLIN(WEMOS) D1 R2 & mini'
Creating BIN file "C:\Users\dok\AppData\Local\Temp\VMBuilds\blinkbuttonandweb\esp8266com_d1_mini\Release/blinkbuttonandweb.ino.bin" using "C:\Users\dok\AppData\Local\Temp\VMBuilds\blinkbuttonandweb\esp8266com_d1_mini\Release/blinkbuttonandweb.ino.elf"
Program size: 320,324 bytes (used 31% of a 1,044,464 byte maximum) (33.24 secs)
Minimum Memory Usage: 28568 bytes (35% of a 81920 byte maximum)
[/code):



Fourth time, no changes..., control-shift-B …
[code]Compiling 'blinkbuttonandweb' for 'LOLIN(WEMOS) D1 R2 & mini'
Creating BIN file "C:\Users\dok\AppData\Local\Temp\VMBuilds\blinkbuttonandweb\esp8266com_d1_mini\Release/blinkbuttonandweb.ino.bin" using "C:\Users\dok\AppData\Local\Temp\VMBuilds\blinkbuttonandweb\esp8266com_d1_mini\Release/blinkbuttonandweb.ino.elf"
Program size: 320,324 bytes (used 31% of a 1,044,464 byte maximum) (4.87 secs)
Minimum Memory Usage: 28568 bytes (35% of a 81920 byte maximum)

Fifth time, same procedure:
[code]Compiling 'blinkbuttonandweb' for 'LOLIN(WEMOS) D1 R2 & mini'
Creating BIN file "C:\Users\dok\AppData\Local\Temp\VMBuilds\blinkbuttonandweb\esp8266com_d1_mini\Release/blinkbuttonandweb.ino.bin" using "C:\Users\dok\AppData\Local\Temp\VMBuilds\blinkbuttonandweb\esp8266com_d1_mini\Release/blinkbuttonandweb.ino.elf"
Program size: 320,324 bytes (used 31% of a 1,044,464 byte maximum) (33.66 secs)
Minimum Memory Usage: 28568 bytes (35% of a 81920 byte maximum)
[/code]

Obviously not expected predictable behavior of a properly functioning make.

Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by Visual Micro on Aug 8th, 2019 at 3:20pm
If it did then I could reset it but if you rename you can delete the new "visual micro" that will be created and rename the old one back after the test.


Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by dok on Aug 8th, 2019 at 3:22pm
About the license - thanks, I'm reverting the registry accordingly, as the suggested workaround didn't have any effect.

Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by dok on Aug 8th, 2019 at 3:41pm
Just to be sure, testing other targets, and cleaning up. But:

Arduino Mini Pro - unaffected.
ESP32 - unaffected.
ESP8266 master branch in git client, cleaned up, toolchain updated (cd tools; py get.py) - as observed and reported above.

Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by Visual Micro on Aug 8th, 2019 at 3:43pm
Thanks, so to be clear you are saying the problem happens with you custom tool chain in documents\arduino\hardware (sketchbook\hardware) but does not happen with packages installed using board manager or built in arduino ide (avr)?

Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by dok on Aug 8th, 2019 at 4:43pm
I didn't exactly say that.
There's no custom toolchain involved, this is HEAD of the master branch of the ESP8266 Arduino project.
To make sure it's not just about the latest version - checking out release 2.5.2 in git.

Using Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WebServer\examples\HelloServer\HelloServer.ino - it's the ESP8266 after all, which is all about the WiFi and web server capability.

Rebuild - 63s.
Build - 3s.
Build - 35s.
Build - 4s.

So it's not about the version.

Moving away my git clone in hardware, and d/l-ing via the board manager (really useless for me, but for testing sake).

Opening a copy of the HelloServer example (Micro Explorer/Examples).

So, rescan toolchain, rebuild (65s), build (3s), build (3s), build (3s).

Fantastic. NOT helpful.

Leaving the board manager BSP package in place, but restoring the git master to …/hardware.

Rebuild (63s), build (4s), build (38s), build (5s).

I am attaching the logs of building with the board manager supplied 2.5.2, and those of building with the git version 2.5.2.
https://www.visualmicro.com/forums/YaBB.pl?action=downloadfile;file=Documents_001.zip ( 35 KB | 2 Downloads )

Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by Visual Micro on Aug 8th, 2019 at 4:52pm
Ah okay, so to be clear. If you have a package installed into board manager and another copy of the same (or similar) core from the package installed into sketchbook\hardware then we see the problem?

edit: I think I have anough to experiment. This seems to be an issue with split or partl duplicate toolchain/cores maybe.

Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by dok on Aug 8th, 2019 at 5:00pm
Not precisely.
The problem does not occur when there is no BSP distribution for the ESP8266 in the sketchbook Arduino/hardware, but one is installed via the board manager (sic).
The problem always occurs when there is an ESP8266 BSP distribution in the sketchbook, independent of the presence of one that's installed via the board manager. Haven't tried non-matching versions ;-)

Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by dok on Aug 8th, 2019 at 5:03pm
I hope you'll be able to figure this out without external help from the FOSS project - it would be so much nicer not having to spend 30s on each compile in vain. Thanks!!!!!!!!

Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by Visual Micro on Aug 8th, 2019 at 5:11pm
Thanks, will take a look over coming few days. It will build a bit faster of you switch on "vMicro>Compiler>Parallel Builds". example: All the wifi library files in a single folder will be compiled at same time (errors can be a little more difficult to read).

Title: Re: Rebuild libraries due to custom hook event generating a new dependent(s)
Post by Visual Micro on Aug 10th, 2019 at 12:19pm
It looks like Visual Micro is basically doing the right thing. The confusion appears to be caused by your prebuild event hooks where you geneterate a new version number .h file.

The version number changes during each build and you have linked the version.h into the libraries (maybe via the core).

The libraries use more than simple date time of each .cpp to determine when rebuild it need. It uses the .d depency file generated by previous build to discover all depencies of each source code file, so that it can compare the dates and times of dependants with the current source file (cpp).

Therefore because your version number changes for each build then we would expect the libraries and project code to be rebuilt each time the build happens.

In the next release of Visual Micro there is a new build event hook that fires only when the date time or a core file has changed called "core.prebuild_if_changed". If you move your core.prebuild event hook then you get a new version.h only when you have changed a core source core.

The fact that you see alternate build is probably due to when the hook is run and the fact that you create it in the core temp build folder itself. Maybe it should have rebuilt every single time to cause less confusion.

Does this make sense?

These are the hooks in your custom core.

[code]recipe.hooks.sketch.prebuild.pattern="{runtime.tools.python.path}/python" "{runtime.tools.signing}" --mode header --publickey "{build.source.path}/public.key" --out "{build.path}/core/Updater_Signing.h"
>>> recipe.hooks.core.prebuild.pattern="{runtime.tools.python.path}/python" "{runtime.tools.makecorever}" --build_path "{build.path}" --platform_path "{runtime.platform.path}" --version "unix-{version}"[/code]



Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by dok on Aug 12th, 2019 at 6:05am
Well, if you replace in your message "your" by "the community ESP8266 Arduino core's", then I'm fine :-)
I hope you understand that your statement

"your custom core"

sounds like assigning blame to me in my ears.

[edit: removed part talking about recent change, I forgot that I was testing 2.5.2 before already].

I don't know what the difference between "packaged release" and Git at the the same version commit is - if any.

Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by dok on Aug 12th, 2019 at 6:47am
So, it's all about this one, right:

...\AppData\Local\Temp\VMBuilds\HelloServer\esp8266com_d1_mini\Release\core\core_version.h ?

It seems in esp8266/tools/makecorever.py, in Git builds - as opposed to releases where core_version.h get placed once and for all into cores/esp8266 - the python script generates the same file contents for core_version.h over and over, but obviously updates the file time stamp. And that's consistent with what you found.

So all it might take is some hacking of the python script... yes, confirmed, if I comment out the writing of core_version.h, the build time without changes becomes a consistent 4.5s.

Thanks for looking into this, I guess I'll pick up from here.

Title: Re: ESP8266 Arduino rebuilds loads of files that weren't changed
Post by dok on Aug 12th, 2019 at 7:18am
Fixed by PR https://github.com/esp8266/Arduino/pull/6414

VS Arduino » Powered by YaBB 2.6.12!
YaBB Forum Software © 2000-2024. All Rights Reserved.