Thanks for the report, and the caching should ensure the core is compiled only once, and only recompiled when any relevant options change.

Can you enable the options shown at the top of the page and attach the complete verbose output for us to review from a rebuild of the project?

Visual Micro looks at changes in build properties and file times to determine what needs to be rebuilt.

If only project code has changed then libraries and core should not be rebuilt.

If switching between debug and release when a previous core/library compilation exists for that configuration yet no other things have changed, the cached version of cores and libraries should be used.

Switching git branch doesn't give us any idea of what the git branch is changing so it is difficult to comment.

There is a lot of intelligence surrounding the cache of cores, libs and project code. It makes sense a compile of core would happen when switching between release and debug for example, but only the first time that happens. 

We would need the information requested in yellow near the top of the page to compare the build properties when you believe recompile should not have happened.

Yes, there are a few painful toolchains out there. The esp32 is one of the slowest.