One of the applications that I manage is a rather complex TFS 2010 beast of a build workflow that is used across the organisation to provide auditable and golden reproducible builds for the teams in the bank. Basically as per the compliance policies that all financial institutions must adhere to the applications that are produced by the back have a regulatory obligation to adhere to strict policies with regards to their source code and release management process. Most big financial institutions thus have centralized automated build systems that ensure all appropriate audit, governance and compliance regulations are adhered to by all the source code that passes through the build system.
So this application then is effectively a central build system which we have designed as a customization of TFS 2010 build system used by a good number of .NET teams within the firm. One of the components of this application is a heavily customized build workflow that ensures all compliance, governance, auditing activities for the builds.
As is expected we have a library of custom activities written which are used within the build workflow. These activities use all standard Microsoft TFS libraries such as Microsoft.TeamFoundation.Build etc. Everything in the entire application is designed to work with TFS 2010 build system.
Recently we updated one of our test build servers to have both Visual Studio 2010 and Visual Studio 2012 on it. To our extreme frustration we found that suddenly we were unable to compile our application on this build server. Bear in mind we were not upgrading the application to support TFS 2012 build system but were simply trying to compile an application that referenes v10.0.0.0 of Microsoft.TeamFoundation* libraries on a build server that has both Visual Studio 2010 and 2012 installed on it.
The error message we kept getting was
c:Program Files (x86)Microsoft Visual Studio 11.0Common7IDEReferenceAssembliesv2.0Microsoft.TeamFoundation.Client.dll: Assembly ‘Microsoft.TeamFoundation.Client, Version=220.127.116.11, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ uses ‘Microsoft.TeamFoundation.Common, Version=18.104.22.168, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ which has a higher version than referenced assembly ‘Microsoft.TeamFoundation.Common, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’
Almost all the links that I could find on the web were aimed at helping to migrate a TFS 2010 build workflow for TFS 2012 build system. The other category involved simply getting a build workflow to compile on a VS 2012 enabled build system. While the problems were similar they weren’t exactly the same since we were not dealing with a build failure due to compilation of our build WF but because of custom activities library.
Further we were not upgrading our application to support TFS 2010.This was a crucial distinction in our case as we needed to ensure that our application can support build servers that only have v10.00* of various assemblies. We needed to ensure our build chain refers to v10.0.* of TFS assemblies and not the new v11.0* ones.
It simply didn’t make sense at that time. Why when our code projects are told to reference v10.0* of a library would they suddenly start referencing v11.0* of an assembly on the build server? Is there a binding redirect that comes with installing VS 2012? Is there a publisher policy that is put in place by MS? Is my entire application and code jinxed and do I need to draw a pentacle at the back of a cemetry and sacrifice a chicken on full moon night? WTF is going on!
For 2 straight days I spent every waking moment debugging, tracing, hooking, probing, profiling, monitoring MSBuild pipeline, the various target files and any such file that might contain a re-direction policy that could be impacting this. To my dismay I found absolutely nothing.
I even asked this question on SO where Nick suggested removing the explicit versioning information from within the code project. While potentially this could have worked (and in our case it did not) this is not what we wanted to achieve. We wanted to absolutely make sure and certain that the build process is referencing v10.0* of libraries and not the recent versions.
The OMFG! moment came in whene I stopped caring about this problem, moved on and by chance when I was eye-balling an article on the task “ResolveAssemblyReference” on MSDN for another piece of work. I am a bit of an SME on MSBuild and customizing its pipeline so it came as a bit of a shock and embarrassment on how did we miss this simple fact! Here is the article and this is the magic text.
SpecificVersion: Boolean value. If true, then the exact name specified in the Include attribute must match. If false, then any assembly with the same simple name will work. If SpecificVersion is not specified, then the task examines the value in the Include attribute of the item. If the attribute is a simple name, it behaves as if SpecificVersion was false. If the attribute is a strong name, it behaves as if SpecificVersion was true.
When used with a Reference item type, the Include attribute needs to be the full fusion name of the assembly to be resolved. The assembly is only resolved if fusion exactly matches the Include attribute.
When a project targets a .NET Framework version and references an assembly compiled for a higher .NET Framework version, the reference resolves only if it has SpecificVersion set to true.
When a project targets a profile and references an assembly that is not in the profile, the reference resolves only if it has SpecificVersion set to true.
This is it! It is the bloody default behaviour of MSBuild! Once this clicked in my head it was rather easy to confirm. I simply edited my build definition and passed in parameter /v:d to MSBuild call which basically tells MSBuild to produce a rather extensive log of what it is upto.
Once we generated the extended level of logging clearly demonstrated that it was actually MSBuild’s native resolution mechanism that was resolving the reference to a more recent version of DLLs. The solution thus was simple and constituted simply editing our .csproj files and specifying SpecificVersion property to be true for these referenced binaries.
Once we confirmed this, it was very easy to simply edit the C# project files and pass in True for the SpecificVersion tag for the assemblies in question. Needless to say the builds worked like a charm from that point onwards.
It also ensured that the MSBuild pipeline would only ever use the exact version of our referenced assemblies thus alleviating that concern as well!