Custom Action Utilities Flagged By Windows Defender...

[Superfreak3]

Hi all,

This is really a two-part question.  First, why are some custom actions reported by the actual .exe or .dll name for some actions and MIS*.tmp for others?  Is that governed by the actual Custom Action type or something to that effect?  Or if some are support files and some are pulled from the Binary table, would that do it?

Secondly, and the main question...  What do you do for actions being flagged by Windows Defender?  Besides whitelisting, will signing those executables help?

With the MSI*.tmp actions/executables, will signing the actual .exe/.dll being called and referenced as .tmp in the log help?

We're starting to see more and more issues with Defender lately.

Thanks for any info you can provide.

[Superfreak3]

@skrueger  Let me know if posting elsewhere might yield answers!

[skrueger]

Maybe a Defender forum or Microsoft support?

[Dan_Galender]

What type of Custom Actions are being flagged?   DLL?  EXE?  VBScript?  Managed Code?

[Superfreak3]

One of the two named binaries, the .exe, is Synchronous (Check Exit Code), Deferred Execution, MSI Type Number 1058.

The other named binary is a .dll, which detects the presence of a third-party app.  It runs in both the UI and Execute sequences.

The Custom Actions are comprised of primarily C# Code.

[Superfreak3]

So, to see if this combats the problem successfully, I've gone ahead and started signing my Custom Action binaries.  I've come to an MSI .dll that is written using a WiX template.  On compile a MyFile.dll and MyFile.CA.dll are generated.  In InstallShield, the MyFile.CA.dll is referenced or pointed to in the Custom Action settings.  When I sign the target file using signtool, the MyFile.dll is digitally signed, but the .CA.dll is not.  So I attempted to add another post-build step to sign the .CA.dll, but that errors out...

EXEC : SignTool error : SignedCode::Sign returned error: 0x800700C1
1>
1> Either the file being signed or one of the DLL specified by /j switch is not a valid Win32 application.
1>
1>EXEC : SignTool error : An error occurred while attempting to sign: C:\Install Source\MyApp\Custom Actions\MyAppCheck\MyAppCheck\bin\Release\MyAppCheck.CA.dll

Does anyone have any experience using WiX MSI .dll Custom Actions.  Which file is actually used in the install pacakge, the main .dll or the .CA.dll?

I'm hoping in some way that the CA .dll signals Install shield to include the regular .dll as that is the file I'm able to sign.

@skrueger  @Dan_Galender 

[Locil95]

No idea if you're still having issues with this, but I had the exact same issue as you, when compiling through Visual Studio, and deploying a package to another machine. Defender would block those CA files. Knowing how much frustration I went through trying to figure out a solution to this, I remembered your post and thought I might come back and share.
As you might have guessed, I have solved this issue at our end. I did so by manually adding a rebuild after signing the .dll in a Post-build event. I had to upgrade to WixToolset 4. But other than that, it now works like a charm.

I found the final hint on this page:
https://wixtoolset.org/docs/tools/dtf/

I figured out what happens when Wix builds those CA projects, which sadly doesn't work with signing (not in Visual studio at least):
When visual studio builds a CA project, it first makes the CustomAction.dll, then Wix uses MakeSfxCA.exe to create a file called CustomAction.CA.dll which contains CustomAction.dll.
If you have a Post-Build event, that signs both files, it won't work, because the unsigned .dll is already packed into the CA.dll.
Now, the thing I did to solve this, was to delete the CA.dll. Make sure the CustomAction.dll file is signed, then run the MakeSfxCA.exe again Post-Build. This will produce a new CustomAction.CA.dll, which now contains a SIGNED version of your CustomAction.dll. Defender should now have no issues with your CA.

Otherwise, take a look at the article I linked. All the commands for custom compiling are in there.

Good luck!

[Superfreak3]

@Locil95 WOW, that is super helpful and critical information! 

So, you compile, sign the base.dll, delete the baseCA.dll, then run MakeSfxCA to repackage the now signed base.dll.  Did you then sign the resulting baseCA.dll again?

I'll mock up a project to ensure that everything is working properly with the scripted compile of the CA .dll.

Thanks so much for this info!!

[Locil95]

Basically yeah. I just to the default build in visual studio, then I handle everything in the post-build event.

My post build looks like this:
del /f "$(TargetDir)$(TargetName).CA.dll"
signtool HowEverYouDoYourSigningHere "$(TargetPath)"
nuget install WixToolset.Dtf.CustomAction -OutputDirectory $(ProjectDir)obj\.nuget\
$(ProjectDir)obj\.nuget\WixToolset.Dtf.CustomAction.4.0.2\tools\WixToolset.Dtf.MakeSfxCA.exe "$(TargetDir)$(TargetName).CA.dll" "$(ProjectDir)obj\.nuget\WixToolset.Dtf.CustomAction.4.0.2\tools\x86\SfxCA.dll" "$(TargetPath)" "$(TargetDir)CustomAction.config" "$(TargetDir)WixToolset.Dtf.WindowsInstaller.dll"
signtool HowEverYouDoYourSigningHere "$(TargetDir)$(TargetName).CA.dll"

1. delete the bad CA with the unsigned .dll
2. sign the CustomAction.dll
3. install WixCustomAction from nuget (to make the project more dynamic)
4. Build the CA.dll with the signed CustomAction.dll (this line will break incase WixToolset.CustomAction updates to a newer version, as the path is hardcoded. I honestly didn't care if that's the only thing I have to change in the future)
5. lastly I sign the rebuilt CA.dll file. To answer your question, yes I think you have to sign this aswell. I think I saw defender fail at it a few times when I tested before I signed it.

[Superfreak3]

@Locil95 

The fun part for me is going to be verifying the changes.  I wasn't able to reproduce the detections seen in the field as I've come to learn we don't have/utilize a robust version of Windows Defender.

So, I have to first mock up something that is detected due to the lack of a signature, make the changes you provided, then test.  If all good, I can proceed to make the changes for our actual installs.  Sadly, mocking up the detection and how-to is beyond the scope of this discussion I'm afraid.

[Locil95]

I was able to provoke a block from defender with every freshly build CA/ msi packed file. (We use the CA files in a MSI on install). On install, defender will block the CA file within the MSI Install, but only once. Defender will scan the file, realize that it is harmless, and won't block a file with that SHA256 value again. Rebuilding the CustomAction.dll inside, will renew the SHA256 value and defender will block the file again, once.

So if we ran the installer until defender had scanned all our CA steps, the install would succeed... Until the next version/next build.

[Superfreak3]

The strange thing is that not all of our CAs were flagged by Defender.  We have many and only 4 or 5 were flagged.  Some were listed as the actual .exe name and I think these were from a few SUPPORTDIR CAs.  The others were .tmp which I'm assuming are some actions from the Binary table.

[Locil95]

Maybe they we're scanned and already whitelisted. I honestly can't say for sure.

I literally have one MSI installer containing 3 CA, which each has a very simple .dll, which all were flagged by defender with each fresh build (different SHA256 value). So I didn't have the biggest dataset to go by.

Also, I had to contact another department to get to look into our defender.... until they got too bothered by me asking every 2minutes, then they gave me read access. Which is when I really made some discoveries as to what was happening 😅

Can't say for sure what is happening on your end. All I know is that for us, defender blocks .dll and .exe files for sure, that it doesn't know of the first time... scans it... if it's fine, it let's it run from there on 🤷‍:male_sign: