Custom Action Utilities Flagged By Windows Defender...

[Superfreak3]

Hi all,

This is really a two-part question.  First, why are some custom actions reported by the actual .exe or .dll name for some actions and MIS*.tmp for others?  Is that governed by the actual Custom Action type or something to that effect?  Or if some are support files and some are pulled from the Binary table, would that do it?

Secondly, and the main question...  What do you do for actions being flagged by Windows Defender?  Besides whitelisting, will signing those executables help?

With the MSI*.tmp actions/executables, will signing the actual .exe/.dll being called and referenced as .tmp in the log help?

We're starting to see more and more issues with Defender lately.

Thanks for any info you can provide.

[Superfreak3]

@skrueger  Let me know if posting elsewhere might yield answers!

[skrueger]

Maybe a Defender forum or Microsoft support?

[dan_galender]

What type of Custom Actions are being flagged?   DLL?  EXE?  VBScript?  Managed Code?

[Superfreak3]

One of the two named binaries, the .exe, is Synchronous (Check Exit Code), Deferred Execution, MSI Type Number 1058.

The other named binary is a .dll, which detects the presence of a third-party app.  It runs in both the UI and Execute sequences.

The Custom Actions are comprised of primarily C# Code.

[Superfreak3]

So, to see if this combats the problem successfully, I've gone ahead and started signing my Custom Action binaries.  I've come to an MSI .dll that is written using a WiX template.  On compile a MyFile.dll and MyFile.CA.dll are generated.  In InstallShield, the MyFile.CA.dll is referenced or pointed to in the Custom Action settings.  When I sign the target file using signtool, the MyFile.dll is digitally signed, but the .CA.dll is not.  So I attempted to add another post-build step to sign the .CA.dll, but that errors out...

EXEC : SignTool error : SignedCode::Sign returned error: 0x800700C1
1>
1> Either the file being signed or one of the DLL specified by /j switch is not a valid Win32 application.
1>
1>EXEC : SignTool error : An error occurred while attempting to sign: C:\Install Source\MyApp\Custom Actions\MyAppCheck\MyAppCheck\bin\Release\MyAppCheck.CA.dll

Does anyone have any experience using WiX MSI .dll Custom Actions.  Which file is actually used in the install pacakge, the main .dll or the .CA.dll?

I'm hoping in some way that the CA .dll signals Install shield to include the regular .dll as that is the file I'm able to sign.

@skrueger  @dan_galender 

[Locil95]

No idea if you're still having issues with this, but I had the exact same issue as you, when compiling through Visual Studio, and deploying a package to another machine. Defender would block those CA files. Knowing how much frustration I went through trying to figure out a solution to this, I remembered your post and thought I might come back and share.
As you might have guessed, I have solved this issue at our end. I did so by manually adding a rebuild after signing the .dll in a Post-build event. I had to upgrade to WixToolset 4. But other than that, it now works like a charm.

I found the final hint on this page:
https://wixtoolset.org/docs/tools/dtf/

I figured out what happens when Wix builds those CA projects, which sadly doesn't work with signing (not in Visual studio at least):
When visual studio builds a CA project, it first makes the CustomAction.dll, then Wix uses MakeSfxCA.exe to create a file called CustomAction.CA.dll which contains CustomAction.dll.
If you have a Post-Build event, that signs both files, it won't work, because the unsigned .dll is already packed into the CA.dll.
Now, the thing I did to solve this, was to delete the CA.dll. Make sure the CustomAction.dll file is signed, then run the MakeSfxCA.exe again Post-Build. This will produce a new CustomAction.CA.dll, which now contains a SIGNED version of your CustomAction.dll. Defender should now have no issues with your CA.

Otherwise, take a look at the article I linked. All the commands for custom compiling are in there.

Good luck!

[Superfreak3]

@Locil95 WOW, that is super helpful and critical information! 

So, you compile, sign the base.dll, delete the baseCA.dll, then run MakeSfxCA to repackage the now signed base.dll.  Did you then sign the resulting baseCA.dll again?

I'll mock up a project to ensure that everything is working properly with the scripted compile of the CA .dll.

Thanks so much for this info!!

[Locil95 · Accepted solution]

Basically yeah. I just to the default build in visual studio, then I handle everything in the post-build event.

My post build looks like this:
del /f "$(TargetDir)$(TargetName).CA.dll"
signtool HowEverYouDoYourSigningHere "$(TargetPath)"
nuget install WixToolset.Dtf.CustomAction -OutputDirectory $(ProjectDir)obj\.nuget\
$(ProjectDir)obj\.nuget\WixToolset.Dtf.CustomAction.4.0.2\tools\WixToolset.Dtf.MakeSfxCA.exe "$(TargetDir)$(TargetName).CA.dll" "$(ProjectDir)obj\.nuget\WixToolset.Dtf.CustomAction.4.0.2\tools\x86\SfxCA.dll" "$(TargetPath)" "$(TargetDir)CustomAction.config" "$(TargetDir)WixToolset.Dtf.WindowsInstaller.dll"
signtool HowEverYouDoYourSigningHere "$(TargetDir)$(TargetName).CA.dll"

1. delete the bad CA with the unsigned .dll
2. sign the CustomAction.dll
3. install WixCustomAction from nuget (to make the project more dynamic)
4. Build the CA.dll with the signed CustomAction.dll (this line will break incase WixToolset.CustomAction updates to a newer version, as the path is hardcoded. I honestly didn't care if that's the only thing I have to change in the future)
5. lastly I sign the rebuilt CA.dll file. To answer your question, yes I think you have to sign this aswell. I think I saw defender fail at it a few times when I tested before I signed it.

[Superfreak3]

@Locil95 

The fun part for me is going to be verifying the changes.  I wasn't able to reproduce the detections seen in the field as I've come to learn we don't have/utilize a robust version of Windows Defender.

So, I have to first mock up something that is detected due to the lack of a signature, make the changes you provided, then test.  If all good, I can proceed to make the changes for our actual installs.  Sadly, mocking up the detection and how-to is beyond the scope of this discussion I'm afraid.

[Locil95]

I was able to provoke a block from defender with every freshly build CA/ msi packed file. (We use the CA files in a MSI on install). On install, defender will block the CA file within the MSI Install, but only once. Defender will scan the file, realize that it is harmless, and won't block a file with that SHA256 value again. Rebuilding the CustomAction.dll inside, will renew the SHA256 value and defender will block the file again, once.

So if we ran the installer until defender had scanned all our CA steps, the install would succeed... Until the next version/next build.

[Superfreak3]

The strange thing is that not all of our CAs were flagged by Defender.  We have many and only 4 or 5 were flagged.  Some were listed as the actual .exe name and I think these were from a few SUPPORTDIR CAs.  The others were .tmp which I'm assuming are some actions from the Binary table.

[Locil95]

Maybe they we're scanned and already whitelisted. I honestly can't say for sure.

I literally have one MSI installer containing 3 CA, which each has a very simple .dll, which all were flagged by defender with each fresh build (different SHA256 value). So I didn't have the biggest dataset to go by.

Also, I had to contact another department to get to look into our defender.... until they got too bothered by me asking every 2minutes, then they gave me read access. Which is when I really made some discoveries as to what was happening 😅

Can't say for sure what is happening on your end. All I know is that for us, defender blocks .dll and .exe files for sure, that it doesn't know of the first time... scans it... if it's fine, it let's it run from there on 🤷‍:male_sign:

[Superfreak3]

So, I'm finally getting around to trying this out and things are ok to this point.

del /f "$(TargetDir)CustomAction1.CA.dll"
powershell.exe -file "C:\Install Source\Digital Signature\Signer.ps1" -fileName "$(TargetPath)"
C:\Nuget.exe\nuget.exe install WixToolset.Dtf.CustomAction -OutputDirectory $(ProjectDir)obj\.nuget\

The .CA.dll is deleted, the base .dll is signed, and the WixToolset.DTF.CustomAction kit is installed.

I have a question about the next phase....

Is this all one command...

$(ProjectDir)obj\.nuget\WixToolset.Dtf.CustomAction.4.0.2\tools\WixToolset.Dtf.MakeSfxCA.exe "$(TargetDir)$(TargetName).CA.dll" "$(ProjectDir)obj\.nuget\WixToolset.Dtf.CustomAction.4.0.2\tools\x86\SfxCA.dll" "$(TargetPath)" "$(TargetDir)CustomAction.config" "$(TargetDir)WixToolset.Dtf.WindowsInstaller.dll"

When I run the post build with all of that as a single command, I get this...

1>EXEC : error : Invalid argument: No CA or UI entry points found in module: C:\Temp\WiXSigningTest\CustomAction1\bin\Release\CustomAction1.dll

My Custom Action Code looks like this...

namespace CustomAction1
{
public class CustomActions
{
[CustomAction]
public static ActionResult WiXSigningTest(Session session)
{
session.Log("Beginning signed CA test...");

MessageBox.Show("DLL launched!");

session.Log("Test Successful!");

return ActionResult.Success;
}
}
}

I read something about making it a public static class, but that didn't seem to make a difference.

Any idea what might be wrong here @Locil95 ?

[Superfreak3]

Oh and I should mention that the custom action is WiX 3x, I believe, but the toolset is 4.0.3.  I wonder if that could be the issue.  If so, ill have to look into upgrading WiX itself to 4x I guess.  And if I have to do that, is the transition from 3 to 4x seamless?

[Superfreak3]

I see now that...

$(ProjectDir)obj\.nuget\WixToolset.Dtf.CustomAction.4.0.2\tools\WixToolset.Dtf.MakeSfxCA.exe "$(TargetDir)$(TargetName).CA.dll" "$(ProjectDir)obj\.nuget\WixToolset.Dtf.CustomAction.4.0.2\tools\x86\SfxCA.dll" "$(TargetPath)" "$(TargetDir)CustomAction.config" "$(TargetDir)WixToolset.Dtf.WindowsInstaller.dll"

... is all one command.  I did get it to generate the .CA.dll once, but I must have changed something and now its not working again, but I'm getting closer!

I wonder how much the architecture of the SfxCA.dll used in the commands matters.

[Superfreak3]

Dang it!  Stumped!  Here are my current Post-build events...

del /f "$(TargetDir)$(TargetName).CA.dll"
powershell.exe -file "C:\Install Source\Digital Signature\Signer.ps1" -fileName "$(TargetPath)"
C:\Nuget.exe\nuget.exe install WixToolset.Dtf.CustomAction -OutputDirectory $(ProjectDir)obj\.nuget\
"$(ProjectDir)obj\.nuget\WixToolset.Dtf.CustomAction.4.0.3\tools\WixToolset.Dtf.MakeSfxCA.exe" "$(TargetDir)$(TargetName).CA.dll" "$(ProjectDir)obj\.nuget\WixToolset.Dtf.CustomAction.4.0.3\tools\x86\SfxCA.dll" "$(TargetPath)" "$(TargetDir)CustomAction.config" "$(TargetDir)WixToolset.Dtf.WindowsInstaller.dll"
powershell.exe -file "C:\Install Source\Digital Signature\Signer.ps1" -fileName "$(TargetDir)$(TargetName).CA.dll"

The initial .dll is getting signed, toolset is getting installed (or detected as already installed), but making the .CA.dll is crapping out with...

EXEC : error : Invalid argument: No CA or UI entry points found in module: C:\Temp\WiXSigningTest\CustomAction1\bin\Release\CustomAction1.dll

[Locil95]

Hmm... I don't remember which errors I got, but I know that I had to upgrade all Wix Projects, including the CustomActions to WiX 4x. I was not able to get it to work with WiX 3x.

I used this tooling/template package called HeatWave from FireGiant to upgrade/recreate the projects:
https://wixtoolset.org/docs/fourthree/
https://www.firegiant.com/docs/heatwave/

Try upgrading and run the commands again. If it's that same, I'll try digging.

[Superfreak3]

Thanks so much for that info.  I've installed the 2019 VS HeatWave extension, but I don't see the Upgrade to WiX v4 menu option.  Thought maybe a VS restart, but still nothing.

[Locil95]

I don't think I used a menu option. I used a command for the WiX installation project, and then I recreated the CustomAction projects from scratch using the templates from HeatWave. Then manually copied the logic from the old WiX 3x CA Projects, into the new 4x Projects.

[Superfreak3]

Yeah, after posting the menu option question, I went ahead and recreated my test .dll from scratch with v4.  That compiled successfully but runtime problems so more hunting...

Error 1723. There is a problem with this Windows Installer package. A DLL required for this install to complete could not be run. Contact your support personnel or package vendor. Action WiXSigningTest, entry: WiXSigningTest, library: C:\Users\BUILDU~1\AppData\Local\Temp\MSI25C4.tmp

I have an InstallShield Basic MSI project and I'm just pointing to the new .dll for the custom action, but failure - is there anything that has to be done to the installation itself I wonder.