Categories


Authors

How to decrypt HTTPS connections for free

Ben Hooper
IT

Intro

Before I get into anything, I have two important disclaimers:

  1. These methods can be scaled up to support multiple users so if you plan on deploying this for more than just yourself then please ensure that you get explicit and informed consent from the other users - they need to be aware that you may very well be able to see their private information such as passwords and banking information.

  2. If HTTPS inspection is going to be in place for sensitive connections and/or permanently then you need to ensure that the system is sufficiently secured (for example, strong authentication, regular patching, attack surface reduction, protection of the root CA private keys, trusted management / admin connections, audit trails, etc) but I won’t be covering that in this guide as that’s way out of scope.

Originally, this blog post was titled How to get offline installers from online installers (as such, some of the examples will be related to this), basically because, in order to do my job well, I need to obtain and keep many versions of the software’s installers but some software developers don’t publish URLs for offline installers. For example, Google with Drive’s Backup and Sync client app. However, as I experimented and researched, I realised that I’d essentially ended up writing “How to decrypt HTTPS for free” and that that would be more useful to more people. It’s also quite a bit of “How to set up a virtual network appliance”.

↑ Back to Index

The simple, less reliable way

Local forward proxies are the easiest way to inspect outbound web connections and I found that Telerik’s Fiddler Web Debugging Proxy was the easiest to install and use, especially with HTTPS decryption.

In my research, I found that some applications simply ignore configured system-level proxies and, therefore, this won’t work in all cases.

Step #1: Download Fiddler

Fiddler can be downloaded from https://www.telerik.com/download/fiddler. (Their form doesn’t require email validation but if you want to avoid it then you can use https://telerik-fiddler.s3.amazonaws.com/fiddler/FiddlerSetup.exe instead.)

Step #2: Install Fiddler

Installation of Fiddler (version 5.0.20202.18177 is the latest as of writing) is very straight-forward - simply agree to the license then click Install. By default, it installs to %localAppData%\Programs\Fiddler\.

Step #3: Set up AppContainer exemptions
To do so:

  1. Open Fiddler (there’s an entry in the Start Menu).

  2. In the window AppContainer Configuration, select WinConfig.

  3. Authorise the User Account Control (UAC) prompt.

  4. If the window Orphaned Exemption Record Found prompts, select No.

  5. In the window AppContainer Loopback Exemption Utility, select Exempt All or your own selection of apps → Save Changes.

Step #4: Set up HTTPS decryption

Fiddler needs to generate a root Certificate Authority (CA) certificate and have Windows trust it so that it can perform Man-in-the-Middle (MITM) operations to decrypt and inspect the HTTPS connections.

To do this:

  1. Select Tools → Options… → HTTPS.

  2. Tick Decrypt HTTPS traffic.

  3. In Fiddler’s window prompt SCARY TEXT AHEAD: Read Carefully!, select Yes.

  4. In both Windows prompts (Security Warning and Add certificate to the Machine Root List?), select Yes.

  5. Fiddler will now elevate itself so, in the User Account Control (UAC) window prompt, select Yes.

  6. You should now see Fiddler window prompt TrustCert Success.

  7. Also, ensure that …from all processes is set.

Step #5: Set up the view
What we’re looking for is HTTP GET requests. The easiest way that I’ve found to see these is to choose any packet, select Inspectors, then select Raw.

Step #6: Monitor connections

Now you can see HTTP GET requests and other traffic.

The advanced, more reliable way

As previously mentioned, some applications aren’t designed to utilise configured proxies. This means that we have to do this at a lower level so the apps have no choice but to have their traffic inspected.

Naturally, I looked into Wireshark as it works at the lower, network interface / adapter level and is, in their own words, “the world's foremost and widely-used network protocol analyzer”. However, according to their own Wiki, it can only decrypt TLS traffic in two ways:

  1. “Key log file using per-session secrets (#Using_the_.28Pre.29-Master-Secret)”.
    I experimented with this and found that it relies on apps recognising and utilising the cross-platform environment variable SSLKEYLOGFILE which only a handful of apps such as Mozilla Firefox and Google Chrome actually do. So, this method isn’t usable for us.

  2. “Decryption using an RSA private key”.
    As far as I’m aware, this requires that we have the actual private key for each encrypted connection which obviously isn’t usable for us either.

So, we have to go another level lower. Down to the gateway level. To do that, we need to set up a virtual network appliance.

Step #1: Download and install VirtualBox

VirtualBox is Oracle’s free hypervisor which is incredibly useful. Its installer can be downloaded from https://www.virtualbox.org/wiki/Downloads and installation is very straight-forward. The Extension Pack is not required.

Step #2: Download and extract pfSense Community Edition

pfSense is an open source firewall/router based on FreeBSD. Its installation media (370+ MB compressed, 710+ MB uncompressed) can be downloaded from https://www.pfsense.org/download/?section=downloads and, in this scenario, you should use the following options:

  • Architecture: AMD64 (64-bit)

  • Installer: CD Image (ISO) Installer

The download is a GZ file so you’ll need to use an app like 7-Zip or WinRAR to extract it.

Step #3: Create a pfSense VM

To do this:

  1. Open VirtualBox.

  2. Select Machine → New… → Expert Mode and set the following basic VM options:

    • Name: pfSense or whatever you prefer.

    • Machine Folder: wherever you prefer.

    • Type: BSD

    • Version: FreeBSD (64-bit)

    • Memory size: 1024 MB or higher

    • Hard disk: Create a hard disk now

  3. Select Create and set the following virtual hard disk options:

    • File location: wherever you prefer

    • File size: 25 GB (expanding later will likely be tricky)

    • Hard disk file type: VMDK (Virtual Machine Disk) or whatever you prefer.
      In this case, this doesn’t really matter but I think it’s a good habit as it has the best compatibility with other software in my experience.

    • Storage on physical hard disk: Dynamically allocated or whatever you prefer.

      Again, in this particular case, this doesn’t really matter but this option saves actual storage drive usage at the cost of a slight performance hit.

Step #4: Configure the pfSense VM

First, we need to “insert” the installation media “disc”. To do this:

  1. Select the pfSense VM → Settings

  2. Select Storage → under Controller: IDE, the disc icon → to the right of Optical Drive:, the disc icon with a downwards-pointing arrow → Choose a disk file…

  3. Choose the ISO file extracted from step #3.

Second and last (for this step), we need to configure the network adapters. Any kind of routing device needs at least two Network Interface Controllers (NICs) - one for the Local Area Network (LAN) and one for the Wide Area Network (WAN). To do this:

  1. Select the pfSense VM → Settings → Network

  2. Select Adapter 1, ensure that Enable Network Adapter is ticked, and set Attached to to Bridged Adapter.

  3. Select Adapter 2, ensure that Enable Network Adapter is ticked, and set Attached to to Bridged Adapter.

Step #5: Check IP addresses

The pfSense’s LAN NIC defaults to static IPv4 address 192.168.1.1/24 and, because we’re using bridged network adapters with the VM which emulate a physical connection, that can temporarily cause conflict problems on the LAN if the IP address is already in use, especially if it’s important. For example, if 192.168.1.1 is the LAN IP address of your actual router / default gateway, as it is by default in a lot of cases, then your devices likely won’t be able to reliably connect to the Internet until the one of the addresses is changed.

If you need to use a different IP address, choose one (ideally outside of the DHCP scope) and ensure that it is not in use. The quickest and most reliable way to do this in my experience is to open Command Prompt or Windows PowerShell and execute command ping <IP address> until the response is Destination host unreachable.

Step #6: Install the pfSense OS

To do this:

1. In VirtualBox, select the pfSense VM → Start.
A new window will open displaying the VM’s console (what you would see if you plugged a monitor into a physical computer) and you’ll see a few boot screens but eventually you should get to the pfSense installer screen.
Tip: If your focus gets “stuck” / your cusor “disappears” in the VM console, press the right-hand Ctrl key.

2. Accept the license / terms by ensuring that Accept is selected (it is by default but press tab if not) then pressing a (as highlighted), return, or enter.

3. Ensure that Install is selected (it is by default) then select OK.

4. Choose a keymap (keyboard layout). By default, it’s set to US so it’s a good idea to correct this if required. I’m British so I choose United Kingdom then Continue with uk.kbd keymap.

5. Select partitioning option Auto (UFS).

6. Wait for the OS to install.

7. Select No to manual changes then Reboot.

8. The VM will keep booting to the ISO file / virtual disc again, rather than its own OS on the virtual hard disk, so we need to reverse step 4.1 by:

1. Selecting Machine → Settings… → Storage → disc icon → disc icon with downwards-pointing arrow → Host Drive → OK.

2. Selecting Machine → Reset.

Step #8: Set pfSense’s LAN IP address

To do this:

  1. At pfSense’s console main menu, input 2 and press return / enter twice.

  2. Complete the configuration wizard:

    • New LAN IPv4 address: The one you chose in step #5.

    • New LAN IPv4 subnet bit count: Copy that of the WAN. It’s probably 24.

    • New LAN IPv4 upstream gateway address: None.

    • New LAN IPv6 address: None.

    • Enable DHCP server on LAN: n

    • Revert webConfigurator (admin web UI) protocol to HTTP: y or n, whatever you prefer.

Step #9: Sign into pfSense’s admin web interface

To do this:

  1. Open a web browser on one of your devices and browse to the URL of your pfSense. In these examples, mine is https://192.168.1.253.

  2. Enter the default admin credentials:

    • Username: admin

    • Password: pfsense

Step #10: Set up and export pfSense’s root CA certificate

As with Fiddler (and any HTTPS inspection software), pfSense needs generate its own root Certificate Authority (CA) certificate so that it can perform Man-in-the-Middle (MITM) operations to decrypt and inspect the HTTPS connections.

To do this:

  1. Select System → Cert. Manager → + Add.

  2. Set the following information:

    • Descriptive name: pfSense Root CA or whatever you prefer.

    • Method: Create an internal Certificate Authority.

    • Key length: 2048 or higher.

    • Digest Algorithm: sha256 or higher.

    • Lifetime: Whatever you prefer but it should be a few years at least.

    • Common Name: pfSense Root CA or whatever you prefer. This is what will be displayed in web browsers, etc.

    • Country Code: Refer to https://www.ssl.com/country-codes/

    • State or Province: Whatever you prefer.

    • City: Whatever you prefer.

    • Organization: Whatever you prefer.

    • Organizational Unit: Whatever you prefer.

  3. Select Save.

  4. Select the star / sun / award symbol to export the CA certificate and save it as you’ll need it later.

Step #11: Install Squid

Squid is forward proxy software that is capable of inspecting HTTPS. To install it in pfSense:

  1. Select System → Package Manager → Available Packages.

  2. Search for squid then select Install → Confirm.

Step #12: Configure Squid

Now that Squid is installed, we need to configure it enabling HTTPS inspection and access logging:

  1. Select Services → Squid Proxy Server.

  2. Select Local Cache → Save to confirm the configuration (needs to be done or the next step will fail and you’ll have to re-set everything again).

  3. Select General and make the following changes to the default configuration:

    • Enable Squid Proxy: Ticked

    • Transparent HTTP Proxy: Ticked

    • HTTPS/SSL Interception: Ticked

    • CA: pfSense Root CA or whatever you named yours

    • Enable Access Logging: Ticked

Step #12: Configure end user device networking

Each end user device needs to be reconfigured so that its default gateway is the pfSense’s LAN IP address. This will force all network traffic to be sent to and inspected by the pfSense VM.

To do this for all devices, you’ll want to reconfigure your DHCP server.

To do for one Windows device:

  1. Open Command Prompt or Windows PowerShell, execute command ipconfig /all, and make a note of the active NIC’s IPv4 address, subnet mask, and DNS servers.

  2. Open the Start Menu and search for and open View network connections.

  3. Alt-select (right-click) on whichever NIC is being used then select Properties → Internet Protocol Version 4 (TCP/IPv4) (the text, not the checkbox) → Properties.

  4. Select Use the following IP address and enter the pfSense’s LAN IP address as the default gateway and everything else the same as it was in the output of ipconfig /all.

  5. Select OK → Close.

Step #13: Configure end user device certificates

Now we need to get the end user devices to trust the pfSense’s root CA. If this isn’t done, they’ll display a ton of security warnings / errors, as you can see below. After all, this is the whole point of Public Key Infrastructure (PKI), HTTPS, etc.

To do this on Windows:

  1. Copy over the CRT file from step #10 and open it.

  2. Select Install certificate… → Current User or Local Machine (depending on your preference) → Next → Place all certificates in the following store → Browse… → Trusted Root Certification Authorities → OK → Next → Finish → Yes.

You can now see that HTTPS connections signed by pfSense are trusted:

Step #14: Monitor connections

If you browse to pfSense → Services → Squid Proxy Server → Real Time → Squid Access Table, you should now see a list of URLs being accessed in real-time (the page automatically updates every few seconds).

Step #15: Whitelist FQDNs, if required

Due to things like HTTP Public Key Pinning (HPKP), client certificate authentication, and certificate attributes that are lost when proxied, some applications simply do not like their HTTPS connections being inspected and refuse to function unless their Fully-Qualified Domain Names (FQDNs) are whitelisted and, therefore, excluded from HTTPS inspection. If this is the case, you’ll likely see something like the following:

TCP_TUNNEL_ABORTED/200 client.dropbox.com:443

To whitelist FQDNs in Squid on pfSense, simply go to Services → Squid Proxy Server → ACLs → Whitelist.

Hello, World!

A CSV of known FQDNs that must be whitelisted is available here: https://mythofechelon.co.uk/s/SSLTLS-decryption-exclusions.csv

↑ Back to Index

Un-trusting root CAs

Chances are that you only wanted HTTPS inspection to inspect a specific application or something so you won’t want to leave the system in place.

So, to delete or “un-trust” a root Certificate Authority (CA) on Windows if your user account does have local administrative permissions:

  1. Open certmgr.msc.
    Even though it says Certificates - Current User, the certificates listed are actually a combination of that of the local machine and current user. So this will work regardless of whether you installed the root CA certificate into the local machine’s or the current user’s store.

  2. Expand Trusted Root Certification Authorities then select Certificates.

  3. Alt-select (right-click) on the pfSense certificate then select Delete.

To delete or “un-trust” a root Certificate Authority (CA) on Windows if your user account does not have local administrative permissions:

  1. Open mmc.exe as admin.

  2. Authenticate the UAC prompt.

  3. Select File → Add/Remove Snap-in… → Certificates → Add > → Computer account → Next → Finish → OK.

  4. Expand Certificates (Local Computer) → Trusted Root Certification Authorities then select Certificates.

  5. Alt-select (right-click) on the pfSense certificate then select Delete.

Your PC will now no longer trust connections signed by pfSense and, therefore, will display warnings again.

↑ Back to Index

Sign-off

I hope that this has been useful.

Feel free to subscribe to my newsletter to be automatically notified of new blog posts in the future.

😊

How to make custom Squarespace Tudor social links

Great YouTube channels