How to Achieve Eternal Persistence Part 3: How to access and recover replicated secrets

To get other types of keys, we need to look at replication traffic. After a password has been changed, the new password must be shared with other domain controllers (DCs). The DC on which the password has been changed will send out an urgent replication request to its replication partner. This request tells the receiving DC “Hey, got some data for you“. The receiving DC will then issue a replication request to the issuing DC to receive the updated data. This is different than the message types detailed in the previous blog post, where the DC will send the new credentials to the PDC first.

This request is wrapped in a struct that looks something like the following[1]:

This request contains the object references, the attributes and something like the version counter of the data the issuing DC currently has. This is taken into account and updated data will be returned to the issuing DC. That reply is built according to the following structure[2]:

There are multiple iterations of both requests and replies, adding more features and fields with newer versions of Windows. These versions were encountered the most during testing, but your mileage may vary.

Replication data

Replication data is sent over the network using the Network Data Representation (NDR) syntax. This is a common practice for API calls in the Windows RPC protocol to send structured data over the wire.

If you have Wireshark set up correctly to decrypt Kerberos messages (see previous blog post for pointers to set this up), enter the following filter drsuapi.opnum == 3 && dcerpc.pkt_type == 2. This filter will display only responses (pkt_type == 2) to replication requests (opnum == 3), and you should be able to see the decrypted replication data: 

This data is the raw NDR formatted data of the DRS_MSG_GETCHGREPLY_V* structure mentioned earlier. In order to iterate through all the objects and attributes, we need to decode it.

There are various tools that can decode NDR data, including:

• Sharpkatz[3]
• Mimikatz[4]
• Impacket[5]
• WindowsProtocolTestSuites[6]
• NtApiDotNet[7]

While Sharpkatz and Mimikatz are able to decode NDR data, they handle the complete process, utilizing Windows Kernel API calls to craft the request message, invoke the RPC call and process the response. Replicating this (Got it?) so it would work with only raw bytes, filling in pointers and such seemed a bit too tedious. Impacket has an NDR decoder which a lot of Impacket tools rely on. This worked in some scenarios, but resulted in decoding issues most of the time.

WindowsProtocolTestSuites is a repository maintained by Microsoft that contains samples and examples for a variety of internal Windows protocols. This includes examples for encoding and decoding NDR data streams. This also worked in some scenarios, but resulted in decoding issues most of the time.
This led me to NtApiDotNet. This project, created by James Forshaw of Google Project Zero, can extrapolate an NDR encoder and decoder from a binary with RPC server logic in it. This means that if we create a binary that invokes the same RPC calls that are invoked during the replication process, NtApiDotNet should be able to generate code to encode data into NDR formatted data or decode NDR data back into the original structure.

We took that route and created the binaries to be used with the NtApiDotNet project. Details on how to replicate this can be found in this GitHub repository: https://github.com/huntandhackett/PassiveAggression. This repository also contains pointers on how to set things up correctly.

Once the binary has been compiled successfully, make sure you have a copy of NtApiDotNet. Go to the output folder, import the NtApiDotNet module and check if the RPC server is recognized:

Next, use the steps documented in this blog post to compile an encoder and decoder:

This should result in a C# class file with methods to deserialize the NDR data back into the right struct. After some refactoring for easier readability and removing unused functions, I was able to deserialize NDR data: 

This worked well, until it didn’t. This pattern became a bit too repetitive for my liking and was a bit confusing. After staring the issue right in the eye for maybe a day, it finally became clear. 

(Pointer) size matters

To understand why deserializing some data worked while other data didn’t, we need some basic understanding of how RPC calls are established.

Step 1: Endpoint mapping

The client sends a mapping request to the server. The client includes the RPC interface it wants to connect to. The server responds with endpoint information, including the port the client can connect to. 

Step 2: Binding

This is where the client establishes a connection with the server and the RPC interface it wants to connect to. This phase includes authentication details and syntax negotiation. 

Step 3: Altering context

Clients may send an alter context request if additional settings or features must be negotiated, such as authentication or a different encryption algorithm.

Step 4: RPC call

This is the step where the actual RPC call is invoked under the negotiated context. 

Step 5: Cleanup

Client or server sends unbind request. 

Now let’s zoom in on the second step. During the bind call, the client will send two context items. These items contain the languages the client can speak. In the example below, the client can handle both 32bit and 64bit NDR syntax. 

When looking at the response, the server rejects the 32bit NDR syntax option and acks the 64bit option:

This means that for all other future calls in the established RPC context, data is transferred over the wire using NDR x64 bit syntax. This correlates with data being decoded successfully, had pointers 32bits in size: 

Packets that were not able to decode use 64bits pointers, as can be seen below:

This meant that the reason why Impacket, ProtoSDK and the C# class NtApiDotNet products didn’t work is because they only support 32bit NDR message syntax. Since we’re dealing with intercepted data and cannot alter the binding process, we have to deal with what the client and server negotiated.

In some scenarios the client may send an alter_context request only providing a 32bit NDR syntax context, which the server accepts. RPC with all its quirks is a mind boggling journey on its own, so we did not spend a lot of time finding out how and why that is.  If you do know why that is and in what context that occurs, please contact us at jobs@huntandhackett.com.

Making it work

Creating a new 64bit NDR decoder from scratch was not something to be desired and since the chosen path was already deep down the rabbit hole with the NtApiDotNet approach, going with this solution seemed the best approach.

Aligning pointers

Tooling such as Mimikatz and Sharpkatz can invoke the GetNCChanges RPC call and request lots of data from a domain controller. When doing so, pointer sizes never exceed 32bits. With this in mind, we assume for now that the last 4 bytes of a pointer in the NDR stream are always null bytes.

Using class NdrUnMarshalBuffer from the NtApiDotNet library, we can load all NDR into a memory stream. This class exposes methods to consume data from the stream. Calculating alignment and consuming the bytes to get data aligned again was straightforward, but in some cases actual data was consumed causing misalignment. Since the data was already consumed and the library did not have any method to restore the data onto the stream, I extended the library with a peek method that restored the position of the stream. 

This allowed for peeking at the stream without actually consuming the data.

Conformant arrays

The other obstacle had something to do with conformant arrays. Let’s assume the following structs: 

With a conformant array, the length of - in this case - StringName is placed before the actual start of the struct. When this array is read from the stream, the length of StringName will be read and placed onto a stack. Next, the complete struct is read from the stream and processed. When the StringName field is processed, the length referent is used that was put before the start of the struct. This works if the size of the pointers are 32bit, but when they’re 64 bit this value will be different, meaning that the StringName field will be null or complete gibberish.

This is clear when looking at the following hex dump. When reading a conformant array, the length of StringName will be read, but the first 8 bytes are all 00s because the stack is misaligned. 

Fixing the alignment by consuming 8 bytes from the stream before the struct is deserialized would fix the issue in this scenario. However, this struct is referenced frequently and the amount of alignment it needs varies. Structurally consuming 8 bytes would result in misalignment somewhere else later on in the stream.

Fixing this issue was trivial, since the data on the stream is linear and we happen to know the length of the StringName field, which is stored in NameLen and only needs to account for the trailing 00-byte.

However, this did cause another issue. When reading conformant arrays, the deserializer checks if there are items left on the stack. If so, the conformant array length will not be put on the stack, resulting in other structs not being parsed. To fix this, two functions were added to the NtApiDotNet library:

The first method manually resets the stack, making sure that future conformant arrays will be read successfully. The second function returns the stack to the caller. The caller can then verify if the item on the stack (if any) is what is to be expected. If not, the stack can be cleared using the first method. A pull request has been created[8] to get this merged into the project, but at the time of writing it still needs to be processed by one of the maintainers of the repository.

After grinding through all structs and aligning all pointers correctly, 64bit NDR messages can now be decoded successfully.

Decoding replication traffic

In the following screenshot, we can see the password change of user test. We see that the domain controller first sends the password reset event to the primary domain controller by invoking the NetrLogonSendToSam RPC call. This call includes both NTLM and LM hashes and the relative User ID (rID) of the user account. Then, the secret is replicated. This does not include the LM hash, but does include the NTLM hash and AES keys.

In the beginning of the first blog post, we stated that having eternal passive persistence was the ultimate goal. For that, we definitely want access to all key material of the krbtgt account, but as seen in this blog post, replication traffic is encrypted and can only be decrypted if you have access to the keys of the domain controller’s computer accounts and the krbtgt account. Domain controllers will have access to these keys by default, since they own their own copy of the NTDS database.

Let’s say that in a domain with two domain controllers, A and B, we initiate a password reset on the krbtgt account on DC A. How is DC B informed of this new password that DC A will create?

The answer is simple: via replication. Just like passwords of any regular user account, the new password of the krbtgt account will be shared using the replication process. Since replication traffic must be encrypted and DC B does not yet know the new password of the krbtgt account, DC A will use the previous password of the krbtgt account.

This creates a rather interesting catch-22 issue. If we, as an attacker, have access to all key material in a domain and have access to the domain controllers or devices through which inter DC traffic is routed we can intercept, decrypt and decode all password changes, including that of the krbtgt and domain controller computer accounts, resulting in practically eternal persistence. It is not necessary to have access to a domain controller, for example the following scenarios should suffice:

• Access to (virtual) switches that switches network traffic between domain controllers
• Access to hypervisors that routes network traffic from and to domain controllers
• Access to span ports on a network device that mirrors all network traffic, including traffic to/ from domain controllers
• Access to a 3rd party security provider that has access to all network flows

This can be demonstrated using the image below:

At this point, you might be thinking “Having access to the krbtgt credential does not mean you can decrypt traffic sent to and received from domain controllers. You need the machine password for that as well". That would be correct.

Let’s reset the password of a domain controller using this command:

netdom resetpwd /s:other.dc.local /ud:rebel\Administrator /pd:*

This will reset the password of the domain controller it is run on, using the domain controller specified in the /s switch. (Fun fact, it seems that it uses the SamrSetInformationUser2 RPC call to reset the password). Other domain controllers are notified of this new password via replication, which we can intercept and decode:

Being able to decode replication data passively has a significant impact. It not only allows for intercepting password changes of user accounts, machine accounts and the krbtgt AD service account, you can also become your own passive domain controller where you can keep track of all replicated data and use this as a method to exfiltrate data. If the replication decoder is running on a switch or router, it can exfiltrate all the data with little chance of being detected.

Checking our goals

In the first blog post of this series, we set a few goals to grade the solutions. This method can be run on any device that receives traffic sent to and received from domain controllers, is completely passive, contains keys used with modern encryption types and most importantly: it survives a remediation process. Regardless of whether the krbtgt account is replicated or the password of a domain controller is changed and communicated to other domain controllers, this should work because of the way AD works.

Figure 1. Results vs predetermined goals for eternal persistence

This has a serious impact on how remediation should be carried out, especially if the actor is sophisticated enough to conduct these kind of attacks.

Remediation

After reading this blog series, you might be wondering and thinking "But doing proper remediation would prevent all of this, right?" That would technically be correct, but in reality it depends on the remediation process. This kind of attack could survive a remediation process where:

• current domain controllers are demoted and replaced with new domain controllers
• all passwords are reset twice
• all servers, workstations and other AD assets and objects are cleaned and sanitized

The reason why this method could survive such a remediation process, is - as stated before - it can be executed on any device in the network, as long as network traffic between domain controllers can be intercepted. This can be a router, switch, hypervisor or a security provider with access to network data.

To fully remediate such a scenario, one could think of disconnecting all domain controllers from existing network equipment and  generate new credential data twice (krbtgt keys, domain controller account passwords, etc) and make sure to fully replicate data across other domain controllers. This could increase remediation costs, effort and complexity, especially if domain controllers are located in multiple data centers.

Given their passive nature, these kinds of attacks are hard to detect. However, it can only be executed after full compromise of the AD domain which underlines the importance of security aspects such as prevention, detection and timely response. If you have questions about our research, require expert advice, or you are simply curious about improving your own organisation's resilience, contact us here. 

References

1. https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-drsr/4304bb4a-e9b5-4c8a-8731-df4d6f9ab567
2. https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-drsr/1317a654-5dd6-45ff-af73-919cbc7fbb45
3. https://github.com/b4rtik/SharpKatz
4. https://github.com/ParrotSec/mimikatz
5. https://github.com/fortra/impacket
6. https://github.com/microsoft/WindowsProtocolTestSuites
7. https://github.com/googleprojectzero/sandbox-attacksurface-analysis-tools
8. https://github.com/googleprojectzero/sandbox-attacksurface-analysis-tools/pull/74