Demystifying the NTDS #1 - Introduction

Part 1: Format, structure, and parsing

In this series of articles, we will analyze the contents of the Active Directory (AD) database, the NTDS.dit file. This database is used as input for our managed service IAMBuster to analyze the security level of our clients’ Active Directory.

In this first part, we present the format and structure of the NTDS file, as well as the solutions for reading (parsing) its contents.

The Format

The NTDS file is the Microsoft Active Directory database (initially called NTDS for NT Directory Services). The Active Directory is the Windows directory service that manages the identification and security information for objects in an Active Directory. The NTDS file is stored on each domain controller and is created when a Windows server is promoted to a domain controller. Its default location is: %SystemRoot%\ntds\NTDS.DIT.

The format chosen by Microsoft for this database is the ESE (Extensible Storage Engine) format, also known as Jet Blue. Its creation began in 1996, when Microsoft was working on its first version of AD. The choice of this format is said to have been made at the expense of the SQL format, which at the time was limiting and constraining in terms of storage and performance. (cf. http://dsblog.azurewebsites.net/?p=762#comment-69).

Several Microsoft applications and services have also relied on the ESE database, including Microsoft Exchange up to version 5.5. Since then, the latter has relied on Active Directory. In 2021, Microsoft released the source code on GitHub.

The Structure

The NTDS file contains different tables :

As part of the analysis of the data stored within the Active Directory, we are interested in the following 3 tables:

• Datatable: contains information about the objects in the directory (users, machines, groups, etc.), including the attributes of each object (statuses, displayed names, last login or update dates, etc.). The answers to different queries that can be performed on the LDAP directory service(1) are found in this table.

• Link_table: contains the links between objects in the directory, including the identifiers of “parent” and “child” objects and the types of relationships, etc. (e.g. the MemberOf attribute of a user contains links to the groups to which they belong).

• Sd_table: contains the security descriptors of objects contained in the Active Directory, the Access Control Entries called ACE(2).

The other tables in the NTDS.dit database are used to store specific information such as storage quotas for directory objects or other technical data for the proper functioning of the AD service.

Exporting and Parsing the NTDS file

Before reading the NTDS, it should be exported first. The ntdsutil(3) utility used with the IFM(4) method, dedicated to backing up and restoring the NTDS.DIT database, is the reliable way to perform a “clean” export:

Indeed, since the NTDS is exclusively opened by the system, it cannot be properly copied during its use. However, there are other ways to extract it:

• Copy the file from a powered-off domain controller;
• Copy the file with the ntbackup or Windows Server Backup utility. However, it will be necessary to include the log and checkpoint files (present in the same folder as the NTDS) in order to rebuild it using the esentutl.exe utility.
• Make a copy from a backup using the Volume ShadowCopy Service mechanism such as VSSadmin or DiskShadow (may also require rebuilding);
• Use third-party backup software.

TIPS: If errors occur when using ntdsutil, it is recommended to check that a security solution such as an EDR does not interfere with the proper execution of the process.

Once the NTDS is retrieved, it can be read through various tools/libraries. Since parsing ESE format files is complex, it is recommended to use the following well-known resources available on the Internet:

• The esentutl.exe utility
• The libesedb C library or its Python wrapper
• NTDSXtract
• The Impacket parser
• The dissect.esedb Python module
• Incident response or backup tools such as Veeam Backup.

To facilitate data analysis, it is preferable to traverse the NTDS and export the data to a different type of database, such as SQL or MongoDB. Indeed, unlike the libesedb library, these types of databases allow queries on data returning results in a reasonable time.

NOTE: For our managed service IAMBuster, we opted for the parser included in the Impacket tool suite and for MongoDB for data storage.

The processing/parsing time of the NTDS varies depending on its size, which depends on the scope of your Active Directory. This operation can take a few minutes for a small 50 MB file, several tens of minutes for a 1 GB NTDS, or even several hours for an NTDS of over 10 GB.

This article has clarified concepts related to the format and structure of the NTDS.DIT file and has also addressed the parsing part dedicated to reading and extracting information from the Active Directory.

In the next parts, we will study the data it contains as well as the relevant information to observe in the context of a security audit.

Translated by Florian Duthu

---

(1) Lightweight Directory Access Protocol (LDAP) est un protocole permettant l’interrogation et la modification de l’Active Directory.
(2) https://learn.microsoft.com/en-us/windows/win32/secauthz/access-control-entries
(3) Disponible à partir de Windows 2008
(4) Install From Media
(5) Pour comprendre en détail dans le format ESE, nous vous invitons à lire les ressources suivantes :
Ressource 1
Ressource 2