qbiocode.utils.find_duplicates module#

File duplicate detection utilities for identifying identical files in directories.

This module provides functions to find duplicate files based on content comparison, useful for cleaning up redundant configuration files or identifying duplicate datasets.

Summary#

Functions:

find_duplicate_files

Find files with identical content in a directory.

Reference#

find_duplicate_files(directory, file_pattern=None, ignore_empty_lines=True, case_sensitive=True, verbose=False)[source]#

Find files with identical content in a directory.

Scans the specified directory for files and compares their content line by line. Identifies files that have identical content, even if they have different names. Optionally filters files by pattern and provides various comparison options.

This is particularly useful for:

Finding duplicate configuration files (e.g., YAML, JSON)
Identifying redundant experiment configurations
Cleaning up duplicate datasets before batch processing
Validating file uniqueness in automated workflows

Parameters:

directory (str) – Path to the directory to search for duplicate files.
file_pattern (str, optional) – File extension or pattern to filter (e.g., ‘.yaml’, ‘.csv’, ‘.txt’). If None, all files are compared. Default is None.
ignore_empty_lines (bool, optional) – If True, empty lines are ignored during comparison. Default is True.
case_sensitive (bool, optional) – If True, comparison is case-sensitive. Default is True.
verbose (bool, optional) – If True, print progress information during comparison. Default is False.

Returns:

List of tuples, where each tuple contains paths of two duplicate files. Returns empty list if no duplicates are found.

Return type:

List[Tuple[str, str]]

Raises:

FileNotFoundError – If the specified directory does not exist.
NotADirectoryError – If the specified path is not a directory.
PermissionError – If files cannot be read due to permission issues.

Examples

Find all duplicate files in a directory:

>>> duplicates = find_duplicate_files("configs/")
>>> if duplicates:
...     print(f"Found {len(duplicates)} duplicate pairs")

Find duplicate YAML configuration files:

>>> duplicates = find_duplicate_files(
...     "configs/qml_gridsearch/",
...     file_pattern='.yaml',
...     verbose=True
... )
>>> for file1, file2 in duplicates:
...     print(f"Duplicate: {file1} == {file2}")

Case-insensitive comparison:

>>> duplicates = find_duplicate_files(
...     "data/",
...     file_pattern='.txt',
...     case_sensitive=False
... )

Integration with QProfiler workflow:

>>> # Check for duplicate configs before batch processing
>>> config_dir = "configs/experiments/"
>>> duplicates = find_duplicate_files(config_dir, file_pattern='.yaml')
>>>
>>> if duplicates:
...     print("Warning: Duplicate configurations found!")
...     for f1, f2 in duplicates:
...         print(f"  {os.path.basename(f1)} == {os.path.basename(f2)}")
...     # Optionally remove duplicates or warn user

Notes

Files are compared line by line after sorting (order-independent)
Binary files are not supported; use for text files only
Large files may consume significant memory during comparison
Symbolic links are followed and treated as regular files
Hidden files (starting with ‘.’) are included in comparison