Setting DO_NOT_ENCRYPT_METADATA flag silently switches output to embedded-files-only mode, leaving stream/string content unencrypted

[ariccy]

Summary

Passing PdfWriter.DO_NOT_ENCRYPT_METADATA to PdfStamper.setEncryption(...) (intended to keep only the /Metadata stream in plaintext while still encrypting the rest of the document) silently produces a PDF in embedded-files-only encryption mode. The resulting file has /StmF and /StrF set to /Identity, so streams and strings are written in plaintext. Adobe Reader opens the file without prompting
for a password because none of the actual content is encrypted — only embedded files would require authentication.

This is a security-sensitive bug: a caller that asks for "AES-128 with plaintext metadata" gets back a PDF that looks encrypted (the /Encrypt dictionary is present, PdfReader.isEncrypted() returns true) but whose content is not actually encrypted.

Versions affected

Confirmed on 2.2.4. Source inspection shows the same setCryptoMode logic is unchanged in 3.0.4, so this affects the entire 2.x and 3.x line.

Reproduction

import org.openpdf.text.*;
import org.openpdf.text.pdf.*;
import java.io.*;

public class Repro {
    public static void main(String[] args) throws Exception {
        // 1) Build a trivial PDF
        ByteArrayOutputStream src = new ByteArrayOutputStream();
        Document doc = new Document(PageSize.A4);
        PdfWriter.getInstance(doc, src);
        doc.open();
        doc.add(new Paragraph("Hello"));
        doc.close();

        // 2) Encrypt with AES-128 + DO_NOT_ENCRYPT_METADATA
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        PdfReader reader = new PdfReader(src.toByteArray());
        PdfStamper stamper = new PdfStamper(reader, out);
        stamper.setEncryption(
                "user".getBytes(),
                "owner".getBytes(),
                PdfWriter.ALLOW_PRINTING,
                PdfWriter.ENCRYPTION_AES_128 | PdfWriter.DO_NOT_ENCRYPT_METADATA);
        stamper.close();
        reader.close();

        byte[] encrypted = out.toByteArray();

        // 3) Adobe Reader / Acrobat opens this file WITHOUT asking for a password.
        // Dumping /Encrypt shows:
        //   /EFF /StdCF
        //   /CF <</StdCF <</AuthEvent /EFOpen /Length 16 /CFM /AESV2>>>>
        //   /StmF /Identity
        //   /StrF /Identity
        //   /EncryptMetadata false
        //
        // The stream and string crypt filter are /Identity (no encryption).
        // The /U field exists, so OpenPDF's own PdfReader still requires the password,
        // but Adobe correctly notices /StmF and /StrF are /Identity and opens the file.

        java.nio.file.Files.write(java.nio.file.Paths.get("out.pdf"), encrypted);
    }
}

Open out.pdf in Adobe Reader / Adobe Acrobat — the document opens immediately, no password prompt.

Expected behavior

DO_NOT_ENCRYPT_METADATA should leave only the /Metadata stream unencrypted (per PDF Reference 7.6.3.2). The actual document content should still be encrypted with AES-128. Adobe should prompt for the user password.

Actual behavior

Stream and string crypt filters are set to /Identity. Content is not encrypted. Adobe opens the file without authentication.

Root cause

org.openpdf.text.pdf.PdfEncryption.setCryptoMode(int mode, int kl):

public void setCryptoMode(int mode, int kl) {
    cryptoMode = mode;
    encryptMetadata = (mode & PdfWriter.DO_NOT_ENCRYPT_METADATA) == 0;
    embeddedFilesOnly = (mode & PdfWriter.EMBEDDED_FILES_ONLY) != 0;
    mode &= PdfWriter.ENCRYPTION_MASK;
    ...
}

with the constants defined in PdfWriter:

public static final int DO_NOT_ENCRYPT_METADATA = 8;   // bit 3        (0b01000)
public static final int EMBEDDED_FILES_ONLY     = 24;  // bits 3 and 4 (0b11000)

EMBEDDED_FILES_ONLY is defined as the combination of DO_NOT_ENCRYPT_METADATA (bit 3) and a dedicated "embedded files only" bit (bit 4). That choice is reasonable, because in PDF the embedded-files-only mode does imply unencrypted metadata.

The problem is the detection check: (mode & 24) != 0 returns true for any mode that has bit 3 OR bit 4 set. So when the caller passes only DO_NOT_ENCRYPT_METADATA (mode & 24 == 8 != 0), embeddedFilesOnly is incorrectly set to true, switching the output to a mode the caller never asked for.

Suggested fix

The check should require both bits of EMBEDDED_FILES_ONLY to be set, not just one:

embeddedFilesOnly = (mode & PdfWriter.EMBEDDED_FILES_ONLY) == PdfWriter.EMBEDDED_FILES_ONLY;

This preserves the intended API:

• mode = AES_128 → encrypt everything (no change)
• mode = AES_128 | DO_NOT_ENCRYPT_METADATA → encrypt everything except /Metadata ✅ (currently broken)
• mode = AES_128 | EMBEDDED_FILES_ONLY → embedded-files-only, metadata unencrypted (no change, since both bits are set)

[andreasrosdal]

Thank you for reporting. This current behavior has been like this since the start of OpenPDF as I understand it. Still it's important for security to fix it.