Secrets in Automation: Three Places a Key Leaks Quietly

The publishing script ran ten times in a row in my terminal. I put it into the task scheduler - and it started failing with an access error. Same script, same machine, same user. By hand it works, on a schedule it doesn't. Digging into it took an unpleasant amount of time, and in the end the problem wasn't permissions on the service, it was which exact key made it to the code at launch time.

This article is about handling secrets in automation: the tokens, access keys and passwords a script presents to external services. There will be no survey of secret stores or threat models. There will be a few simple rules, each of which came out of a specific mistake of mine, and a way to check that you're following them.

flowchart LR
  S[Секрет] --> A[Аргумент команды]
  S --> B[Токен в строке адреса в коде]
  S --> C[Ключ в адресе запроса]
  A --> A1[Список процессов и история оболочки]
  B --> B1[Исходники и резервные копии]
  C --> C1[Текст ошибки в журнале]
  A1 --> F1[Чтение из файла окружения]
  B1 --> F2[Штатное хранение учетных данных]
  C1 --> F3[Маскирование перед записью]

Three places a secret leaks without you noticing

A leak rarely looks like a leak. Nobody publishes a key on purpose - it just ends up somewhere you never counted as storage.

The first place is command arguments. The most natural way to hand a script a token is to write it straight into the launch line. The problem is that command-line arguments are visible in the process list - the roster of running programs that any user on the machine can look at. While the command is running, the key hangs there in plain text. And once it finishes, it stays in the shell history - the file where the terminal records everything you typed. History survives a reboot and ends up in backups of your home directory.

The second place is the code itself. A classic: to push changes to a repository, the token gets substituted right into the address string - you end up with an address that has a key inside it. That string is assembled in the source, which means it lives in the source. From there it travels into every copy of the project, into archives, into backups, into chat when you send a colleague a chunk of the script with "see what's wrong here." A secret that has made it into code stops being yours - you no longer control the list of places it sits in.

The third place is error messages. This one stings the most, because here you did everything right and the leak happened anyway. Some services accept the key directly in the request address. While everything works, you don't notice. On a failure the client library builds the error text, inserts the whole request address into it - and your handler dutifully writes that text to the log. Logs are kept for a long time, are often read by people who were never granted access to the secret, and frequently get shipped off to log collection systems.

Rule: read the secret where it's used

The secret is not passed as an argument and is not stored in the source. It lives in an environment file - an ordinary text file next to the script, with lines holding "variable name = value" pairs. The script reads the value it needs at the moment it's about to use it, and doesn't drag it through half the program.

Permissions on that file: owner only, not the group, not anyone else. An environment file that's world-readable differs from a token in the source only in being harder to copy by accident. Exactly one line when you create the file - and one check when you're investigating an incident.

The trap: whose key won

Here's that scheduler story. The environment-file loader - the library that reads the file and spreads the values into the process environment - was by default running in "don't overwrite what's already there" mode. The logic is understandable: the environment is treated as a more explicit instruction than the file.

Under the scheduler, the environment already held a key for a different integration - under the same variable name. The file with "my" key was honestly read, the value was ignored as already set, and the script silently went to the wrong place. It didn't crash, it didn't warn - it presented someone else's key to someone else's endpoint and got an access denial. By hand everything worked, because in my interactive session that variable simply wasn't in the environment.

The conclusion I now apply by default: the file next to the script must beat the inherited environment. Automation runs in an environment you didn't build and can't see, and it inherits everything in it indiscriminately. If the file doesn't overwrite the environment, you get silent errors that can't be reproduced by hand - and that's the worst class of error, because debugging happens precisely by hand.

Rule: the built-in mechanism instead of a homemade one

The second leak - a token inside the address for pushing to a repository - isn't cured by being careful, it's cured by giving up the do-it-yourself approach. The version control system has a built-in mechanism for storing credentials: it remembers what to authorize with and supplies it when it talks to the server. The script doesn't need to assemble anything into a string, which means there's nothing to end up in the source.

Generalized, the rule sounds like this: if a tool has its own way of storing credentials, use it, even if yours looks five lines shorter. A built-in mechanism separates the secret from the code at the level of how the tool is built; a homemade one rests entirely on your discipline. Discipline breaks on the very first evening when something needs an urgent fix.

import logging
import os
from pathlib import Path

from dotenv import load_dotenv

# override=True: файл рядом со скриптом важнее того,
# что унаследовано от планировщика
load_dotenv(Path(__file__).with_name(".env"), override=True)


def token() -> str:
    # читаем там, где используем, а не в глобальной константе
    return os.environ["PUBLISH_TOKEN"]


def mask(text: str) -> str:
    # замена по значению, а не по шаблону «похоже на ключ»
    return text.replace(token(), "***")


def publish(payload):
    try:
        return send(payload, api_key=token())
    except Exception as err:
        logging.error("публикация не удалась: %s", mask(str(err)))
        raise

Rule: mask in logs

The third leak can't be closed at the entrance - the error text comes from someone else's code, and you don't control what they put in it. So the filter goes at the exit: before writing to the log, the text is run through a replacement that finds the token's value in it and substitutes a placeholder.

An important detail: mask by the secret's value, not by a pattern like "a string that looks like a key." A pattern falls behind reality the first time the token format changes, while replacement by the actual value always works - you're searching for exactly the string you presented to the service yourself.