6.2 Leave Searchable Logs Behind

Prerequisites: Understanding Errors and Call Stacks
Estimated time: 20 minutes
Exercise environment: Python 3.9+ standard library; logging to a temporary directory
You are ready to continue when: you can write searchable and non-leaking logs, and know where to find them

Evidence Must Survive the Terminal

If a program crashes while nobody is watching the terminal, debugging still needs the failure time, request identity, and preceding steps. Keeping a terminal open is not observability; it is merely a very committed screen. The useful artifact is an event record that can be collected, searched, and correlated.

Logs, Output Streams, and Storage Locations

Logs are events recorded by applications. The logging framework is responsible for levels, fields, formatting, and routing, while the final destination depends on deployment:

Local applications can write to the console or files;
the systemd journal can collect service logs;
Containers typically write to stdout / stderr, which are then collected by the runtime or cluster;
Centralized systems send logs from multiple machines to a unified backend.

Therefore, "no one looks at stdout in production" and "logs are definitely in /var/log" are both incorrect. Read the project configuration and deployment instructions first.

bash

# File logs
tail -n 100 application.log
less application.log

# systemd service
journalctl -u myapp.service --since '30 minutes ago'

# Kubernetes container
kubectl logs pod/myapp --since=30m

What a Useful Log Looks Like

json

{"timestamp":"2026-06-23T14:32:15.456Z","level":"ERROR","event":"job.failed","service":"catalog-api","request_id":"req-7f2a","item_id":"item-1024","message":"item unavailable"}

At least consider these fields:

Field	Reason
Timestamp with time zone	Align across machines; typically uses UTC
Level	Control and filter detail according to team policy
Stable event name	Queries do not depend on natural-language wording
Service, version, environment	Confirm who generated the log
Request or trace ID	Link multiple logs from a single operation
Domain identifier	Locate an object, but must be minimized and comply with data policies
Result and exception	Explain what happened, and retain exception chains if necessary

Structured logging means more than emitting JSON. Field names and types must remain stable; for example, duration_ms should always be numeric.

Log Levels Have No Absolute Definition Across Projects

Common frameworks provide the following levels, but the specific meanings should be written into team guidelines:

Level	A Practical Convention	Common Noise
TRACE	Observing very short execution paths for a brief period	Often enabled in high-frequency loops
DEBUG	Targeted troubleshooting context	Outputs full objects or request bodies
INFO	Normal but worth retaining state changes	Logs every function entry
WARN	An exceptional condition occurs, but operations can continue	Alerts for every normal retry
ERROR	Current operation fails or requires explicit handling	All expected user input errors are elevated to ERROR

The level configuration is a threshold. For example, if a logger is set to INFO, it typically retains INFO and more severe events, while filtering DEBUG and TRACE. Different frameworks may also provide FATAL, CRITICAL, or custom levels.

"Fixing DEBUG in development and INFO in production" is just the starting point, not a rule. High-traffic services may need to be more restrained. During incident investigation, you can temporarily lower the log level of a specific logger and agree on a time to revert.

Hands-On: Writing File Logs with the Python Standard Library

Create an isolated directory and save the following as worker.py. It uses only the standard library; the basicConfig encoding argument requires Python 3.9+:

bash

log_workdir=$(mktemp -d)
cd "$log_workdir"

python

import logging

logging.basicConfig(
    filename="application.log",
    encoding="utf-8",
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s %(message)s",
    datefmt="%Y-%m-%dT%H:%M:%S%z",
)
logger = logging.getLogger("catalog.worker")

def process_item(item_id: str) -> None:
    logger.info("job.started item_id=%s", item_id)
    try:
        raise RuntimeError("item unavailable")
    except RuntimeError:
        logger.exception("job.failed item_id=%s", item_id)
        raise

process_item("item-1024")

bash

status=0
python3 worker.py || status=$?
printf 'status=%s\n' "$status"
tail -n 20 application.log
grep -F 'item_id=item-1024' application.log

This example writes text logs with fields; the earlier JSON shows a separate format. It has no rotation configured. grep returns lines containing the ID but does not automatically include the multiline traceback that follows, so inspect the original file for complete evidence.

logger.exception() should be called within an exception handling block. It will attach the current exception stack. The example re-raises the exception, so the process exits with a non-zero status. After saving the evidence, clean up:

bash

cd
ls -ld -- "$log_workdir"
rm -r -- "$log_workdir"

In real applications, it's important to agree on logging boundaries. If the underlying layer, service layer, and entry layer all "log and re-raise," the same failure can generate multiple duplicate ERROR messages. A common practice is to log once at a layer where it's possible to add sufficient business context and where the result is certain to be handled.

Choosing Between Output and Logging

print() or System.out are suitable for:

Outputting normal results to a command-line tool for users;
Demonstrating a single value in a teaching example;
Local short-term validation of control flow.

They cannot alone provide a unified level, context, fields, routing, and retention strategy. Production applications should generate manageable records through a logging framework; the framework can still send these records to standard output for collection by a container platform.

What Should Not Be Logged

Handle these fields according to their sensitivity. Omit passwords, tokens, and keys by default; use a reviewed sanitization scheme only for identifiers that must be retained. Hashing or encryption does not automatically make sensitive data suitable for logging:

Passwords, access tokens, API keys, and encryption keys;
Cookies, session identifiers, and database connection strings;
Unnecessary personal information, health data, and customer data;
Full request bodies, payment data, and source code;
Unprocessed user input containing newlines or control characters.

Don't rely on developers to remember this each time. Prioritize uniform filtering in logging wrappers, serializers, or data collection pipelines, and ensure comprehensive test coverage.

Rotation, Retention, and Permissions

When writing logs to disk, at least the following questions must be answered:

Should logs be split by size or by time?
How many copies and for how many days should they be retained?
When should compression and deletion occur?
Who alerts when the disk is near its capacity limit?
Which accounts are permitted to read or modify the logs?

Rotation is responsible for splitting files, while the retention policy determines the retention period. Logs cannot be deleted before the compliance deadline, nor should they be retained indefinitely. When sending logs to a remote location, the transmission channel must be protected.

Logs, Metrics, and Tracing

Signal	What It Can Answer
Logs	What happened during a specific event
Metrics	How error rates, latency, and throughput change
Tracing	Which components a single request went through and how long each part took

Injecting trace_id, span_id, or a request ID into logs can retrace the sequence of events for a single operation. However, a custom request ID alone does not equate to a complete trace.

Before Continuing

Run the Python example to verify that file logs, exception stacks, and non-zero exit codes are correctly recorded;
Write a structured record containing timestamp, level, event name, service name, and request ID;
List at least five categories of data that should not be logged verbatim;
Explain why containers often write logs to standard output;
Clarify the difference between rotation and retention strategies.

References

The data is now saved and linked. Next, you need to safely pass the error to a search tool or another developer, ensuring that the other person can actually reproduce it.

→ Next Step: Secure Search and Minimal Reproduction

Chapter 1: Propositional Logic: From Requirement Sentences to Decidable Formulas

Chapter 2: Sets and Operations: Describing Object Scope via Membership

Chapter 3: Direct Proof, Contrapositive, and Case Analysis: Building Verifiable Reasoning Chains

Chapter 4: Recursive Definitions and Mathematical Induction: Proving Along the Construction Process

Chapter 5: Binary Relations: Properties, Composition, and Closure

Chapter 6: Counting Principles: Addition, Multiplication, Inclusion-Exclusion, and the Pigeonhole Principle

Chapter 7: Graph Models and Traversal: First Define Edges, Then Choose Representation

Chapter 8: Probability Space and Conditional Probability: First Build the Model, Then Update Your Judgment

Chapter 9: Divisibility, Congruence, and Euclid's Algorithm: The Fundamental Structure of Integers

Chapter 10: Vectors and Inner Products: Turning Objects into Comparable Coordinates

Chapter 11: Limits, Derivatives, and the Chain Rule: Describing Local Change

Chapter 12: Self-Information, Entropy, and Coding: How Uncertainty Becomes Bits

6.2 Leave Searchable Logs Behind

Evidence Must Survive the Terminal

Logs, Output Streams, and Storage Locations

What a Useful Log Looks Like

Log Levels Have No Absolute Definition Across Projects

Hands-On: Writing File Logs with the Python Standard Library

Choosing Between Output and Logging

What Should Not Be Logged

Rotation, Retention, and Permissions

Logs, Metrics, and Tracing

Before Continuing

References