6.4 Comprehensive Troubleshooting Exercise

Prerequisites: Pages 1-3 of this chapter

Estimated time: 30 minutes

Exercise environment: Bash, Python 3.9+; Java section requires JDK

You are ready to continue when: you can save exit codes and two output streams, and write a minimal troubleshooting log

The previous lessons analyzed existing failures. This lab provides a crashing Python script and a Java class. Preserve the evidence first, then run experiments that change one variable at a time.

All outputs are placed into a temporary directory, ensuring that logs and class files are not mixed into the project:

workdir=$(mktemp -d)
cd "$workdir"

Warm-up 1: Python Runtime Exceptions

Save as crash.py:

values = [10, 20, 30]
print(values[5])

status=0
python3 crash.py >stdout.log 2>stderr.log || status=$?
printf 'status=%s\n' "$status"
sed -n '1,20p' stdout.log
sed -n '1,20p' stderr.log

Record:

• Exception type and message;
• Filename and line number;
• What is the difference between stdout.log and stderr.log;
• Is the exit code 0?

Then, only change values[5] to values[2], and rerun with the same set of commands. Compare the four pieces of evidence, don't just look at the terminal colors.

Warm-up 2: Java Compilation and Runtime Stages

Save a program that compiles but fails at runtime:

CrashMe.java

public class CrashMe {
    public static void main(String[] args) {
        int[] values = {10, 20, 30};
        System.out.println(values[5]);
    }
}

compile_status=0
javac CrashMe.java >compile.stdout.log 2>compile.stderr.log || compile_status=$?
run_status=not-run
if [ "$compile_status" -eq 0 ]; then
  run_status=0
  java CrashMe >run.stdout.log 2>run.stderr.log || run_status=$?
fi
printf 'compile_status=%s\n' "$compile_status"
printf 'run_status=%s\n' "$run_status"
sed -n '1,20p' compile.stderr.log
if [ "$run_status" != not-run ]; then
  sed -n '1,20p' run.stderr.log
fi

Now, intentionally remove one semicolon. First, remove the old output, and only run the compiler:

rm -- CrashMe.class
compile_status=0
javac CrashMe.java >compile.stdout.log 2>compile.stderr.log || compile_status=$?
printf 'compile_status=%s\n' "$compile_status"
sed -n '1,20p' compile.stderr.log
test ! -e CrashMe.class

Comparison: Does a compilation error have a Java runtime stack? Does the directory still contain a runnable CrashMe.class? Real scripts should also check the compilation exit code first. You shouldn't continue running the previous class if the compilation fails.

Challenge: Logs, Exceptions, and Duplicate Logging

Use worker.py from “Log Basics”:

1. Run it once and save the exit code, stderr, and application.log;
2. Identify how many times the same exception occurred;
3. Catch and log again around the call to process_item(), then observe the duplicate logs;
4. Remove one layer of logging, leaving only the one with sufficient context;
5. Add a fictional request_id=req-7f2a to the logs, use grep -F to align the same operation.

The goal is one complete, correlated, secret-free record of the failure, not the largest possible pile of ERROR lines.

Troubleshooting Case 1: NoClassDefFoundError

When starting a Java service, the following error is reported:

java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonFactory

This error occurs during the runtime phase when the JVM needs the class definition. Investigation steps:

Issue	Check Method
Were the dependencies resolved	mvn dependency:tree
Is the scope appropriate for runtime	Check provided, test, and exclusions
What classpath was used in the startup command	Check the script, IDE run configuration, or container entrypoint
Is the class present in the runtime artifacts	jar tf target/myapp.jar; inspect external dependencies separately for a regular JAR, or nested JARs under BOOT-INF/lib for Spring Boot
Was the executable or fat JAR explicitly built	Check Spring Boot, Shade, or other plugin configurations
Has the class failed initialization previously	Look earlier for ExceptionInInitializerError and the original exception

Two common misconceptions:

• The default regular JAR produced by mvn package typically does not include dependency classes; simply switching from compile to package does not guarantee that all classes are present at runtime.
• dependency:tree indicates that Maven resolved the dependency, but it does not prove that the startup command has placed it into the runtime classpath.

Preserve the complete exception chain and startup command before changing scopes or packaging plugins. A missing class is not a reason to add several arbitrary versions.

Troubleshooting Case 2: Can't Find Log File

The configuration specifies a relative path ./logs/app.log, which is visible locally, but not found inside the container.

Relative paths are relative to the process's working directory, not the source code directory. First, check the deployment method:

# Regular Linux process; replace PID with the actual value
readlink -f /proc/PID/cwd

# systemd service
systemctl show myapp.service -p WorkingDirectory
journalctl -u myapp.service --since '30 minutes ago'

# Kubernetes
kubectl logs pod/myapp --since=30m

Container applications may not write files at all, but instead pass logs to standard output for collection. Confirm the configuration before searching for the path; don't blindly find from the root of the file system, as that is both slow and may cross permission boundaries.

Troubleshooting Case 3: Seems Like the Same Import Error

Error	What to Check First
Python ModuleNotFoundError	Current interpreter, virtual environment, package name and import name, installation location
Java ClassNotFoundException	Classpath when actively loading a class by name
Java NoClassDefFoundError	Things visible at compile time but not available at runtime, missing dependencies, or initialization failures

These errors cannot be simply attributed to "missing dependencies." First, confirm which interpreter or JVM is actually being used, and what path is being used in the startup command.

Minimum Troubleshooting Log

Fill out the following after completing any case:

Symptoms:
First occurrence time and time zone:
Impact scope:
Version and environment:
Complete reproduction command:
Expected result:
Actual result and exit code:
Related logs / request_id / trace_id:
Current hypothesis:
Evidence supporting this hypothesis:
Next experiment that could invalidate it:

Acceptance

• [ ] Can separately save stdout, stderr, and exit codes;
• [ ] Can explain the difference between compilation failures and runtime exceptions;
• [ ] Can identify types, messages, and related frames in the Java or Python stack;
• [ ] Can explain that ordinary Maven JARs do not automatically include runtime dependencies;
• [ ] Can query a set of logs using a request ID, and the logs do not contain real secrets;
• [ ] Can deliver the minimal reproducible case containing environment, command, expected, and actual results.

Keep the necessary troubleshooting records before cleaning, and delete after confirming the temporary directory:

cd
ls -ld -- "$workdir"
rm -r -- "$workdir"

Keep the reproduction command, both output streams, and exit code, then compare the results before and after the fix. Next, put the program in a Docker container and examine what determines its runtime environment and which resources are isolated.

→ Next: Docker Basics