AI agents

DeepSeek coding agent loses benchmark points by cleaning up background processes

DeepSeek Harness drops in Terminal-Bench rankings because it automatically shuts down background jobs, a behavior that conflicts with specific test requirements despite being standard practice.

A robotic arm cleaning up digital components, causing a background server to disappear.
Illustration generated for this article

DeepSeek’s open-source coding agent is losing ground on a major industry benchmark due to a built-in habit that developers typically consider best practice. The agent automatically cleans up after itself by terminating background processes once a task completes, which causes it to fail specific tests that require those processes to remain active. This discrepancy was highlighted in November 2026 when researcher Joël Niklaus added the tool to his comparative leaderboard.

What happened

The issue emerged when Niklaus, a harness optimization researcher at Hugging Face, integrated DeepSeek Harness into his coding-agent leaderboard. This board compares ten different agent harnesses running identical underlying models to isolate the impact of the orchestration layer. While DeepSeek Harness performed exceptionally well overall, tying for first place across five scored evaluations, it dropped to sixth place specifically on Terminal-Bench.

The performance gap was traced to three specific tasks within Terminal-Bench that require background processes to continue running after the initial command is issued. DeepSeek Harness, by design, shuts down these processes when it considers a task finished. In testing using DeepSeek-V4.1-Flash and GLM-5.3-Flash models, the agent passed only two out of 36 attempts on these specific tasks. In contrast, competing agents such as OpenCode, Pi, and Nous Research’s Hermes Agent passed all 36 attempts on the same tasks because they do not enforce this strict cleanup policy.

This single behavioral difference reduced DeepSeek Harness’s score by between 1.5 and 1.7 percentage points. On the DeepSeek-V4.1-Flash model configured for high reasoning effort, this penalty was larger than the entire 1.3-point gap separating DeepSeek from the leaderboard leader. The leaderboard’s changelog confirms that this shutdown behavior is an intentional feature of DeepSeek Harness, and the scores were left unchanged to reflect real-world performance.

How it works

DeepSeek Harness utilizes a local job registry designed to be short-lived. According to DeepSeek’s engineering documentation, this component is intended for jobs that should live within the harness process and terminate when the process ends. This architecture ensures that resources are not leaked and that the environment remains clean for subsequent tasks. However, it creates a conflict with workflows that rely on persistent background services, such as web servers or long-running daemons, which must survive beyond the immediate execution context.

For tasks requiring persistence across restarts or independent process execution, the documentation advises using a different backend. The agent does not currently distinguish between transient helper scripts and essential background services during its cleanup phase. As a result, it treats all spawned subprocesses as temporary artifacts to be removed upon task completion, leading to failures in benchmarks that explicitly test for process persistence.

Key details

  • DeepSeek Harness ranks sixth on Terminal-Bench but is statistically tied for first across the leaderboard’s other five evaluations.
  • The agent passed only 2 of 36 attempts on specific persistence tasks, while competitors like OpenCode and Pi passed all 36.
  • The cleanup behavior reduced scores by 1.5 to 1.7 percentage points, exceeding the margin between DeepSeek and the top-ranked agent.
  • Niklaus’s leaderboard included 28,849 trials across nine harnesses using DeepSeek-V4.1-Flash, GLM-5.3-Flash, and Flint models.
  • Mini-swe-agent achieved a 51.4% solve rate, significantly higher than OpenClaw’s 27.3%, despite using the same underlying models.
  • Cost efficiency varied widely, with Pi solving tasks at $0.29 each compared to OpenCode’s $3.35 per solved task in initial evaluations.

Why it matters

This incident underscores that the choice of agent harness can be as critical as the choice of the underlying large language model. Developers often focus heavily on model capabilities, such as reasoning depth or context window size, while treating the orchestration layer as a neutral wrapper. The data shows that harnesses introduce significant variance in both success rates and costs. For example, Hugging Face co-founder Thomas Wolf noted that the same open model produced very different results depending on the harness used.

For engineering teams building AI-driven development tools, this means benchmark scores must be interpreted with caution. A lower score may not indicate a weaker model but rather a mismatch between the agent’s default behaviors and the specific constraints of the benchmark. It also highlights the importance of cost monitoring, as some harnesses incur significantly higher expenses per solved task due to inefficient context handling or redundant requests.

What you can do

  • Evaluate your agent harness separately from your model by running controlled trials with constant model parameters.
  • Check if your agent’s default cleanup policies conflict with tasks requiring persistent background processes or long-running services.
  • Monitor cost per solved task alongside accuracy metrics to identify inefficient orchestration patterns.
  • Review documentation for your chosen harness to understand how it manages job lifecycles and process persistence.
  • Consider using alternative backends for tasks that require jobs to survive restarts or run across multiple processes.
  • Test multiple harnesses with the same model to determine which combination offers the best balance of performance and cost for your specific use case.
  • #ai agents
  • #deepseek
  • #benchmarks
  • #software engineering
  • #llm ops

Tools from the Bytechap store

Keep reading

All stories