Writing HTCondor DAG Files: Job Dependencies, Variables, and Concurrency Limits
Part 4 of 6 in Building and Operating an HTCondor Pool for Research Computing. Running multiple HTCondor jobs becomes more complicated when one task depends on the results of another. A preprocessing job may need to finish before analysis can begin, and a final aggregation job may need to wait for several analyses to succeed. HTCondor's Directed Acyclic Graph Manager (DAGMan) lets you define…
1. A directed acyclic graph (DAG) is a structure consisting of nodes and directed edges with no cycles. In HTCondor, a node represents a unit of work associated with a submit description, which may execute one or more jobs. Dependencies between nodes are represented by edges, and a DAG file declares these nodes and dependencies using JOB, PARENT, and CHILD directives.
HTCondor's Directed Acyclic Graph Manager (DAGMan) submits nodes' jobs when dependencies allow and monitors them for completion. A child node becomes eligible for submission once all its parent nodes have succeeded.
2. To pass a result from one job (A) to another (B), create a directory and save three files within it. job A will echo "sample-001" and save it to A.out, while job B will read A.out and print the same ID. Configure both jobs to run on different machines, with A transferring its output file to B's working directory after completing. Submit the DAG using condor_submit_dag, and verify the output verification process by comparing A.out and B.out files.
3. Define nodes within a DAG using the JOB directive, specifying the node name and submit file. Node names must be unique within a DAG and can be simple names containing letters, digits, and underscores. The submit file for a node must contain a single queue statement. If a node should be retried independently, give it a separate node.
Dependencies between nodes are expressed using the PARENT and CHILD directives. Multiple parents and children can be declared on the same line, allowing for complex dependency structures like diamond-shaped DAGs.
4. Use the VARS directive to pass per-node parameters across multiple DAG nodes using a single submit description. In the provided example, RUN1 and RUN2 nodes share the same submit description but are differentiated by idx and alpha parameters. Each job receives these values as submit macros, which can be accessed within the job's command-line arguments. By including the run index and HTCondor cluster ID in file names, each job retains its own output and error files, even if a retry occurs.
5. To create a log directory for the DAG, use the tail command to display the last line of the DAGMan log file (my.dag.dagman.out) and verify the output verification process by comparing A.out and B.out files.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
