Federated Heterogeneous Compute and Storage Infrastructure for PUNCH4NFDI

1. Introduction

PUNCH4NFDI (Particles, Universe, NuClei and Hadrons for the National Research Data Infrastructure) is a major German consortium funded by the DFG (Deutsche Forschungsgemeinschaft). It represents approximately 9,000 scientists from particle, astro-, astroparticle, hadron, and nuclear physics communities. The consortium's prime goal is to establish a federated, FAIR (Findable, Accessible, Interoperable, Reusable) science data platform. This contribution specifically details the architectural concepts—Compute4PUNCH and Storage4PUNCH—designed to unify access to the highly heterogeneous compute (HPC, HTC, Cloud) and storage resources contributed in-kind by member institutions across Germany.

2. Federated Heterogeneous Compute Infrastructure – Compute4PUNCH

The Compute4PUNCH initiative addresses the challenge of providing seamless access to a diverse pool of existing compute resources without imposing major changes on the operational models of resource providers.

3. Federated Storage Infrastructure – Storage4PUNCH

Storage4PUNCH aims to federate community storage systems, primarily based on dCache and XRootD technologies, which are well-established in High-Energy Physics (HEP).

4. Technical Details & Mathematical Framework

The core scheduling logic in COBalD/TARDIS can be modeled as an optimization problem. Let $R = \{r_1, r_2, ..., r_n\}$ be the set of resource requests from the HTCondor pool, and $B = \{b_1, b_2, ..., b_m\}$ be the set of available backend resource types (e.g., HPC node, Cloud VM). Each request $r_i$ has requirements (cores, memory, software). Each backend $b_j$ has a cost function $C_j(r_i)$ and a provisioning time $T_j(r_i)$.

The meta-scheduler's objective is to find a mapping $M: R \rightarrow B$ that minimizes a total cost function, often a weighted sum of financial cost and time-to-completion, subject to constraints like backend quotas and software availability:

$$\min_{M} \sum_{r_i \in R} \left[ \alpha \cdot C_{M(r_i)}(r_i) + \beta \cdot T_{M(r_i)}(r_i) \right]$$

where $\alpha$ and $\beta$ are weighting factors. This formalizes the "dynamic and transparent" integration challenge.

5. Prototype Results & Performance

The paper reports on initial experiences with scientific applications running on available prototypes. While specific quantitative benchmarks are not detailed in the provided excerpt, the successful execution implies:

• Functional Integration: The HTCondor/COBalD/TARDIS stack successfully routed jobs to different backend systems (HTC, HPC, Cloud).
• Software Delivery: CVMFS and containers reliably provided the necessary software environments across heterogeneous worker nodes.
• User Access: JupyterHub and login nodes served as effective entry points for researchers.

Conceptual Diagram: The system architecture can be visualized as a three-layer model:

1. User Access Layer: JupyterHub, Login Nodes, Token AAI.
2. Federation & Scheduling Layer: HTCondor Pool + COBalD/TARDIS Meta-scheduler.
3. Resource Layer: Heterogeneous backends (HPC clusters, HTC farms, Cloud VMs) and federated storage (dCache, XRootD instances).

6. Analysis Framework: A Use Case Scenario

Scenario: A nuclear physics researcher needs to process 10,000 Monte Carlo simulation tasks, each requiring 4 CPU cores, 16 GB RAM, and a specific software stack (Geant4, ROOT).

1. Submission: The researcher logs into the PUNCH JupyterHub, writes an analysis script, and submits 10,000 jobs to the local HTCondor scheduler.
2. Meta-Scheduling: COBalD/TARDIS monitors the HTCondor queue. It evaluates available backends: University A's HTC farm (low cost, high queue time), Institute B's HPC cluster (moderate cost, specialized hardware), and a commercial cloud (high cost, immediate availability).
3. Decision & Execution: Using its cost model, TARDIS might decide to burst 2,000 immediate jobs to the cloud to start quickly, while steadily draining the rest on the cheaper HTC farm. It uses the token AAI for authentication on all systems.
4. Software & Data: Each job, regardless of backend, pulls its Geant4/ROOT environment from CVMFS. Input data is fetched from the federated Storage4PUNCH namespace (e.g., via XRootD), and output is written back to a designated storage endpoint.
5. Completion: The researcher monitors and aggregates results from the single HTCondor job queue, oblivious to the underlying multi-infrastructure execution.

7. Critical Analysis & Expert Perspective

Core Insight: PUNCH4NFDI isn't building another cloud; it's engineering a federation layer of remarkable political and technical pragmatism. Its true innovation lies in the COBalD/TARDIS meta-scheduler, which acts as a "diplomatic translator" for resource sharing, not a conquering unifier. This acknowledges the sovereignty of existing institutional clusters—a non-negotiable reality in German academia—while still creating a functional supra-resource.

Logical Flow: The logic is impeccable: start with the user (JupyterHub/login), abstract the chaos via a battle-tested scheduler (HTCondor), then use a smart broker (TARDIS) to map abstract requests onto concrete, politically feasible backends. The reliance on CVMFS and containers for software is a masterstroke, solving the "dependency hell" problem that plagues most federations. The storage strategy is wisely conservative, building on the proven dCache/XRootD duo from HEP, avoiding the quagmire of trying to force a single new technology.

Strengths & Flaws:

• Strengths: Minimal invasion is its superpower. It doesn't require providers to change their local policies. The use of mature, community-driven tools (HTCondor, CVMFS, dCache) drastically reduces risk and increases sustainability, unlike projects built on bespoke frameworks. The focus on FAIR principles aligns perfectly with modern funding mandates.
• Flaws & Risks: The meta-scheduler approach introduces a single point of complexity and potential failure. COBalD/TARDIS, while promising, is not as battle-hardened as the other components. The "evaluation" of caching/metadata tech (like Rucio) hints at the hardest part being ahead: intelligent data management. Without it, this is a compute federation with a attached storage directory, not a cohesive data-centric platform. There's also a lurking risk of performance unpredictability for users, as their jobs hop between fundamentally different architectures.

Actionable Insights:

1. For PUNCH Architects: Double down on making TARDIS robust and observable. Its metrics and decision logs are gold for optimization and trust-building. Prioritize the integration of a data management layer (like Rucio) next; compute without smart data is half a solution.
2. For Other Consortia: This is a blueprint worth emulating, especially the "integration over replacement" philosophy. However, assess if your community has an equivalent to CVMFS—if not, that's your first build/buy decision.
3. For Resource Providers: This model is low-risk for you. Engage with it. The token-based AAI is a clean way to offer access without compromising local security. It's a net gain for visibility and utilization.

8. Future Applications & Development Roadmap

The PUNCH4NFDI infrastructure lays the groundwork for several advanced applications and research directions:

• Cross-Domain Workflows: Enabling complex, multi-step analysis pipelines that seamlessly move between simulation (HPC), high-throughput event processing (HTC), and machine learning training (Cloud GPUs).
• Data-Centric Scheduling: Integrating the storage federation more deeply with the compute scheduler. Future versions of COBald/TARDIS could factor data locality (minimizing WAN transfers) and pre-staging into its cost function, moving towards data-aware scheduling.
• Integration with FAIR Data Repositories: Serving as the high-performance compute backbone for national FAIR data repositories, allowing researchers to analyze large datasets directly where they are stored, following the "compute-to-data" paradigm.
• AI/ML as a Service: The JupyterHub interface and scalable backend could be extended with curated environments for specialized AI/ML frameworks (PyTorch, TensorFlow) and access to GPU resources, democratizing AI for the physical sciences.
• Expansion to International Resources: The federation model could be extended to incorporate resources from European initiatives like the European Open Science Cloud (EOSC) or LHC computing grid (WLCG) sites, creating a truly pan-European research infrastructure.

The roadmap likely involves hardening the current prototype, scaling the number of integrated resources, implementing the evaluated metadata/caching solutions, and developing more sophisticated policy and accounting mechanisms for fair-share resource usage across the consortium.

9. References

1. PUNCH4NFDI Consortium. (2024). PUNCH4NFDI White Paper. [Internal Consortium Document].
2. Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency and computation: practice and experience, 17(2-4), 323-356.
3. Blomer, J., et al. (2011). The CernVM file system. Journal of Physics: Conference Series, 331(5), 052004.
4. COBalD/TARDIS Documentation. (n.d.). Retrieved from https://tardis.readthedocs.io/
5. dCache Collaboration. (n.d.). dCache: A distributed storage system. https://www.dcache.org/
6. XRootD Collaboration. (n.d.). XRootD: High performance, scalable fault tolerant access to data. http://xrootd.org/
7. Wilkinson, M. D., et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific data, 3(1), 1-9.
8. European Open Science Cloud (EOSC). (n.d.). https://eosc-portal.eu/
9. Worldwide LHC Computing Grid (WLCG). (n.d.). https://wlcg.web.cern.ch/