Compute4PUNCH & Storage4PUNCH: Tsarin Kayayyakin Haɗin Kai na PUNCH4NFDI

1. Gabatarwa & Bayyani

Ƙungiyar PUNCH4NFDI (Barbashi, Sararin Samaniya, Tsakiya da Hadrons don Tsarin Bayanan Bincike na Ƙasa), wadda Gidauniyar Binciken Jamus (DFG) ta ba da kuɗi, tana wakiltar kimanin masana kimiyya 9,000 daga ƙungiyoyin kimiyyar barbashi, taurari, barbashin taurari, hadron, da makaman nukiliya a Jamus. Babban manufarta ita ce kafa dandalin bayanan kimiyya na haɗin kai, mai bin ƙa'idodin FAIR (Ana iya Samu, Ana iya Kaiwa, Ana iya Haɗawa, Ana iya Sake Amfani da su). Babban ƙalubalen da ake magana da shi shi ne haɗawa cikin sauƙi da samun dama ga ɗimbin albarkatun lissafi (HPC, HTC, Girgije) da ajiya waɗanda cibiyoyin membobi a duk faɗin Jamus suka bayar. Wannan takarda ta yi cikakken bayani game da ra'ayoyin Compute4PUNCH da Storage4PUNCH waɗanda aka ƙera don shawo kan waɗannan ƙalubalen haɗawa.

2. Tsarin Lissafi na Haɗin Kai (Compute4PUNCH)

Compute4PUNCH yana nufin ƙirƙirar tsarin aiki na haɗin kai a cikin ƙasa baki ɗaya, yana ba da damar samun dama ga albarkatun lissafi daban-daban ba tare da sanya canje-canje masu mahimmanci ga tsarin aiki na yanzu, waɗanda ƙungiyoyi da yawa ke rabawa ba.

2.1 Tsarin Tsakiya & Abubuwan Haɗawa

An gina tsarin a kusa da tsarin aiki na haɗin kai na HTCondor. Mai tsara albarkatun COBalD/TARDIS yana haɗa albarkatu daban-daban (tarukan HPC, gonakin HTC, misalan girgije) cikin wannan tafki ɗaya ta hanyar motsa jiki. Hanyoyin shiga ga masu amfani sun haɗa da nodes na shiga na gargajiya da sabis na JupyterHub, suna ba da hanyoyin sadarwa masu sassauƙa ga duk yanayin albarkatun.

2.2 Shiga & Tabbatar da Ainihi (AAI)

Tsarin Tabbatar da Ainihi da Izinin (AAI) wanda ya dogara da alamar yana ba da ingantaccen shiga mai aminci a duk albarkatun haɗin kai, yana sauƙaƙa ƙwarewar mai amfani da haɓaka tsaro.

2.3 Samar da Yanayin Software

Don sarrafa buƙatun software daban-daban, tsarin yana amfani da fasahohin kwantena (misali, Docker, Singularity/Apptainer) da Tsarin Fayil na Na'ura Mai Kwakwalwa na CERN (CVMFS). CVMFS yana ba da damar isar da tarin software na musamman na al'umma da bayanan gwaji a cikin rarraba, yana tabbatar da daidaito da rage nauyin ajiya na gida akan nodes na lissafi.

3. Tsarin Ajiya na Haɗin Kai (Storage4PUNCH)

Storage4PUNCH yana mai da hankali kan haɗa tsarin ajiya da al'umma suka samar, musamman bisa fasahohin dCache da XRootD, waɗanda suka kafu sosai a cikin Kimiyyar Makamashi Mai Girma (HEP).

3.1 Fasahar Haɗin Ajiya

Haɗin kai yana ƙirƙirar sararin suna guda ɗaya, yana ba masu amfani damar samun dama ga bayanai a cikin tsarin ajiya na cibiyoyi da yawa kamar dai albarkatu guda ɗaya ne. Wannan yana amfani da ƙa'idodi da ra'ayoyin da aka tabbatar a cikin manyan haɗin gwiwa kamar Grid na Lissafi na Duniya na LHC (WLCG).

3.2 Dabarun Ajiyar Bayanai da Metadata

Aikin yana kimanta fasahohin da ake da su don ajiyar bayanai mai hankali da sarrafa metadata. Manufar ita ce haɗawa mai zurfi don inganta sanya bayanai, rage jinkiri, da inganta gano bayanai bisa ƙa'idodin FAIR.

4. Aiwar Fasaha & Cikakkun Bayanai

4.1 Tsarin Lissafi don Tsara Albarkatun

Za a iya tunanin mai tsarawa na COBalD/TARDIS a matsayin magance matsalar ingantawa. Bari $R = \{r_1, r_2, ..., r_n\}$ ya zama tarin albarkatu daban-daban, kowannensu yana da siffofi kamar tsarin gine-gine, ƙwayoyin da ake da su, ƙwaƙwalwar ajiya, da farashi. Bari $J = \{j_1, j_2, ..., j_m\}$ ya zama tarin ayyuka tare da buƙatu. Mai tsarawa yana nufin haɓaka aikin amfani $U$ (misali, gabaɗayan aiki, adalci) bisa ga ƙuntatawa:

$$\text{Ƙara } U(\text{Rarraba}(R, J))$$

$$\text{bisa ga: } \forall r_i \in R, \text{Amfani}(r_i) \leq \text{Ƙarfin ɗauka}(r_i)$$

$$\text{kuma } \forall j_k \in J, \text{Buƙatu}(j_k) \subseteq \text{Siffofi}(\text{Albarkacin da aka Sanya}(j_k))$$

Wannan hanya mai motsi, wacce ke bin manufa, tana da sassauƙa fiye da tsarin jerin gwano na gargajiya.

4.2 Sakamakon Ƙirar Farko & Aiki

Ƙirar farko sun yi nasarar nuna haɗin albarkatu daga cibiyoyi kamar KIT, DESY, da Jami'ar Bielefeld. Manyan ma'auni na aikin da aka lura sun haɗa da:

Jinkirin Gabatar da Aiki: Tsarin rufi yana ƙara ƙaramin nauyi, tare da gabatar da aiki zuwa tafkin HTCondor na tsakiya yawanci ƙasa da dakika 2.
Amfani da Albarkatu: Tafkin motsa jiki wanda TARDIS ya ba da damar ya nuna yuwuwar haɓakar amfani da albarkatu gabaɗaya ta hanyar cika "rataye" a cikin jadawalin gungu na kowane mutum.
Samun Bayanai ta CVMFS: Lokutan farawa na software daga CVMFS sun yi kama da na shigarwa na gida bayan ajiyar farko, yana tabbatar da amfani da shi don rarraba software mai faɗi.
Ƙwarewar Mai Amfani: Ra'ayoyin farko sun nuna cewa mu'amalar JupyterHub da AAI mai alamar sun rage matakin shiga ga masu amfani da ba su saba da tsarin aiki na umarni ba.

Lura: Cikakkun ma'auni na ƙididdiga da ke kwatanta aikin haɗin kai da keɓaɓɓen aiki wani ɓangare ne na aikin da ke gudana.

5. Tsarin Bincike & Nazarin Lamari

Nazarin Lamari: Binciken Taurari Mai Yawan Saƙo

Yi la'akari da masanin kimiyyar barbashin taurari yana nazarin abin da ya faru na fashewar gamma-ray. Aikin yana haɗawa da:

Gano Bayanai: Yin amfani da sararin suna na ajiya na haɗin kai don gano bayanan da suka dace daga gamma-ray (Fermi-LAT), na gani (LSST), da ma'ajin igiyar nauyi (LIGO/Virgo), duk ana iya kaiwa ta hanyar hanya ɗaya (misali, /punche/data/events/GRB221009A).
Gabatar da Aiki: Mai binciken yana amfani da tashar JupyterHub don tsara rubutun bincike mai matakai da yawa. Rubutun ya ƙayyade buƙatun duka na sarrafa hoto mai saurin GPU (don bayanan gani) da ayyukan CPU masu ƙwaƙwalwar ajiya (don daidaita siffar siffa).
Aiki Mai Motsi: Haɗin kai na Compute4PUNCH, ta hanyar COBalD/TARDIS, yana tura aikin GPU zuwa gungun jami'a mai samar da nodes na V100/A100 da aikin ƙwaƙwalwar ajiya zuwa cibiyar HPC mai manyan nodes na ƙwaƙwalwar ajiya, ba tare da sa hannun mai amfani ba.
Yanayin Software: Duk ayyukan suna jawo yanayi mai daidaitaccen kwantena tare da kayan aikin taurari na musamman (misali, Astropy, Gammapy) daga CVMFS.
Haɗa Sakamako: Ana sake rubuta sakamakon tsaka-tsaki zuwa ajiyar haɗin kai, kuma ana ƙirƙira zane-zane na ƙarshe, duk ana sarrafa su a cikin zaman tabbatar da ainihi ɗaya.

Wannan lamarin yana nuna yadda haɗin kai ya ɓoye rikitaccen tsarin kayayyaki, yana barin masanin kimiyya ya mai da hankali kan matsalar kimiyya.

6. Bincike Mai Zurfi & Ra'ayi na Masana'antu

Fahimta ta Tsakiya: PUNCH4NFDI ba ta gina wani babban girgije ba; tana injiniyan Layer na Haɗin Kai—"meta-operating system" don rarraba kayayyakin bincike na ƙasa, masu cin gashin kansu. Wannan amsa ce mai ƙarfi da ƙarfi ga rarrabuwar yanayin e-kimiyya na Turai, tana ba da fifiko ga haɗawa maimakon maye gurbin. Yana kwatanta falsafar gine-ginen da ke bayan manyan tsarin nasara kamar Kubernetes don kade-kade na kwantena, amma an yi amfani da shi a matakin cibiyoyin bayanai gabaɗaya.

Kwararar Ma'ana: Ma'anar ba ta da aibi: 1) Amince da bambance-bambance da saka hannun jari na yanzu a matsayin ƙuntatawa maras canzawa. 2) Gabatar da ƙaramin Layer na abstraction mara kutsawa (HTCondor + TARDIS) don lissafi, da haɗin sararin suna don ajiya. 3) Yi amfani da tsaka-tsakin tsaka-tsaki, wanda al'umma ke tafiyar da shi (CVMFS, dCache, XRootD) a matsayin tubalan gini don tabbatar da kwanciyar hankali da amfani da ƙwarewar da ake da su. 4) Bayar da mafi zamani, hanyoyin shiga masu daidaita mai amfani (JupyterHub, alamar AAI). Wannan kwararar tana rage rikici na siyasa da fasaha ga masu samar da albarkatu, wanda ke da mahimmanci don karɓa.

Ƙarfi & Kurakurai: Babban ƙarfin aikin shine sake amfani da fasahohin balagaggu daga al'ummar HEP, yana rage haɗarin ci gaba. Mayar da hankali kan rufi mara kutsawa yana da hikima ta siyasa. Duk da haka, hanyar tana ɗaukar bashi na fasaha. Rikitarwar gyara matsalolin aiki ko gazawa a cikin yankuna gudanarwa masu zaman kansu da yawa, manufofin cibiyar sadarwa daban-daban, da masu tsarawa masu yawa (na gida + haɗin kai) zai zama mai ƙarfi—ƙalubalen da aka rubuta sosai a cikin wallafe-wallafen lissafi na grid. Dogaro akan HTCondor, duk da yake da ƙarfi, bazai zama mafi kyau ga duk tsarin aikin HPC ba, yana iya barin aiki akan tebur don ayyukan MPI masu haɗaka. Bugu da ƙari, yayin da takardar ta ambaci ƙa'idodin bayanan FAIR, aiwatar da cikakkun katalojin metadata na haɗin al'umma—ƙalubali mai girma—da alama an jinkirta shi zuwa kimantawa na gaba.

Fahimta Mai Aiki: Ga sauran ƙungiyoyi, abin da ya fi mahimmanci shine dabarun "rufi-farko". Kafin ƙoƙarin gina ko ba da umarnin kayan aiki na gama gari, saka hannun jari a cikin manne software. Tarin PUNCH4NFDI (HTCondor/TARDIS + CVMFS + Ajiyar Haɗin Kai) yana wakiltar kayan aikin buɗe ido masu jan hankali don ƙaddamar da binciken girgije na ƙasa. Duk da haka, dole ne su saka hannun jari da gaske a cikin kayan aikin lura na yanki daban-dabanSLURM mai mai da hankali kan HPC ko masu tsarawa na asalin girgije don faɗaɗa aikace-aikacen fiye da HTC. Nasarar wannan haɗin kai za a auna shi ba ta hanyar kololuwar flops ba, amma ta hanyar rage "lokacin zuwa fahimta" ga masana kimiyya 9,000.

7. Aikace-aikacen Gaba & Taswirar Ci Gaba

Kayayyakin PUNCH4NFDI sun kafa tushe don aikace-aikace masu ci gaba da yawa:

Horar da AI/ML a Girma: Tafkin albarkatun haɗin kai na iya samar da tarukan nodes na GPU a hankali don horar da manyan samfura akan bayanan kimiyya masu rarraba, bin tsarin kama da waɗanda aka bincika ta ma'auni na MLPerf HPC.
Nazari Mai Mu'amala & Na Ainihi: Ƙarfafa goyon baya ga zaman mu'amala da sabis masu haɗawa da rafukan bayanai na ainihi daga na'urorin hangen nesa ko na'urori masu gano barbashi, suna ba da damar "rayuwa" na nazarin bayanan lura.
Koyo na Haɗin Kai don Bayanai Masu Sirri: Za a iya daidaita kayayyakin don tallafawa ayyukan koyo na haɗin kai masu kiyaye sirri, inda ake horar da samfuran AI a cikin cibiyoyi da yawa ba tare da raba ɗanyen bayanai ba—dabarar da ke samun karbuwa a cikin hoton likita da sauran fannoni.
Haɗawa da Girgije na Buɗe Kimiyya na Turai (EOSC): Yin aiki a matsayin babban node na ƙasa mai ƙarfi, haɗin kai na PUNCH4NFDI zai iya ba da damar shiga cikin sauƙi ga sabis na EOSC da albarkatu, da kuma akasin haka, yana ƙara tasirinsa.
Ayyukan Haɗin Kai na Quantum: Yayin da gwaje-gwajen lissafi na quantum suka zama samuwa, haɗin kai zai iya tsara ayyukan pre-/post-processing na gargajiya tare da ayyukan co-processor na quantum, yana sarrafa dukan aikin haɗin kai.

Taswirar ci gaba za ta mai da hankali ne kan ƙarfafa sabis na samarwa, faɗaɗa tafkin albarkatu, aiwatar da manufofin sarrafa bayanai masu ci gaba, da zurfafa haɗin kai tsakanin Layer na lissafi da ajiya.

8. Nassoshi

Ƙungiyar PUNCH4NFDI. (2024). Takardar Fari ta PUNCH4NFDI. [Takardar Ƙungiya ta Ciki].
Thain, D., Tannenbaum, T., & Livny, M. (2005). Lissafi mai rarraba a aikace: ƙwarewar Condor. Haɗin kai da Lissafi: Ƙwarewa da Ƙwarewa, 17(2-4), 323-356. https://doi.org/10.1002/cpe.938
Blomer, J., et al. (2011). Tsarin Fayil na CernVM. Journal of Physics: Taron Taro, 331(5), 052004. https://doi.org/10.1088/1742-6596/331/5/052004
Fuhrmann, P., & Gulzow, V. (2006). dCache, tsarin ajiya na adadi mai yawa na bayanai. Taron IEEE na 22 akan Tsarin Ajiya da Fasaha (MSST'05). https://doi.org/10.1109/MSST.2005.47
Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Fassarar Hotuna zuwa Hotuna mara Haɗaɗɗu ta amfani da Cibiyoyin Adawa na Haɗin kai. Proceedings of the IEEE International Conference on Computer Vision (ICCV). (An ambata a matsayin misali na rikitaccen algorithm, mai buƙatar albarkatu mai yawa wanda ke motsa buƙatar lissafi).
Ƙungiyar MLCommons. (2023). Ma'auni na MLPerf HPC. https://mlcommons.org/benchmarks/hpc/ (An ambata a matsayin nassi don ayyukan AI/ML akan tsarin HPC).
Hukumar Turai. (2024). Girgije na Buɗe Kimiyya na Turai (EOSC). https://eosc-portal.eu/