r2 - 06 Dec 2018 - 15:05:50 - LaurentDuflotYou are here: TWiki >  Atlas Web  >  Informatique > GuideAnalyse > UsingLGD


How to save your data on LOCALGROUPDISK

When running jobs on grid, output files are stored on SCRATCHDISK area of the site where jobs ran, and are erased typically after two weeks. You can retrieve your files locally using rucio, but tou may want to save them on LOCALGROUPDISK areas which are Tier3 and so managed by end-users. You have access to two such LOCALGROUPDISK areas :

  • on CCIN2P3 site, called IN2P3-CC_LOCALGROUPDISK

There are several ways to get your data on these sites :

If your data are not yet on grid it is possible to upload them, following these instructions. An example :

  • Create a dataset with this command line : rucio add-dataset mydataset
  • upload the content of the directory mydir with : rucio upload --rse GRIF-LPNHE_LOCALGROUPDISK mydataset mydir
  • check that files can be retrieved by : rucio download mydataset

How to access your data from LOCALGROUPDISK

In case your data are located on grid (e.g LOCALGROUPDISK) you can access them directly from almost any server using xrootd protocol. Use the following command to get the list of files (with their path) from a given dataset:

  • rucio list-file-replicas --rse LOCALGROUPDISK --protocol root mydataset

This will give you sthing like :

  • /lpnse1.in2p3.fr:1094//dpm/in2p3.fr/home/atlas/atlaslocalgroupdisk/rucio/user/derue/5b/71/user.derue.13584469._000001.output.root

Special case at CC-IN2P3: rucio returns the xrootd access for external sites which is throttled, please replace ccxrdatlas by ccrootdatlas as server name in order to use the internal xrootd servers at CC

To create a list with only the path and names, without the other informations, you may adapt this commande line:

  • rucio list-file-replicas --rse GRIF-LPNHE_LOCALGROUPDISK --protocol root user.derue.410470.PhPy8EG.DAOD_TOPQ5.e6337_s3126_r9364_p3629.21248-ttjpsi-EMTopo-1d_output.root | awk '{if  ($2 == "user.derue") {print $12}}' >& list-410470-FS-TOPQ5-MC16a.txt

To use these files:

  • from ROOT:
    TFile *f = TFile::Open("root://lpnse1.in2p3.fr:1094//dpm/in2p3.fr/home/atlas/atlaslocalgroupdisk/rucio/user/derue/5b/71/user.derue.13584469._000001.output.root");
  • locally:
    xrdcp root ://lpnse1.in2p3.fr:1094//dpm/in2p3.fr/home/atlas/atlaslocalgroupdisk/rucio/user/derue/5b/71/user.derue.13584469._000001.output.root mylocalfile
  • from prun:
    Instead of giving --inDS myDataset option, provide it with --pfnList mylist.txt where the filelist mylist.txt contains the file above.

-- FredericDERUE - 02 Dec 2018

Edit | WYSIWYG | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r2 < r1 | More topic actions
Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback