Skip to content

Instantly share code, notes, and snippets.

@aofarrel

aofarrel/batch_download_by_shard_from_terra.py

Created October 28, 2022 19:13
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save aofarrel/f013cee20423f996445c3b865be8b029 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save aofarrel/f013cee20423f996445c3b865be8b029 to your computer and use it in GitHub Desktop.
Download ZIP
Download files per shard from Terra.
Raw
batch_download_by_shard_from_terra.py
import os
vcfs = []
bgas = []
diffs = []
for vcf in vcfs:
for bga in bgas:
for diff in diffs:
split = diff.split("/")
shard = split[-3:][0]
os.system(f"mkdir {shard}")
os.system(f"cd {shard}")
os.system(f"gsutil cp {diff} .")
os.system(f"gsutil cp {bga} .")
os.system(f"gsutil cp {vcf} .")
os.system("cd ..")
exit(0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment