Skip to content

Instantly share code, notes, and snippets.

@shhommychon

shhommychon/Download-FMA-Dataset-with-expensive-Colab-Pro-and-Google-One-2TB.ipynb

Last active August 4, 2022 06:52
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save shhommychon/cd40110379dd7efcc0ce65c9b62ce946 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save shhommychon/cd40110379dd7efcc0ce65c9b62ce946 to your computer and use it in GitHub Desktop.
Download ZIP
Download BIG files to Colab and upload to Google Drive FS
Display the source blob
Display the rendered blob
Raw
Download-FMA-Dataset-with-expensive-Colab-Pro-and-Google-One-2TB.ipynb
{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Download-FMA-Dataset-with-expensive-Colab-Pro-and-Google-One-2TB.ipynb","provenance":[],"collapsed_sections":[],"toc_visible":true,"machine_shape":"hm","mount_file_id":"1wB58cCz1vvK8shEyspwd1AqBJGqtUm4a","authorship_tag":"ABX9TyMxa2S6V15yfkK1bDZeH/cM"},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"MyColabBadge"},"source":["※ 이 창에서 열지 말고 새 탭에서 열어주세요: ", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/FeetCodingHommy/cd40110379dd7efcc0ce65c9b62ce946)"]},{"cell_type":"markdown","metadata":{"id":"m_bl2Y_SnWPt"},"source":["# 코랩 환경 확인\n"," * 코랩 끊김 방지 ([출처](https://teddylee777.github.io/colab/google-colab-%EB%9F%B0%ED%83%80%EC%9E%84-%EC%97%B0%EA%B2%B0%EB%81%8A%EA%B9%80%EB%B0%A9%EC%A7%80))"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":17},"id":"u_yzkztnhy1G","executionInfo":{"status":"ok","timestamp":1619058922939,"user_tz":-540,"elapsed":2535,"user":{"displayName":"Bruh","photoUrl":"https://lh4.googleusercontent.com/-B27YxbbbhoY/AAAAAAAAAAI/AAAAAAAAAas/v9L6B4sJXt4/s64/photo.jpg","userId":"00000000000000000000"}},"outputId":"e242c9b7-5f2f-4179-933f-1af631c0887a"},"source":["# 아래 코드를 크롬 F12 콘솔에 실행\n","\n","%%javascript\n","// function ClickConnect(){\n","// console.log(\"7분!\"); \n","// document.querySelector(\"colab-toolbar-button\").click() \n","// }\n","// setInterval(ClickConnect, 7 * 60 * 1000)\n","console.log(\"download and copy to Google Drive\")"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"application/javascript":["// function ClickConnect(){\n","// console.log(\"1분!\"); \n","// document.querySelector(\"colab-toolbar-button\").click() \n","// }\n","// setInterval(ClickConnect, 60 * 1000)\n","console.log(\"unzip tar\")"],"text/plain":["<IPython.core.display.Javascript object>"]},"metadata":{"tags":[]}}]},{"cell_type":"markdown","metadata":{"id":"AYkVIS3Xwfvf"},"source":["# 받을 파일 : FMA-dataset \n"," * [Github 링크](https://github.com/mdeff/fma)\n"," * fma_metadata.zip (342 MiB)\n"," * fma_small.zip: 8,000 tracks of 30s, 8 balanced genres (GTZAN-like) (7.2 GiB)\n"," * fma_medium.zip: 25,000 tracks of 30s, 16 unbalanced genres (22 GiB)\n"," * fma_large.zip: 106,574 tracks of 30s, 161 unbalanced genres (**93 GiB**)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"4oUdBowKbdFn","executionInfo":{"status":"ok","timestamp":1619058974397,"user_tz":-540,"elapsed":16299,"user":{"displayName":"Bruh","photoUrl":"https://lh4.googleusercontent.com/-B27YxbbbhoY/AAAAAAAAAAI/AAAAAAAAAas/v9L6B4sJXt4/s64/photo.jpg","userId":"00000000000000000000"}},"outputId":"7a42ec76-5ddb-4051-b91f-625f520b9f11"},"source":["!curl -O https://os.unil.cloud.switch.ch/fma/fma_metadata.zip\n","!echo \"f0df49ffe5f2a6008d7dc83c6915b31835dfe733 fma_metadata.zip\" | sha1sum -c -"],"execution_count":null,"outputs":[{"output_type":"stream","text":[" % Total % Received % Xferd Average Speed Time Time Time Current\n"," Dload Upload Total Spent Left Speed\n","100 341M 100 341M 0 0 24.1M 0 0:00:14 0:00:14 --:--:-- 27.8M\n","fma_metadata.zip: OK\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"UHb0Wy0OcMaF","executionInfo":{"status":"ok","timestamp":1619068959756,"user_tz":-540,"elapsed":453626,"user":{"displayName":"Bruh","photoUrl":"https://lh4.googleusercontent.com/-B27YxbbbhoY/AAAAAAAAAAI/AAAAAAAAAas/v9L6B4sJXt4/s64/photo.jpg","userId":"00000000000000000000"}},"outputId":"ce6bd456-4288-46fc-ce8f-63f2816abb74"},"source":["!curl -O https://os.unil.cloud.switch.ch/fma/fma_small.zip\n","!echo \"f0df49ffe5f2a6008d7dc83c6915b31835dfe733 fma_small.zip\" | sha1sum -c -\n","# 얘는 자꾸 FAILED라고 뜨는데 아무리 봐도 SHA코드 상 문제인거 같다.\n","# 그냥 확실히 하기 위해 깃헙에서 직접 받아서 올려야겠다 으"],"execution_count":null,"outputs":[{"output_type":"stream","text":[" % Total % Received % Xferd Average Speed Time Time Time Current\n"," Dload Upload Total Spent Left Speed\n","100 7323M 100 7323M 0 0 19.2M 0 0:06:21 0:06:21 --:--:-- 19.9M\n","fma_small.zip: FAILED\n","sha1sum: WARNING: 1 computed checksum did NOT match\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"85sG6ExJcTfU","executionInfo":{"status":"ok","timestamp":1619070409480,"user_tz":-540,"elapsed":1903342,"user":{"displayName":"Bruh","photoUrl":"https://lh4.googleusercontent.com/-B27YxbbbhoY/AAAAAAAAAAI/AAAAAAAAAas/v9L6B4sJXt4/s64/photo.jpg","userId":"00000000000000000000"}},"outputId":"ea83392a-9884-46dd-b099-e0cce31cf912"},"source":["!curl -O https://os.unil.cloud.switch.ch/fma/fma_medium.zip\n","!echo \"c67b69ea232021025fca9231fc1c7c1a063ab50b fma_medium.zip\" | sha1sum -c -"],"execution_count":null,"outputs":[{"output_type":"stream","text":[" % Total % Received % Xferd Average Speed Time Time Time Current\n"," Dload Upload Total Spent Left Speed\n","100 22.1G 100 22.1G 0 0 18.4M 0 0:20:28 0:20:28 --:--:-- 17.7M\n","fma_medium.zip: OK\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"_KlItuooe7vS","executionInfo":{"status":"ok","timestamp":1619166202720,"user_tz":-540,"elapsed":4871078,"user":{"displayName":"Bruh","photoUrl":"https://lh4.googleusercontent.com/-B27YxbbbhoY/AAAAAAAAAAI/AAAAAAAAAas/v9L6B4sJXt4/s64/photo.jpg","userId":"00000000000000000000"}},"outputId":"d3cbbf13-482b-49aa-e579-2418f821edbb"},"source":["!curl -O https://os.unil.cloud.switch.ch/fma/fma_large.zip\n","!echo \"497109f4dd721066b5ce5e5f250ec604dc78939e fma_large.zip\" | sha1sum -c -"],"execution_count":null,"outputs":[{"output_type":"stream","text":[" % Total % Received % Xferd Average Speed Time Time Time Current\n"," Dload Upload Total Spent Left Speed\n","100 93.4G 100 93.4G 0 0 24.2M 0 1:05:46 1:05:46 --:--:-- 23.5M\n","fma_large.zip: OK\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"FkpXXUMouejN","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1619173333498,"user_tz":-540,"elapsed":11999322,"user":{"displayName":"Bruh","photoUrl":"https://lh4.googleusercontent.com/-B27YxbbbhoY/AAAAAAAAAAI/AAAAAAAAAas/v9L6B4sJXt4/s64/photo.jpg","userId":"00000000000000000000"}},"outputId":"1a2324ff-9000-4b05-8055-82da0d5eeb3b"},"source":["from google.colab import drive\n","drive.mount(\"/content/drive\",force_remount=True) # Google Drive I/O 에러 방지"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Mounted at /content/drive\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"nfoaxZWHc4ZE","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1619174437795,"user_tz":-540,"elapsed":13100043,"user":{"displayName":"Bruh","photoUrl":"https://lh4.googleusercontent.com/-B27YxbbbhoY/AAAAAAAAAAI/AAAAAAAAAas/v9L6B4sJXt4/s64/photo.jpg","userId":"00000000000000000000"}},"outputId":"d2f82282-4266-4c2a-fcf6-442396f76143"},"source":["!cp /content/fma_metadata.zip \"/content/drive/MyDrive/DATA/fma/\"\n","!cp /content/fma_small.zip \"/content/drive/MyDrive/DATA/fma/\"\n","!cp /content/fma_medium.zip \"/content/drive/MyDrive/DATA/fma/\"\n","!cp /content/fma_large.zip \"/content/drive/MyDrive/DATA/fma/\""],"execution_count":null,"outputs":[{"output_type":"stream","text":["cp: error writing '/content/drive/MyDrive/DATA/fma/fma_large.zip': No space left on device\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"KGuiQlzzhL83"},"source":["## 90기가 짜리 fma_large.zip에서 자꾸 에러가 난다\n"," * Google Drive REST API를 써봐야겠다\n"," * [Document > Uploading file data](https://developers.google.com/drive/api/v3/manage-uploads)\n"," * [Colab example](https://colab.research.google.com/notebooks/io.ipynb#scrollTo=c2W5A2px3doP)"]},{"cell_type":"code","metadata":{"id":"jiR15f4mevEZ","executionInfo":{"status":"ok","timestamp":1619177197997,"user_tz":-540,"elapsed":21493,"user":{"displayName":"Bruh","photoUrl":"https://lh4.googleusercontent.com/-B27YxbbbhoY/AAAAAAAAAAI/AAAAAAAAAas/v9L6B4sJXt4/s64/photo.jpg","userId":"00000000000000000000"}}},"source":["from google.colab import auth\n","auth.authenticate_user()\n","from googleapiclient.discovery import build\n","drive_service = build(\"drive\", \"v3\")"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"lj6v0kJZjTER"},"source":[" * drive_service.files().create()의 body에 필요한 파라미터\n"," * ```name``` : 파일 이름\n"," * ```mimeType```\n"," * .zip 파일 mimeType이 뭔데 구글 10duck아...\n"," * [Mozilla Document > MIME 타입의 전체 목록](https://developer.mozilla.org/ko/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types)"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"1QoKCkMXn0MP","executionInfo":{"status":"ok","timestamp":1619179571902,"user_tz":-540,"elapsed":639,"user":{"displayName":"Bruh","photoUrl":"https://lh4.googleusercontent.com/-B27YxbbbhoY/AAAAAAAAAAI/AAAAAAAAAas/v9L6B4sJXt4/s64/photo.jpg","userId":"00000000000000000000"}},"outputId":"287d0a24-c0ca-4c38-a46d-715ba2c26a40"},"source":["from googleapiclient.http import MediaFileUpload\n","\n","file_metadata = {\n"," \"name\": \"fma_large.zip\",\n"," \"mimeType\": \"application/zip\"\n","}\n","media = MediaFileUpload(\"/content/fma_large.zip\", \n"," mimetype=\"application/zip\",\n"," resumable=True)\n","created = drive_service.files().create(body=file_metadata,\n"," media_body=media,\n"," fields=\"id\").execute()\n","print(f\"File ID: {created.get('id')}\")"],"execution_count":null,"outputs":[{"output_type":"stream","text":["File ID: 깃허브올릴땐가려야지홀롤롤롤\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"KGuiQlzzhL84"},"source":[" * 오 이 API로 해보니 개잘됨 깃헙에 올려야징"]}]}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment