jxmorris12

import multiprocessing

manager = multiprocessing.Manager()
all_hashes_set = manager.dict()
def deduplicate(examples, all_hashes_set):
    print(len(all_hashes_set))
    input_ids = examples['input_ids']
    hashes = [
        hash(tuple(input_ids[i]))
        for i in range(len(input_ids))

gwangjinkim

This gist I write, because I couldn't find step by step instructions 
how to install and start postgresql locally (using conda within a conda environment - with the result
that you can run it without sudo/admin rights on your machine!)
and not globally in the operating system (which requires sudo/admin rights on that machine).

I hope, this will help especially people new to postgresql (and those who don't have sudo/admin rights on a specific machine but want
to run postgresql there)!

####################################
# create conda environment

ericmjl

UPDATE: I have baked the ideas in this file inside a Python CLI tool called pyds-cli. Please find it here: https://github.com/ericmjl/pyds-cli

How to organize your Python data science project

Having done a number of data projects over the years, and having seen a number of them up on GitHub, I've come to see that there's a wide range in terms of how "readable" a project is. I'd like to share some practices that I have come to adopt in my projects, which I hope will bring some organization to your projects.

Disclaimer: I'm hoping nobody takes this to be "the definitive guide" to organizing a data project; rather, I hope you, the reader, find useful tips that you can adapt to your own projects.

Disclaimer 2: What I’m writing below is primarily geared towards Python language users. Some ideas may be transferable to other languages; others may not be so. Please feel free to remix whatever you see here!

pksunkara

# vi: ft=dosini
[user]
	name = Pavan Kumar Sunkara
	email = pavan.sss1991@gmail.com
	username = pksunkara
[core]
	editor = nvim
	whitespace = fix,-indent-with-non-tab,trailing-space,cr-at-eol
	pager = delta
[column]