Gregg Donovan greggdonovan

Shell scripting with Markdown documents

alias ~~~=":<<'~~~sh'";:<<'~~~sh'

Solr needs opinions, because Solr documentation gives you way too many options. It's hard to navigate the best practices for using Solr. Some of my opinions based on dozens of Solr projects :)

Prefer preloading static, classic config files over managed schema, config API, or schemaless

Schema files are a good thing. They are declarative, and not letting them change at runtime prevents all kinds of security issues. Further, classic schema / solrconfig support all of Solr's functionality and are well documented with tons of examples online in blog articles and stackoverflow. Using managed schema or the config API takes a lot of experimentation.

Static configurations can also be easily version controlled. As I've learned as a long time Elasticsearch user, this is one of Solr's advantages. Having an API for changing every underlying config option of your index means finding the code that made the change is rather time consuming.

Static configuration is also good separation of concerns. You cleanly separate

Profiling performance

Using perf:

$ perf record -g binary
$ perf script | stackcollapse-perf.pl | rust-unmangle | flamegraph.pl > flame.svg

NOTE: See @GabrielMajeri's comments below about the -g option.

I've had many people ask me questions about OpenTracing, often in relation to OpenZipkin. I've seen assertions about how it is vendor neutral and is the lock-in cure. This post is not a sanctioned, polished or otherwise muted view, rather what I personally think about what it is and is not, and what it helps and does not help with. Scroll to the very end if this is too long. Feel free to add a comment if I made any factual mistakes or you just want to add a comment.

So, what is OpenTracing?

OpenTracing is documentation and library interfaces for distributed tracing instrumentation. To be "OpenTracing" requires bundling its interfaces in your work, so that others can use it to time distributed operations with the same library.

So, who is it for?

OpenTracing interfaces are targeted to authors of instrumentation libraries, and those who want to collaborate with traces created by them. Ex something started a trace somewhere and I add a notable event to that trace. Structure logging was recently added to O

Latency numbers every programmer should know

L1 cache reference ......................... 0.5 ns
Branch mispredict ............................ 5 ns
L2 cache reference ........................... 7 ns
Mutex lock/unlock ........................... 25 ns
Main memory reference ...................... 100 ns             
Compress 1K bytes with Zippy ............. 3,000 ns  =   3 µs
Send 2K bytes over 1 Gbps network ....... 20,000 ns  =  20 µs
SSD random read ........................ 150,000 ns  = 150 µs

Read 1 MB sequentially from memory ..... 250,000 ns = 250 µs

	tiltfile(
	name = "example",
	images = {
	"//example:image": "example/image",
	},
	k8s_objects = [
	"//example:k8s",
	],
	template = "Tiltfile.template",
	)

	import scala.util.Try

	object PathSerializer {

	trait SerDe[A] {
	// By using a path dependent type, we can be sure can deserialize without wrapping in Try
	type Serialized
	def ser(a: A): Serialized
	def deser(s: Serialized): A
	// If we convert to a generic type, in this case String, we forget if we can really deserialize

	#!/bin/bash

	# runs a command in the same network namespaces as a process pid

	set -e

	function usage {
	test -n "$" & echo "$"
	echo "usage: $0 PID COMMAND [ARG...]" >&2
	exit 1

	""" Trains an agent with (stochastic) Policy Gradients on Pong. Uses OpenAI Gym. """
	import numpy as np
	import cPickle as pickle
	import gym

	# hyperparameters
	H = 200 # number of hidden layer neurons
	batch_size = 10 # every how many episodes to do a param update?
	learning_rate = 1e-4
	gamma = 0.99 # discount factor for reward

	/**
	git clone https://github.com/twitter/scalding.git
	cd scalding
	./sbt scalding-repl/console
	*/

	import scala.io.Source
	val alice = Source.fromURL("http://www.gutenberg.org/files/11/11.txt").getLines
	// Add the line numbers, which we might want later
	val aliceLineNum = alice.zipWithIndex.toList