Skip to content

Instantly share code, notes, and snippets.

View pepijnve's full-sized avatar

Pepijn Van Eeckhoudt pepijnve

  • Belgium
  • 15:18 (UTC +01:00)
View GitHub Profile
@pepijnve
pepijnve / bench_sh.patch
Created June 11, 2025 10:51
Datafusion benchmark script patch
diff --git a/benchmarks/bench.sh b/benchmarks/bench.sh
index b34c646c5..d8742c25f 100755
--- a/benchmarks/bench.sh
+++ b/benchmarks/bench.sh
@@ -43,6 +43,7 @@ DATA_DIR=${DATA_DIR:-$SCRIPT_DIR/data}
CARGO_COMMAND=${CARGO_COMMAND:-"cargo run --release"}
PREFER_HASH_JOIN=${PREFER_HASH_JOIN:-true}
VIRTUAL_ENV=${VIRTUAL_ENV:-$SCRIPT_DIR/venv}
+ITERATIONS=50
```
Index: datafusion/core/tests/execution/infinite_cancel.rs
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/datafusion/core/tests/execution/infinite_cancel.rs b/datafusion/core/tests/execution/infinite_cancel.rs
--- a/datafusion/core/tests/execution/infinite_cancel.rs (revision 287b2a9e9eb7f779d7bca99306a6b76d58965d92)
+++ b/datafusion/core/tests/execution/infinite_cancel.rs (date 1749487490790)
@@ -220,11 +220,8 @@
Comparing baseline and cancel_safety
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ baseline ┃ cancel_safety ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │ 3952.81ms │ 3948.19ms │ no change │
│ QQuery 1 │ 1926.73ms │ 2075.42ms │ 1.08x slower │
│ QQuery 2 │ 3949.43ms │ 3803.91ms │ no change │
Comparing baseline and cancel_safety
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ baseline ┃ cancel_safety ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │ 3949.63ms │ 4009.24ms │ no change │
│ QQuery 1 │ 1943.87ms │ 1945.33ms │ no change │
│ QQuery 2 │ 3901.66ms │ 3874.57ms │ no change │
@pepijnve
pepijnve / infinite_test.rs
Created May 28, 2025 14:51
Infinitely running DataFusion query
use arrow::array::{Int64Array, RecordBatch};
use arrow::datatypes::{DataType, Field, Fields, Schema, SchemaRef};
use datafusion::execution::{RecordBatchStream, SendableRecordBatchStream, TaskContext};
use datafusion::functions_aggregate::sum;
use datafusion::physical_expr::aggregate::AggregateExprBuilder;
use datafusion::physical_expr::{EquivalenceProperties, Partitioning};
use datafusion::physical_plan::aggregates::{AggregateExec, AggregateMode, PhysicalGroupBy};
use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
use datafusion::physical_plan::stream::RecordBatchReceiverStreamBuilder;
use datafusion::physical_plan::{DisplayAs, DisplayFormatType, ExecutionPlan, PlanProperties};
@pepijnve
pepijnve / gist:8f0e5c2a399c0e1fe1f2ea06b230427a
Last active October 7, 2020 16:10
Asciidoc Pikchr example
= Doc Title
This is an example of embedding Pikchr in asciidoc with live preview
.A silly little Pikchr diagram
[pikchr]
----
arrow
box "foo"
down
@pepijnve
pepijnve / gist:2d4059db5d827462b5cd
Created August 22, 2014 19:51
Updated Cacoo implementation
require 'json'
require 'time'
require_relative '../api/diagram'
module Asciidoctor
module Diagram
# @private
module Cacoo
def self.get_diagram_metadata(diagram_id, api_key)
# NOTE: See API document at https://cacoo.com/lang/en/api and
@pepijnve
pepijnve / proguard.rb
Last active December 16, 2015 05:48
Buildr extension that adds support for obfuscation using Proguard
require 'buildr'
module Buildr
class ProguardTask < Rake::FileTask
# Proguard version number.
VERSION = '4.7'
OPTIONS = [:configuration,
:target,
:forceprocessing,