jeromeku

Prompt:

I am trying to understand the lifecycle of the cuda graph memory pool, e.g., as interacted with in _cuda_releasePool. In particular, the implementation of _use_cuda_memory_pool_manager seems to suggest that the pool is released at the end of the context manager. But in the call sites of this function, it doesn't seem that the pool should be released (especially when we are allocating inputs for the cuda graph region). What is going on?

Response:

You're right to notice this apparent discrepancy. Let's break down what's happening with the CUDA graph memory pool lifecycle:

1. The _use_cuda_memory_pool_manager context manager is indeed calling _cuda_releasePool at the end of its context. However, this doesn't necessarily mean the pool is being destroyed or fully released.
2. The _cuda_releasePool function is part of the CUDA caching allocator's API, and its behavior is more nuanced than just destroying the pool:
   • It decrements the reference count for the pool.

• If the reference c

jeromeku

AIConfigurator: Fast-Track Your LLM Deployment on NVIDIA Dynamo

What is NVIDIA Dynamo?

NVIDIA Dynamo is a high-throughput, low-latency inference framework for serving generative AI models across multi-node GPU clusters. As LLMs grow beyond what a single GPU can handle, Dynamo solves the orchestration challenge of coordinating shards, routing requests, and transferring KV cache data across distributed systems.

Key capabilities:

• Disaggregated serving — Separates prefill and decode phases for optimized GPU utilization
• KV-aware routing — Routes requests to workers with the highest cache hit rate
• KV Block Manager — Offloads KV cache to CPU, SSD, or remote memory (G2/G3/G4) for higher throughput

jeromeku

import torch
import unittest
from torch import Tensor
from torch.distributed.tensor import (
    DTensor,
    DeviceMesh,
    distribute_tensor,
    init_device_mesh,
    Partial,
    Replicate,

jeromeku

A guide to George Polya's "How to Solve It"

Introduction

George Polya (1887-1985) was a Hungarian mathematician and educator who revolutionized our understanding of problem-solving and teaching. While he made significant contributions to mathematics, his most enduring legacy is his insights into how people learn and solve problems. His 1945 book "How to Solve It" has influenced educators across numerous fields, extending far beyond mathematics.

Polya's Teaching Philosophy

Polya's approach centered on three key principles. First, he championed active learning, believing students learn best by discovering solutions themselves rather than being told answers. Second, he emphasized heuristic thinking - the art of guided discovery and learning from experience, viewing problem-solving as a skill that can be developed through practice. Third, he developed sophisticated questionin

jeromeku

#include <Python.h>
#include <numpy/arrayobject.h>
#include "chi2.h"

/* Docstrings */
static char module_docstring[] =
    "This module provides an interface for calculating chi-squared using C.";
static char chi2_docstring[] =
    "Calculate the chi-squared of some data given a model.";

jeromeku

from typing import Optional, Callable, Sequence, Any

import torch
from torch import nn, fx
from torch.library import Library
import torch.nn.functional as F
import torch._inductor
import torch._inductor.compile_fx

mirage_lib = Library("mirage", "FRAGMENT")  # noqa

jeromeku

Debugging Spurious Copies: Mini Example
https://gist.github.com/matthias-springer/81748fe1e530974dd5ff6b3ad57e3eeb

Debugging Spurious Copies: Matmul, Tiled
https://gist.github.com/matthias-springer/372162baa30e79c49180bb3ace216995
https://gist.github.com/matthias-springer/b664feb23be0159f72726025923bb9ca

Empty Tensor Elimination
https://gist.github.com/matthias-springer/b3f40d1667c977c29a76cc7a469cc1a0
https://gist.github.com/matthias-springer/e531580242d27f14e0a239e0b6fe80ae

jeromeku

import math
import cutlass.cute as cute            
import cutlass

def visualize_tv_layout(
    tiler_mn: tuple[int, int],
    tv_layout,                       # (((thr_shape),(val_shape)),
                                    #  ((thr_stride),(val_stride)))
    *,
    font_size: int = 10,

jeromeku

from typing import List

import cutlass
import cutlass.cute as cute
from cutlass.cute.runtime import from_dlpack


@cute.kernel
def kernel_use_vec_as_arg(vec, res: cute.Tensor):

jeromeku

import cutlass
import cutlass.cute as cute
from cutlass.cute.runtime import from_dlpack


@cute.kernel
def kernel_use_vec_as_arg(vec, res: cute.Tensor):
    # cute.print_tensor(vec)
    res.store(vec)