Introduction to Kubeflow

01-Installation.md

Installation of Kubeflow Pipelines

The best guide to install Kubeflow or any of its components is the official documentation.

https://www.kubeflow.org/docs/started/

In this tutorial, we will learn how to Install, Configure and Use the most popular component of Kubeflow that is KFP (Kubeflow Pipelines).

Step 1: Install Docker Desktop

Follow the below guide and follow the steps according to your OS Distribution.

https://docs.docker.com/desktop/setup/install/mac-install/

Step 2: Install KIND

Easy way to learn Kubeflow Pipelines is to setup a KIND k8s cluster and configure Kubeflow Pipelines on the cluster.

https://kind.sigs.k8s.io/docs/user/quick-start/

Step 3: Install KFP in Kubernertes

Follow the steps to setup KFP.

https://www.kubeflow.org/docs/components/pipelines/operator-guides/installation/

Step 4: Install Python kfp package

1. Create a folder - mkdir kfp
2. Create a Python virtual environment - python3 -m venv .kfp
3. Source the virtual environment - source .kfp/bin/activate
4. Install the package - pip3 install kfp==2.9.0

02-hello_pipeline.py

from kfp import dsl
from kfp import compiler

# Define a simple component using a Python function
@dsl.component
def say_hello(name: str) -> str:
    """A simple component that says hello to a given name."""
    hello_text = f'Hello, {name}!'
    print(hello_text)
    return hello_text

# Define the pipeline using the @dsl.pipeline decorator
@dsl.pipeline(
    name="hello-world-pipeline",
    description="A basic pipeline that prints a greeting."
)
def hello_pipeline(recipient: str = "World") -> str:  # Add a default value
    """This pipeline runs the say_hello component."""
    hello_task = say_hello(name=recipient)
    return hello_task.output  # Return the output of the component

if __name__ == "__main__":
    # Compile the pipeline into a YAML file
    compiler.Compiler().compile(hello_pipeline, 'hello_world_pipeline.yaml')

    # To run the pipeline, you would typically use the Kubeflow Pipelines UI
    # or the KFP SDK Client if you have a running Kubeflow Pipelines backend:
    # from kfp.client import Client
    # client = Client(host='<YOUR-KFP-ENDPOINT>')  # Replace with your endpoint
    # run = client.create_run_from_pipeline_func(hello_pipeline, arguments={'recipient': 'Kubeflow'})
    # print(f"Pipeline run details: {run}")

03-irish_pipeline.py

import kfp
from kfp import dsl
from typing import NamedTuple, List

# Step 1: Load data into memory and return as lists
@dsl.component(
    base_image="python:3.8-slim",
    packages_to_install=["pandas", "scikit-learn"]
)
def load_data() -> NamedTuple("Outputs", [("features", List[List[float]]), ("labels", List[int])]):
    from sklearn.datasets import load_iris
    iris = load_iris()
    return (iris.data.tolist(), iris.target.tolist())

# Step 2: Train model and return accuracy
@dsl.component(
    base_image="python:3.8-slim",
    packages_to_install=["scikit-learn"]
)
def train_model(
    features: List[List[float]],
    labels: List[int]
) -> NamedTuple("Output", [("accuracy", float)]):
    from sklearn.ensemble import RandomForestClassifier
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import accuracy_score

    X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.2)

    clf = RandomForestClassifier()
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    acc = accuracy_score(y_test, y_pred)

    print(f"Model accuracy: {acc}")
    return (acc,)

# Step 3: Define the pipeline
@dsl.pipeline(
    name="iris-no-artifacts-pipeline",
    description="ML pipeline without file artifacts, returns accuracy."
)
def iris_pipeline():
    data = load_data()
    train_model(
        features=data.outputs["features"],
        labels=data.outputs["labels"]
    )

# Step 4: Compile
if __name__ == "__main__":
    kfp.compiler.Compiler().compile(
        pipeline_func=iris_pipeline,
        package_path="iris_pipeline.yaml"
    )

NAME READY STATUS RESTARTS AGE
cache-deployer-deployment-568c969444-tbml4 1/1 Running 0 125m
cache-server-864749847f-btcdg 1/1 Running 0 126m
controller-manager-9bc5fd5df-6j5rd 1/1 Running 0 111m
metadata-envoy-deployment-5c97bdbd97-4tcr5 1/1 Running 0 126m
metadata-grpc-deployment-7cd994d449-dm9sw 1/1 Running 7 (119m ago) 126m
metadata-writer-689cbbfc88-p9zcv 1/1 Running 0 126m
minio-6bdf7c6c9b-22qsl 1/1 Running 0 4m38s
ml-pipeline-d94cb5f45-wfrc6 1/1 Running 0 3m26s
ml-pipeline-persistenceagent-5d5b784486-86mm8 1/1 Running 18 (10m ago) 126m
ml-pipeline-scheduledworkflow-b4f959f59-sb44d 1/1 Running 0 2m39s
ml-pipeline-ui-5dcfff6ddd-7r78r 1/1 Running 0 126m
ml-pipeline-viewer-crd-7cdd4745b8-szwgv 1/1 Running 0 126m
ml-pipeline-visualizationserver-b5c797b95-wrnt2 1/1 Running 0 126m
mysql-5bb7cf8b95-kdtj2 1/1 Running 0 126m
workflow-controller-5b497bfffd-g7r2t 1/1 Running 0 126m

my all pod is running fine, need to understand the error. (sone pods need resources, and minio image need to be change,)