S3_Storage_Observability

s3_storage_observability.txt

# S3 on OpenStack NERC, migrated from minio on 2025-06-13 (minio started Jan 15 2025)
## 2025-09-23 03:00pm
### Bucket: acm-metrics
Total Objects: 2512
   Total Size: 477.3 GiB

### Bucket: acm-metrics-hypershift2
Total Objects: 1339
   Total Size: 14.8 GiB

### Bucket: acm-metrics-test
Total Objects: 1057
   Total Size: 20.0 GiB

### Bucket: loki-logs-test
Total Objects: 0
   Total Size: 0 Bytes

### Bucket: open-telemetry
Total Objects: 0
   Total Size: 0 Bytes

### Bucket: open-telemetry-test
Total Objects: 0
   Total Size: 0 Bytes

#!/bin/bash

ENDPOINT="https://stack.nerc.mghpcc.org:13808"

echo "| Bucket | Objects | Size (GiB) |"
echo "|---------------------------|---------|------------|"

for bucket in acm-metrics acm-metrics-hypershift2 acm-metrics-test open-telemetry open-telemetry-test; do
result=$(aws s3 ls "s3://$bucket/" --recursive --summarize --endpoint-url "$ENDPOINT" 2>&1 | tail -2)
objects=$(echo "$result" | grep "Total Objects" | awk '{print $3}')
bytes=$(echo "$result" | grep "Total Size" | awk '{print $3}')
gib=$(echo "scale=2; ${bytes:-0}/1024/1024/1024" | bc)
printf "| %-25s | %7s | %10s |\n" "$bucket" "${objects:-0}" "$gib"
done

Bucket	Objects	Size (GiB)
acm-metrics	3610	708.58
acm-metrics-hypershift2	2169	52.24
acm-metrics-test	1057	19.95
open-telemetry	0	0
open-telemetry-test	0	0

loki logs running into timeout (5min+, too many small files?)

aws s3 ls --endpoint-url https://stack.nerc.mghpcc.org:13808
List all S3 buckets

Erstellt	Bucket
2025-06-13	acm-metrics
2025-08-06	acm-metrics-hypershift2
2025-07-02	acm-metrics-test
2025-06-18	loki-logs
2025-07-02	loki-logs-test
2025-06-26	open-telemetry
2025-07-02	open-telemetry-test

better skript

#!/bin/bash

BUCKET="${1:-acm-metrics}"
DAYS="${2:-30}"
ENDPOINT="https://stack.nerc.mghpcc.org:13808"

echo "Fetching listing for $BUCKET..."
LISTING=$(aws s3 ls "s3://$BUCKET/" --recursive --endpoint-url "$ENDPOINT" 2>&1)

echo ""
echo "| Date | Objects | Size (MiB) | Size (GiB) |"
echo "|------------|---------|------------|------------|"

for i in $(seq 0 $DAYS); do
DAY=$(date -d "-$i day" +%Y-%m-%d)
echo "$LISTING" | grep "$DAY" | awk -v d="$DAY" '
BEGIN{sum=0; count=0}
{sum += $3; count++}
END {
mib = sum/1024/1024
gib = sum/1024/1024/1024
printf "| %s | %7d | %10.2f | %10.2f |\n", d, count, mib, gib
}'
done

echo ""
echo "TOTAL:"
echo "$LISTING" | awk '
BEGIN{sum=0; count=0}
{sum += $3; count++}
END {
printf "Objects: %d, Size: %.2f GiB\n", count, sum/1024/1024/1024
}'