# AgentForge — Bring Your Own Open Weights to Vertex AI Model Garden (Managed Serving & Global Serverless Tuning, Zero GKE)

> **Disclaimer:** Independent personal open-source reference project—not affiliated with, endorsed by, or supported by Google LLC, Alibaba Cloud, Hugging Face, or any third-party vendor. Provided **"AS IS"** without warranty under Apache License 2.0, and users are solely responsible for all Google Cloud runtime, storage, and GPU endpoint billing charges ([`DISCLAIMER.md`](https://github.com/mbettan/AgentForge/blob/main/DISCLAIMER.md)). All timing and dollar figures are **illustrative order-of-magnitude estimates only (not official pricing)**.

## Summary & Scope

- **Product / Repo**: `AgentForge` (`https://github.com/mbettan/AgentForge`)
- **Live Documentation**: `https://mbettan.github.io/AgentForge/`
- **Core Capability**: End-to-end reference pipeline and one-command Google Cloud Shell runner (`./deploy/run_cloud_shell.sh`) for importing Hugging Face `.safetensors` checkpoints (`Qwen/Qwen3-4B`, Llama 3.1/3.3, Gemma 3, Mistral, DeepSeek) from Google Cloud Storage into **Vertex AI Model Garden** (`vertexai.preview.model_garden.CustomModel`), serving them on **Managed Dedicated Endpoints** (`vLLM` / `SGLang`, zero GKE clusters), customizing weights with overridden hyperparameters via **Global Serverless Open Model Tuning** (`google-genai` SDK with `location="global"`), and evaluating accuracy via both the **OpenAI Chat Completions SDK** and `endpoint.predict(..., use_dedicated_endpoint=True)`.

## 4-Stage Automated Pipeline & Expected Wait Times (Estimates)

1. **Stage 01 — Stage Open Weights & Validated JSONL (`~3–5 min`)**:
   - `src/generate_dataset.py` builds deterministic Cymbal enterprise incident triage JSONL splits (`120` train, `12` val, `8` test prompts; `val.jsonl` kept `<= 25%` of `train.jsonl` byte size at `10.4%`).
   - `src/stage_hf_weights.py` downloads `Qwen/Qwen3-4B` (`11` files, `~8.06 GB`) from Hugging Face Hub and stages `.safetensors` + JSONL splits to `gs://<YOUR_PROJECT_ID>-mg-byom-demo`.
2. **Stage 02 — Deploy As-Is `CustomModel` & Benchmark Baseline (`~9–15 min` deploy + `~2–3 min` eval)**:
   - `src/deploy_custom_weights.py --machine-type=auto` calls `CustomModel.list_deploy_options(available_machines=True, filter_by_user_quota=True)` to auto-select available regional GPU hardware (e.g., `g4-standard-48` with `1x NVIDIA_RTX_PRO_6000` deploys in `~9–15 min`).
   - `src/evaluate_endpoint.py` evaluates `8` triage prompts via OpenAI `/v1beta1/.../chat/completions` and `endpoint.predict(use_dedicated_endpoint=True)` (untuned baseline: `0 / 8` / `0.0%` exact schema match, `17.50s` avg latency).
3. **Stage 03 — Tear Down Base Endpoint & Run Global Serverless Fine-Tuning (`~40 min–3.5 hours` wall-clock wait)**:
   - Tears down the temporary base GPU endpoint first to avoid `~$13` in idle GPU replica charges during tuning.
   - `src/finetune_open_model.py` defaults to `--location="global" --sdk="google_genai"` (`genai.Client(vertexai=True, project=PROJECT_ID, location="global").tunings.tune(...)`) to route across supported multi-GPU regions and avoid single-region `code: 8` capacity exhaustion (measured run: routed to `us-west1` in `0.7s`, completed in `41m 15s`).
   - Overrides default hyperparameters (`FULL` mode, `epochs=5`, `learning_rate=5e-5` vs. defaults `epochs=3`, `learning_rate=7.5e-5`) directly on `custom_base_model="gs://.../hf-weights/Qwen--Qwen3-4B"`.
   - Exports portable Hugging Face `.safetensors` shards + tokenizer files to `{output_uri}/postprocess/node-0/checkpoints/final` (`100%` open weights, zero vendor lock-in).
4. **Stage 04 — Re-Deploy Tuned `.safetensors`, Compare & Auto-Cleanup (`~12–16 min`)**:
   - Deploys the exported `.safetensors` checkpoint via `CustomModel.deploy()`, re-runs the 8-prompt evaluation suite, generates `results/comparison_report.md` (`src/compare_results.py`), and deletes active GPU endpoints (`src/cleanup.py` or `./deploy/run_cloud_shell.sh --cleanup`).
   - Measured result on `g4-standard-48`: average latency drops `17.50s → 3.28s` (`-81.3%`) on the compact 120-row × 5-epoch tutorial split (`~5` multi-GPU optimizer steps) by suppressing verbose `<think>` chains; strict 5-field `[CYMBAL-TRIAGE]` tag memorization (`0 / 8` on the 5-step toy split) requires `50–100+` optimizer steps (`~1,000–2,500` rows or `epochs=20–30`).

## Illustrative Cost Model (Not Official Pricing)

- **Tutorial Total (end-to-end with `--auto-cleanup`)**: **`~$0.60 – $2.20`**, with an expected wall-clock wait of **`~1 – 3.5 hours`**.
- **Serverless Open Model Tuning**: Billed per 1M training tokens (`dataset_tokens * epochs`), with `$0` charge for queue wait, pod spin-up, `.safetensors` export, or rejected capacity attempts (`Qwen 3 4B` ~`$1.35`/1M tok -> `120` rows × `5` epochs ≈ `~64K` billable tokens ≈ **`~$0.09`**; a larger `10K` rows × `500` tok × `3` epochs run ≈ `15M` tokens ≈ **`~$20.25`**).
- **Managed Dedicated GPU Endpoints**: Billed per replica-hour while deployed (`g2-standard-12` `1x L4` ~`$1.15`/hr; `g4-standard-48` `1x RTX PRO 6000` ~`$5.18`/hr -> two ~12m deploy+eval cycles with `--auto-cleanup` ≈ **`~$0.50 – $2.10`**).
- **Cloud Storage**: ~`$0.020`/GB-month for ~16 GB of base + tuned `.safetensors` ≈ **`~$0.32`/month** (~`$0.01`/day).
- **Official Pricing References**:
  - [Vertex AI Generative AI Pricing](https://cloud.google.com/vertex-ai/generative-ai/pricing)
  - [Vertex AI Pricing](https://cloud.google.com/vertex-ai/pricing)
  - [Cloud Storage Pricing](https://cloud.google.com/storage/pricing)

## Quickstart CLI (Google Cloud Shell)

```bash
git clone https://github.com/mbettan/AgentForge.git
cd AgentForge

export PROJECT_ID="<YOUR_PROJECT_ID>"
export REGION="us-central1"

# 1. Read-only dry-run: emit exact API, bucket, and bucket-scoped least-privilege IAM commands
./deploy/run_cloud_shell.sh --emit-iam

# 2. Read-only environment & project preflight check
./deploy/run_cloud_shell.sh --preflight

# 3. Execute all 4 stages end-to-end (~1-3.5 hours unattended) and delete GPU endpoints when finished
./deploy/run_cloud_shell.sh --deploy --auto-cleanup
```

## Critical Vertex AI SDK Rules & Gotchas

- Import `CustomModel` from **`vertexai.preview.model_garden`** (not `vertexai.model_garden`).
- Import `SourceModel` (when using `vertexai_sft`) from **`vertexai.tuning`** (not `vertexai.preview.tuning`).
- Pass **`--location="global"`** with **`--sdk="google_genai"`** for Serverless Open Model Tuning to avoid single-region `code: 8` capacity exhaustion.
- `machine_type`, `accelerator_type`, and `accelerator_count` on `CustomModel.deploy()` follow an **all-or-none** rule; do **not** pass `use_dedicated_endpoint=True` to `.deploy()`, only to `endpoint.predict(..., use_dedicated_endpoint=True)`.
- Grant bucket-scoped **`roles/storage.objectUser`** on `gs://<BUCKET>` to both `service-<PROJECT_NUMBER>@gcp-sa-aiplatform.iam.gserviceaccount.com` and `service-<PROJECT_NUMBER>@gcp-sa-vertex-moss-ft.iam.gserviceaccount.com` (the latter is lazily created on a project's first tuning call; `run_cloud_shell.sh` automatically binds IAM and retries).

## Key Links

- [GitHub Repository & Step-by-Step Guide](https://github.com/mbettan/AgentForge)
- [Live GitHub Pages Command Center](https://mbettan.github.io/AgentForge/)
- [Disclaimer & Cloud Billing Responsibility](https://github.com/mbettan/AgentForge/blob/main/DISCLAIMER.md)
- [Apache License 2.0](https://github.com/mbettan/AgentForge/blob/main/LICENSE)
- [Deploy Models with Custom Weights in Model Garden](https://cloud.google.com/vertex-ai/generative-ai/docs/model-garden/deploy-models-with-custom-weights)
- [Tune an Open Model on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/models/open-model-tuning)