Picture a company whose customer-facing document assistant runs on a server in a back office. Usage is climbing, the hardware is getting old, and someone suggests moving it to Google Cloud.
That suggestion hides two decisions. The first is where the workload should run. The second is who plans and carries out the move. Answering one doesn’t settle the other.
For AI workloads, the first decision has a twist that ordinary server migrations don’t. An inference workload is bursty. It sits idle overnight, then spikes when customers log in. The NIST definition of cloud computing lists rapid elasticity and measured service among cloud’s five essential characteristics, and those two matter most when traffic swings like this. They only save money if someone actually scales the GPU down when nobody’s using it.
All prices here are in USD and were checked in November 2026.
The idle GPU is the number to watch
Google lists one NVIDIA T4 (16 GB of GDDR6) at $0.35 an hour on demand in Iowa (us-central1). That’s the GPU alone, before the host VM, storage, and networking. Here’s how the monthly GPU bill changes depending on how you run it, assuming Google’s standard 730-hour month:
| How the T4 runs | Hourly rate | Monthly GPU cost |
| Always on, on demand | $0.35 | $255.50 |
| Always on, 1-year commitment | $0.22 | $160.60 |
| Always on, 3-year commitment | $0.16 | $116.80 |
| Business hours only (≈176 hrs), on demand | $0.35 | $61.60 |
A document assistant that only gets used during the workday costs about a quarter as much on demand if someone shuts the GPU off at night. Commitments cut the hourly rate, but they assume you’ll keep the GPU busy for a year or three.
On-premises, the idle-hour problem looks different. A GPU you already own costs the same whether it answers one request or a million. Its cost was paid up front, and power, cooling and the eventual replacement keep coming either way.
There’s also a third option that skips GPU hours entirely. Managed model endpoints usually bill per input and output token at each model’s rate, so idle time costs nothing. You give up control over which model answers and how it’s tuned.
What AI hosting really costs
A GPU hourly rate and a server purchase price describe different slices of the bill. Put one-time and recurring costs on the same planning period, such as 36 months, and keep in mind that a different horizon can flip the result.
| Cost line | Google Cloud | On-premises |
| Compute or hardware | VM and GPU hours, or managed model usage | Hardware over its remaining life, plus replacement |
| Storage and backups | Storage, snapshots and retention | Disks, backup media or service, an off-site copy |
| Data movement | Outbound transfer, VPN or interconnect | Office bandwidth and firewall |
| Software | Licenses, monitoring, model API fees | Licenses, virtualization software, monitoring |
| Staff time | Configuration, cost review, access, patching | Patching, repairs, access, on-call work |
| Support | Support plan or partner retainer | Warranties and vendor contracts |
| Migration and overlap | Assessment, dry runs, running both environments | None if staying; overlap costs if replacing |
Google’s Quick TCO Estimator in Migration Center accepts manually entered infrastructure totals or an RVTools export from VMware. Treat its output as a starting point for the infrastructure lines, then add application, staffing, migration, and support costs yourself.
Two practical traps:
- Trial credits: eligible new users get $300 in free credit for 90 days, but trial accounts can’t add GPUs to VM instances. A GPU pilot needs a paid billing account first.
- Budgets: a standard alerts-only budget emails you when spend crosses a threshold but doesn’t stop usage. Google now offers spend-cap budgets for some services, so check whether yours qualifies. Either way, name the person who shuts down idle capacity.
These per-hour economics sit inside a much bigger story. Hyperscalers’ AI infrastructure spending is under investor scrutiny, and GPU pricing and availability can move with it. Recheck rates before you sign any commitment.
Three ways to run the same assistant
The document assistant could call a managed model endpoint, run an open model on a cloud GPU, or run that same model on local hardware. Treat these as three separate options, because different models can give different answers to the same question.
Keep the task, quality bar, traffic pattern and test period identical across all three, and score each on four things:
- Answer quality: a fixed set of representative questions, graded pass or fail against a rubric your team writes.
- Cost per accepted answer: using only the cost rows each option actually incurs.
- Response time: under realistic traffic, including your busiest expected period.
- Recovery: after a forced restart, a dropped connection or a full disk.
Write down the assumptions behind each result. The T4’s 16 GB doesn’t guarantee your chosen model and input lengths will fit, and concurrent requests change how much capacity you need.
Managed AI can also run locally in specialized cases. On October 20, 2026, Google announced a preview of enterprise search with Gemini on Google Distributed Cloud air-gapped, where prompts, documents, and embeddings stay inside a disconnected network. It’s built for regulated sectors rather than the typical small business, but it shows the cloud-or-local line is blurring.
Security comes down to who owns each control
Under Google’s shared responsibility model, Google secures the underlying infrastructure and customers secure their data and access. The exact split varies by product. On-premises, your organization owns all of it, including physical access, firmware, virtualization software, and failed hardware.
Ask the same five questions of every option:
- Who approves and revokes access, including service accounts and API keys?
- Who patches the operating system, runtime, and model-serving software?
- Who protects backups, tests restores, and controls where copies live?
- Who reviews logs, and how fast does an anomaly reach a person?
- Who declares an incident, tells customers and documents the fix?
Managed services shift some of this work, never all of it. Even on a managed database like Cloud SQL, customers still handle access, authentication, and authorization.
AI adds a data question that ordinary apps don’t raise as sharply. Google’s terms for its Gemini Enterprise Agent Platform (formerly Vertex AI) say customer data won’t be used to train or fine-tune models without the customer’s permission or instruction. That’s a training commitment, not a promise of zero retention. For prompts, uploaded documents, embeddings, logs, and backups, record how long each is kept, where it lives, who can read it, and how it gets deleted. Embeddings are numerical representations of your content, so they belong in that review too.
DIY or partner-assisted migration?
If the pilot points to the cloud, assign the move. Google frames migration as assessment, planning, the move itself, and optimization, and each stage needs a named owner.
| Task | DIY | Partner-assisted |
| Discovery and dependency mapping | Your engineers document what connects to what | Partner runs an assessment and returns an inventory |
| Target configuration | You build projects, networks, permissions and storage | Partner builds them and hands over the configuration code |
| Testing | You script a rehearsal and validate data | Partner runs agreed dry-run migrations and reports results |
| Cutover | A named internal owner with rollback authority | Shared execution, with your organization approving the move |
| Documentation | Written as you go | Knowledge transfer and runbooks defined in the contract |
For outside help, Promevo’s Google cloud migration service covers dependency mapping, infrastructure as code, dry-run test migrations, and onboarding your team after cutover. Infrastructure as code keeps environment settings in reviewable, reusable files, which makes the handoff far easier to own afterward. Confirm deliverables and exclusions in any proposed scope.
How to choose:
- DIY fits when the application is well understood, the scope is bounded, engineers have time, and one person owns rollback. Retiring unneeded servers first can shrink the job.
- Partner-assisted fits when skills or capacity are thin, dependencies are unclear, or a failed cutover would hit customers. Weigh the engagement cost against the risk it removes.
- Co-managed keeps ownership in-house and buys help for selected stages, such as assessment and rehearsal. Write down the boundaries so no task falls between the two teams.
From pilot to cutover
Each step should produce evidence that the workload is ready for the next one:
- Inventory the application and its dependencies, including scheduled jobs, credentials, and external services.
- Classify sensitive data, current access, and the controls the target must enforce.
- Record baseline answer quality, response time, and cost using the pilot scorecard.
- Build an isolated pilot with no production traffic or shared credentials.
- Validate data, outputs and behavior against the baseline.
- Rehearse cutover and rollback end to end, timing every step.
- Approve the production window in writing, and name who can call it off.
- Hand over operating instructions, remove temporary access, and monitor against the baseline.
Fix the scorecard thresholds before the pilot starts, and apply them to every option. Write the rollback trigger down in advance, such as a specific validation failure or a time limit, so nobody debates criteria mid-incident.
Check the data, not just the app. Row counts, document checksums, and sample retrieval queries show whether the assistant moved intact. After cutover, keep watching: AI pipelines are among the automations that fail without warning, and a retrieval index that quietly stopped updating won’t throw an error.
Making the call
The right placement is whichever one your evidence supports. Record what each option needs before approval:
| Placement | Evidence to record before approval |
| Google Cloud | Pilot quality and response times, full cost estimate including idle GPU hours, budget alerts, named security owners |
| On-premises | Measured capacity, maintenance coverage, tested recovery, a replacement plan, any local-processing rule |
| Selective placement | Component ownership, data transfer costs, access boundaries, behavior when the connection drops |
Before cutover, attach the acceptance results, open issues, the rollback trigger and the names of the people authorized to proceed or stop.
FAQs
Q.1 Is Google Cloud cheaper than on-premises for AI?
Not automatically. It depends on traffic patterns, idle GPU hours, staff time and how much life your current hardware has left. Compare full costs over the same planning period.
Q.2 Can I test GPUs on the Google Cloud free trial?
No. Trial accounts can’t add GPUs to VM instances, so a GPU pilot needs a paid billing account.
Q.3 Does Google train its models on my business data?
Google’s Gemini Enterprise Agent Platform terms say customer data isn’t used to train or fine-tune models without permission or instruction. Retention is a separate question, so review it for prompts, logs and embeddings.
Q.4 Should I migrate myself or hire a partner?
DIY suits well-understood, bounded workloads with a named rollback owner. A partner makes more sense when skills are thin, dependencies are unclear or a failed cutover would affect customers.
