『Skills, Context, and Trust: The New Agentic Coding Stack』のカバーアート

Skills, Context, and Trust: The New Agentic Coding Stack

Skills, Context, and Trust: The New Agentic Coding Stack

無料で聴く

ポッドキャストの詳細を見る
Dan and Jonathan Bown open with the talk Jonathan gave at ODSC, "Practical Agent Ops: From POC to Prod with MLflow 3.0." MLflow 3.0 arrived last summer as the first stable release built for generative AI rather than traditional machine learning, and Jonathan's team used it to build an agent for pre-enrollment students. The centerpiece of that work was evaluation-driven development. Instead of jumping straight into a working prototype they aligning the business up front on what quality actually looks like before signing off on a model with inherently non-deterministic output.The turning point came from an overlooked asset: a data science team had already assembled 150 ground truth examples in an Excel file, untested and set aside while engineers focused on code. Jonathan's team paused the coding work and ran a simple foundation model against those examples first, landing at what amounted to a coin flip of useful versus hallucinated answers. From there they refined the examples with the business, loaded them into MLflow's evaluation datasets built from live traces, and iterated by versioning prompts and agent configurations.Tooling came up repeatedly. MLflow's open source repo now ships a skill file that plugs into coding tools like Claude Code, which Jonathan called a game changer for keeping up with an API that changes at roughly a release a month. The Databricks AI Dev Kit, released around March, bundles skills for the Databricks SDK, CLI, data engineering, and analytics work, usable either inside Databricks' Genie Code pane or in outside tools such as Claude Code, AWS Kiro, or Google Antigravity. Jonathan said installing it produced a dramatic jump in output accuracy compared to coding assistants working from stale or incomplete context about Databricks and MLflow APIs.Dan raised the idea that LLMs and agentic tools are becoming users of software in their own right, alongside humans, and Jonathan tied that to broader changes at WGU: more of the business, not just engineers, now writes system prompts and builds their own copilot-style agents. His own day to day has moved from core development toward AI enablement, meaning security review, best practices, and helping non-technical staff adopt evaluation-driven habits for the prompts and agents they build themselves.Jonathan's path to WGU ran through Pentara, a biostatistics consultancy, and Zions Bancorporation, where he did quant finance work before a stint simulating financial products for WGU students. He became a founding member of WGU's MLOps team in 2023, when the university's machine learning was still traditional work like random forests and ensembles for predicting student outcomes, well before Databricks had built out MLOps tooling. Dan connected this to Hamel Husain's essay The Revenge of the Data Scientist, and Jonathan agreed that evaluation-driven development brings the work full circle: checking evals, correctness, tool call accuracy, and safety is the generative AI analogue of checking a confusion matrix.The pre-enrollment agent's rollout became the clearest illustration of the method. The first release, a bare foundation model with no WGU context, drew heavy negative feedback from the employees testing it, some of whom wanted to shut the project down. Jonathan's team treated that feedback as fuel, folding the failed questions into an evaluation dataset and iterating until they reached roughly 82 percent correctness and near-total relevance, at which point the same employees became enthusiastic supporters. He credited MLflow's architecture for building subject matter experts directly into the agent ops workflow rather than treating evaluation as a purely technical exercise.Jonathan was candid about where his trust runs out. He does not trust a tool's first output even after a full planning session, citing a Kiro planning cycle from the day before that failed on the first try despite extensive back and forth. He is cautious about MLflow's fast release cadence outpacing its own skill files, and notably guarded about tools like OpenClaw and Claude Cowork that can reach into email or personal documents. Given how much effort WGU puts into protecting student data, he extends the same caution to his own personal information and limits what such agents can access.On his team, Jonathan resists banning AI-generated code or stigmatizing it in review, and instead pushes everyone toward reviewing code outside their usual specialty, using AI review tools like Amazon Q or GitHub Copilot as a starting point rather than a final answer. He pushed back on the idea that tool usage equals productivity, warning about AI slop and noting that some of the heaviest users he knows are not the most productive. The thread ties back to evaluation-driven development's real thesis: start from value, not from the tool, a point he illustrated with WGU's Academic Virtual Assistant pilot, where a surprising result showed that students chatting with the assistant were ...
adbl_web_anon_alc_button_suppression_t1
まだレビューはありません