Sydney is home to some of the best AI engineers in the world, from early-stage startups to global enterprises. SYD AI Meet Up is where they come together. Whether you're shipping AI features in production, experimenting with AI SDK, or figuring out how AI fits into your stack - this is your room.
The headline: Collaboration with Clickhouse
What to Expect
6:00 – 6:30 PM: Pizza & Networking
Grab a slice, meet fellow engineers, and connect with the community.
6:30 – 7 PM: Lightning Talks
"We planted real CVEs in 16 repos to find out which AI model actually catches vulnerabilities. The best model is a surprise." Patrick Collins CEO, Dam Secure
From Traces to Trust: Shipping Reliable LLM Apps with Langfuse by Muhammad Ali, Senior Solution Architect @ ClickHouse
7 PM – 9:00 PM: Networking & Wrap Up
More details on the lightning talks
Patrick Collins CEO, Dam Secure
We planted real CVEs in 16 repos to find out which AI model actually catches vulnerabilities. The best model is a surprise."
Every new model is benchmaxxing and also test against the same public vulnerable repos, like OWASP Juice Shop. That makes the standard benchmarks essentially useless.
So we built our own eval suite and benchmark. We mine CVEs from open-source commit history and replay the reverted fix back onto the codebase, across 16 repos.
I'll show the results and give a sneak peak into how we built that eval suite and the results. The leading model, which we ran through Vercel will surprise you!
This is a talk for developers interested in behind-the-scenes of eval suites and benchmarks.
Traces to Trust: Shipping Reliable LLM Apps with Langfuse
Building an LLM demo is easy; running one in production is not. This deep dive walks through how Langfuse helps teams move from "it works on my prompt" to measurable, debuggable, cost-aware AI systems. We'll explore tracing and spans for complex agent pipelines, prompt management and versioning, evaluation workflows to catch regressions before users d