🚀 About us

We're a fast-growing Seed-stage startup building the next-generation web search engine designed for AI and LLM applications. We are building a product which will operate on clusters of 1,000+ machines, processing petabytes of data and powering large-scale retrieval for production AI workloads.

This is a rare opportunity to own core infrastructure end-to-end and shape how AI systems interact with the modern web.

🎯 The role

You will design and own the distributed crawling infrastructure that powers our entire search and indexing pipeline. The systems you'll build process millions of pages per day, handle the full complexity of the modern web (dynamic content, client-side rendering, rate limits, anti-bot challenges), and feed everything downstream from indexing to retrieval.

This is foundational systems work. You will write Rust that runs in production at scale from day one. You will work directly with our CTO and founding engineers on a component the rest of the company depends on.

📋 What you'll own

• Design, develop, and maintain a scalable distributed crawling infrastructure that powers our entire search and indexing pipeline.
• Collaborate on indexing, vectorization, and storage systems driven by the crawler.
• Ensure operational excellence, observability, and system reliability.
• Work closely with other team members to develop, test, and deploy new features and enhancements for our platform.
• Troubleshoot, debug, and optimize performance.

👤 Who you are

• You have built distributed crawling systems that processed several millions of pages per day in production.
• You are proficient in Rust or another high-performance systems language.
• You have a deep understanding of scalable content-fetching, politeness policies, scheduling, deduplication, and URL management.
• You are comfortable handling the full complexity of the modern web: dynamic pages, client-side rendering, rate limits, anti-bot challenges.
• You communicate clearly with both technical and non-technical team members.

💡 Bonus

• Experience with cloud platforms (AWS, GCP, Azure).