Yinyo

A wonderfully simple API to reliably execute many long running scrapers in a super scaleable way. Built on top of Kubernetes.

Get Started
photo for Easy and scaleable

Easy and scaleable

Easily run as many scrapers as you like across a cluster of machines without having to sweat the details. Powered by Kubernetes.
photo for Languages you love

Languages you love

Use the language and libraries you love for writing scrapers. Supports Python, JavaScript, Ruby, PHP and Perl via Heroku Buildpacks.
photo for Flexible

Flexible

Supports many different use cases through a simple, yet flexible API that can operate synchronously or asynchronously.
photo for Open source

Open source

Made specifically for developers of open source and proprietary scraper systems. No chance of vendor lock-in because Yinyo is open source and liberally licensed under the Apache 2.0 license.

Why we created Yinyo

The OpenAustralia Foundation hosts the world’s largest site for open scrapers, morph.io, and has many, many years of experience writing scrapers.

Originally we designed morph.io to run on top of Docker. As the site has continued to grow over the years we have hit the limitations of some of the early design decisions we made. That’s why we created Yinyo.

It is intended as a much improved, much more scaleable low level underpinnings of morph.io. But it’s a lot more than that.

It’s intended as a foundation upon which other developers can build other scraper systems quite different from morph.io. Yinyo isn’t opinionated about how scrapers store their data or what languages they’re written in. Yet using it to develop your scraper system will save you an enormous amount of effort.

Get Started

This is going to make morph.io so much better! I wish we had done this ages ago.

Matthew Landauer
Matthew Landauer

Yup, Yinyo is fantastic.

Kat Szuminska
Kat Szuminska

Morph made it easy to run our scrapers and consume all the memory and CPU of a single VM. Now with Yinyo we can consume all the resources across a whole cluster!

James Polley