COMMUNIUM.AI

PuerkitoBio/fetchbot

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

790
GoBSD 3-Clause "New" or "Revised" License
crawlerrobots-txt
View on GitHub
Stars

790

Updated

Oct 3, 2025

Stars Over Time

Top Contributors

OB

obitech

1 contributions

XU

xuxiaoguang1

1 contributions

MI

MindTooth

1 contributions

WA

WaVerCon

1 contributions

GO

gosom

1 contributions

MI

mitghi

1 contributions

BO

bojand

1 contributions

FL

floxo115

1 contributions

IN

indieterminacy

1 contributions

SZ

szzhiyang

1 contributions

CH

chris124567

1 contributions

JO

jordanbangia

1 contributions

PE

Pengxn

1 contributions

LE

LexKarei

1 contributions

TR

trollixx

1 contributions

NY

nyimbi

1 contributions

OR

orzogc

1 contributions

EG

EgeBalci

1 contributions

ZE

zerdnem

1 contributions

YK

ykankaya

1 contributions

HI

higsx

1 contributions

WI

wignesh

1 contributions

FL

flotothemoon

1 contributions

SA

SantoshSrinivas79

1 contributions

TI

Tinram

1 contributions

DE

denghuafeng

1 contributions

RI

richklee

1 contributions

M0

m040601

1 contributions

I3

i3here

1 contributions

BB

bbkane

1 contributions

SO

sobamchan

1 contributions

AN

andrewwxy

1 contributions

LE

leviliangtw

1 contributions

CA

catthehacker

1 contributions

MR

mrluanma

1 contributions

FR

free-gate

1 contributions

HE

hehehai

1 contributions

DA

darcyturk

1 contributions

DR

dreamway

1 contributions

BA

Bashayr29

1 contributions

YS

ysinjab

1 contributions

SA

Sam9799

1 contributions

JN

jnorrid

1 contributions

BE

bergwald

1 contributions

DB

DBHeise

1 contributions

WO

wozuo

1 contributions

AN

anthonygedeon

1 contributions

AN

anh

1 contributions

LE

leepeterson

1 contributions

RE

re-taro

1 contributions

TR

tracure1337

1 contributions

AN

An0nYm0u5101

1 contributions

ZH

zhangjunfang

1 contributions

JD

jdpedrie

1 contributions

0N

0n10n

1 contributions

PA

paralin

1 contributions

DL

dlo

1 contributions

WA

wardn

1 contributions

PA

pat3icki

1 contributions

AY

ayatoy

1 contributions

GO

gofullthrottle

1 contributions

AI

AidenHadisi

1 contributions

CO

CozyRui

1 contributions

LO

Los-had

1 contributions

DO

domderen

1 contributions

RA

rapjul

1 contributions

AL

AlexMills0

1 contributions

AL

alexferrari88

1 contributions

ME

MeizhaiTechnology

1 contributions

MA

markthethomas

1 contributions

ID

idgenchev

1 contributions

OM

omasanori

1 contributions

MA

mariusrugan

1 contributions

VG

vgecko

1 contributions

MA

marcel

1 contributions

BR

breadchris

1 contributions

CH

chrisdlees

1 contributions

DA

davidxifeng

1 contributions

YA

yashade

1 contributions

XU

xuedipiaofei

1 contributions

LO

Lord1Egypt

1 contributions

KO

kom0055

1 contributions

DA

dante-e-v

1 contributions

BO

bonnie-banjax

1 contributions

YO

Youandme143

1 contributions

SI

simone-cominato

1 contributions

MA

ma-a-sc

1 contributions

DA

DanielShemesh

1 contributions

LA

lazazael

1 contributions

MO

Mohammad-Almusilhi

1 contributions

VL

Vlaaaaaaad

1 contributions

SA

Salz150

1 contributions

AS

asterryplace

1 contributions

IA

iamoeg

1 contributions

CL

clemfromspace

1 contributions

MI

mitar

1 contributions

PO

poysa213

1 contributions

AM

am-sokolov

1 contributions

CH

chenyansong1

1 contributions

JA

jakoblorz

1 contributions

Related Repositories

sourcegraph/webloop

1,366 stars

grammarly/rocker

1,333 stars

goplus/xgo

9,380 stars

Track developers from PuerkitoBio/fetchbot

Join 1,000+ companies finding quality developer leads