istresearch/scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

1,224

PythonMIT License

distributedkafkapythonredisscrapingscrapy

View on GitHub

Stars

1,224

Updated

Nov 20, 2025

Stars Over Time

Top Contributors

ashrafnezhad-hamidreza

1 contributions

hartzelljkh

1 contributions

456wyc

1 contributions

cam1985

1 contributions

Falamarcao

1 contributions

der-ofenmeister

1 contributions

jarretjeter

1 contributions

shivamstaq

1 contributions

benqian

1 contributions

AndyLee1024

1 contributions

cuigrey

1 contributions

josephvore

1 contributions

mateozaratefw

1 contributions

michael-sola

1 contributions

marcialwushu

1 contributions

Zhizekai

1 contributions

sorianocarlos

1 contributions

drogbadvc

1 contributions

ShawnZDesign

1 contributions

MuhammadMudassirRaza12345

1 contributions

hossam-elshabory

1 contributions

ethereal-sandeep

1 contributions

manhhungdt06

1 contributions

dmend3z

1 contributions

leo-sai

1 contributions

suthree

1 contributions

djun

1 contributions

vitor-mafra

1 contributions

ArkiKirov

1 contributions

anil-gunda

1 contributions

Olamidipupo-favour

1 contributions

Puppeteer2024

1 contributions

dwiprash

1 contributions

dapoos

1 contributions

NoahLu1105

1 contributions

Narayan-21

1 contributions

hominhtu1703

1 contributions

xiaog929

1 contributions

haxorthematrix

1 contributions

blackfenzer

1 contributions

kensteele

1 contributions

alicanbozkurt

1 contributions

giglr

1 contributions

mrks111

1 contributions

saidiahd

1 contributions

connorodea

1 contributions

minner-fun

1 contributions

rcortx

1 contributions

ncldavies

1 contributions

shubhampachori12110095

1 contributions

aakarsh

1 contributions

MLDLfn

1 contributions

pmartins

1 contributions

gvoropaev

1 contributions

kreia

1 contributions

Ginko-L

1 contributions

drunksu

1 contributions

tonydavis629

1 contributions

jdgg777

1 contributions

chen-shixin

1 contributions

excommunicades

1 contributions

Noixas

1 contributions

fendouai

1 contributions

ngudev

1 contributions

dbianbian

1 contributions

tcosnr

1 contributions

Macorai

1 contributions

artinnok

1 contributions

anasmnasri2023

1 contributions

AhmedZeer

1 contributions

torres-federico-e

1 contributions

rhiznomad

1 contributions

ebatelchyrok

1 contributions

huangpd

1 contributions

davodsaraei

1 contributions

Cz664

1 contributions

Melaniecode

1 contributions

nke680

1 contributions

bennyho382

1 contributions

luzihang123

1 contributions

movila

1 contributions

effuone

1 contributions

corelchen

1 contributions

Vic6190

1 contributions

VageLO

1 contributions

glafkosch

1 contributions

andrzejk87

1 contributions

bellyfat

1 contributions

themreza

1 contributions

uosj-t

1 contributions

Gagan-Polukonda

1 contributions

YI-J-L

1 contributions

Baronofthehorde

1 contributions

nghilethanh09052000

1 contributions

SamhandsomeLee

1 contributions

lsixu

1 contributions

blankors

1 contributions

fortranlee

1 contributions

Mornante

1 contributions

danielckv

1 contributions

Related Repositories

Track developers from istresearch/scrapy-cluster

Join 1,000+ companies finding quality developer leads