The archivist's web crawler: WARC output, dashboard for all crawls, dynamic igno

Python1233other

3 months ago

archivingcrawlcrawler

Java library for reading and writing WARC files with a typed API

Java43apache-2.0

2 months ago

WARC writing MITM HTTP/S proxy

Python357

6 months ago

A search interface and wayback machine for the UKWA Solr based warc-indexer fram

Java91apache-2.0

2 months ago

simple script to convert web resources to a single warc file

Python14mit

11 months ago

Streaming WARC/ARC library for fast web archive IO

Python335apache-2.0

5 months ago

pythonpywbwarc

WARC and ARC indexing and discovery tools.

Java114

4 months ago

Java application to download WARCs from WASAPI

Java6other

3 months ago

applicationinfrastructurejava

Parse And Create Web ARChive (WARC) files with node.js

JavaScript90mit

last year

chrome-remote-interfacepupeteerwarc

Tool and library for handling Web ARChive (WARC) files.

Python134gpl-3.0

last year

python

An easy-to-use and highly customizable crawler that enables you to create your o

Scala24mit

7 years ago

Converts WARC files to static HTML

Java36apache-2.0

last year

Command line tools and libraries for handling and manipulating WARC files (and H

Python140mit

4 years ago

Converts HTTrack crawls to WARC files

Java27apache-2.0

2 years ago

web-archiving

Convert HTTP Archive (HAR) -> Web Archive (WARC) format

Python41apache-2.0

6 years ago

Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.

Python40mit

6 years ago

golang readers for ARC and WARC webarchive formats

Go17apache-2.0

last year