web_trawler summary

Plugin Name: web_trawler
Version: 0.1.0
Author: @melezhik
Realease Date: 2017-09-29 12:23:09
Short Description: Simple wrapper for web_trawler script
Category: utilities
Plugin web page: https://gitlab.com/melezhik/web_trawler
Download link: web_trawler-v0.001000.tar.gz
Latest version link: https://sparrowhub.org/info/web_trawler

web_trawler documentation


Simple wrapper for web_trawler


  $ sparrow plg install web_trawler


Basic usage:

  $ sparrow plg run web_trawler --param url=$url -- <web_trawler_params>

For example:

  $ sparrow plg run web_trawler \
  --param url=http://www.ldoceonline.com/dictionary/make-out -- \
  --processes 2 \
  --whitelist '*.mp3'
  --target ~/dictionary

See parameters description at https://gitlab.com/dlab-indecol/web_trawler

If you need some automation:

  $ sparrow project create english

  $ sparrow task add english longman-dict web_trawler

  $ sparrow task ini english/longman-dict

      target: /home/melezhik/dictionary
      processes: 2 
      whitelist: "*.mp3"

  $ sparrow task run english/longman-dict url=http://www.ldoceonline.com/dictionary/make-out


  • The author of main script is Gorm Roedder ( gormroedder_at_gmail.com )
  • The plugin maintainer is Alexey Melezhik