getting a list of URLs from a website
Use lynx -dump -listonly -nonumbers foo.bar | grep -e barfoo > foobar.txt
, where foo.bar is the web page's URL, barfoo is a text string to base included URLs on, for example a file extension, and foobar.txt is the outputted list of URLs.
licensing
No rights reserved: CC0 1.0.
prior work
The -dump
and -listonly
options for Lynx were introduced to me by an answer on Stack Exchange by michas.