Thursday, October 05, 2006

WGET: Using Wget through a Proxy Server

I get annoyed that I always forget how to do this, so I am going to post on it so I will always know where to go.

In order to use WGET through a proxy server, the environment variable HTTP_PROXY needs to be set.

export http_proxy=http://proxy.example.com:8080

Then, run wget with the --proxy option.
Wget --proxy http://site_to_Retrieve

FTP_PROXY will work with FTP sites as well.

5 comments:

Andrew from Vancouver said...

John, I use wget a lot too, and have some suggestions.

Instead of using the environment variable (or the .wgetrc file), specify the option on the command line with the -e parameter, e.g.

wget --execute=http_proxy=proxy.example.com:8080

Another thing you might want to keep handy are the parameters if that proxy server needs authentication:

--proxy-user=USER
--proxy-password=PASS

Often, I want wget to impersonate Internet Explorer, so I use a script that does:

wget --user-agent="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" %1 %2 %3 %4 %5 %6 %7 %8 %9

I call it IEget.cmd in the Windows world.

Andrew.

Anonymous said...

Andrew you thanks for that but you got the syntax just tiny bit wrong. you wrote
"wget --execute=http_proxy=proxy.example.com:8080"

when in reality the the proper command is:
wget -e "http_proxy = http://ip:port" "http://content"

I used -e instead of -execute, the equals sign you put after -execute breaks the command syntax. Thanks again for your comment, it saves me editing me wget config files when I want to use a proxy.

Anonymous said...

Hi I am using a proxy server with authentication, I have used following command but it failed to link
Command:
wget -e http_proxy=http://172.31.1.4:8080 --proxy-user=username --proxy-password=password -k http://iiita.ac.in
please help.

John Ward said...

Anonymous,

The -e flag will try to execute an external command, it is not what you want. You want to set an environment variable for HTTP_PROXY before you run WGET. Or, you can try to change the -e flag to run like -e "export http_proxy=http://proxy.example.com:8080".

Anonymous said...

There was a little hint which without knowing it made me a hard time with wget.
I got used to set only the HTTP_PROXY env variable but the version of wget "GNU Wget 1.11.4 Red Hat modified" does not use it - instead it uses http_proxy variable(lowercase).