I've been trying to use wget to retrieve a page that is only accessible
to logged-in users (user stats - so that I can analyse them and keep a running record of changes).
Basically, I can't seem to get the correct syntax for the site to receive/recognise my name and password in the first place, let alone to
serve up the stats page requested....
I've been trying to use wget to retrieve a page that is only accessible
to logged-in users (user stats - so that I can analyse them and keep a >running record of changes).
Basically, I can't seem to get the correct syntax for the site to >receive/recognise my name and password in the first place, let alone to
serve up the stats page requested....
I don't really know how to use the relevant features of
wget and have been flailing around rather at random. Simply using
wget --ask-password STATS_URL
doesn't produce the desired result; it prompts for the password all
right, but when I supply it the fetch then gets redirected to retrieve
the log-in page instead, just as if I had supplied no password or the
wrong one.
wget --user=USERNAME --ask-password STATS_URL
prompts "Password for user" instead of just "Password", but still
doesn't seem to pass the required data.
Same result from
wget --user=USERNAME --password=PASS STATS_URL
(the retrieved page states 'sorry, you don't have access to view the
page you were trying to reach, please log in')
After looking for advice on the Web I tried fetching the log-in page
directly using the same methods and using --keep-session-cookies before >running a second command to fetch the stats page immediately afterwards,
but that didn't work. It fetches the login page, then redirects and
fetches it again under a different name, the only difference being the
error:
<div class="flash error">Sorry, you don't have permission to access the page you were trying to reach. Please log in.</div>
I then tried using --save-cookies followed by --load-cookies for the
second request, but that didn't work, doubtless because the resulting >'cookies' file had no content:
# HTTP cookie file.
# Generated by Wget on 2022-01-21 13:37:33.
# Edit at your own risk.
I then tried
wget --post-data 'user_login=USERNAME&user_password=PASS' LOGIN_URL
where the relevant form reads
<dt><label for="user_login">User name or email:</label></dt>
<dd><input type="text" name="user[login]" id="user_login"/></dd>
<dt><label for="user_password">Password:</label></dt>
<dd><input type="password" name="user[password]" id="user_password"/></dd>
<dt><label for="user_remember_me">Remember me</label></dt>
<dd><input name="user[remember_me]" type="hidden" value="0"/><input type="checkbox" value="1" name="user[remember_me]" id="user_remember_me"/></dd>
<dt class="landmark">Submit</dt>
<dd class="submit actions">
<input type="submit" name="commit" value="Log in" class="submit"/>
</dd>
but still had no luck.
I'm simply not managing to submit the name/password combination in any
way that the site will acknowledge.
In message <dd8749ae59.harriet@bazleyfamily.co.uk>
Harriet Bazley <harriet@bazleyfamily.co.uk> wrote:
I've been trying to use wget to retrieve a page that is only accessible
to logged-in users (user stats - so that I can analyse them and keep a >running record of changes).
Basically, I can't seem to get the correct syntax for the site to >receive/recognise my name and password in the first place, let alone to >serve up the stats page requested....
If you use the -S option you get the server response, which if used woth
the -o option you can then save it and see what the server is saying.
Well, that's interesting - it's doing a 'set-cookie', but no cookies are being stored by wget....
On 21 Jan, Harriet Bazley wrote in message
<569f5fae59.harriet@bazleyfamily.co.uk>:
Well, that's interesting - it's doing a 'set-cookie', but no cookies are being stored by wget....
Are you using wget's --save-cookies and --keep-session-cookies options? You then load then with --load-cookies on subsequent calls.
I've been trying to use wget to retrieve a page that is only accessible
to logged-in users (user stats - so that I can analyse them and keep a running record of changes).
Basically, I can't seem to get the correct syntax for the site to receive/recognise my name and password in the first place, let alone to
serve up the stats page requested....
On 21/01/2022 13:43, Harriet Bazley wrote:
I've been trying to use wget to retrieve a page that is only accessible
to logged-in users (user stats - so that I can analyse them and keep a running record of changes).
Basically, I can't seem to get the correct syntax for the site to receive/recognise my name and password in the first place, let alone to serve up the stats page requested....
It will be possible to do this with wget (or curl), but as you can see
from the other responses, it involves sprinkling fairy dust over the
correct magic runes. It may be easier to use Python with the requests
module for this, as it can set up auth headers and suchlike.
Sysop: | Keyop |
---|---|
Location: | Huddersfield, West Yorkshire, UK |
Users: | 285 |
Nodes: | 16 (2 / 14) |
Uptime: | 60:41:05 |
Calls: | 6,488 |
Calls today: | 1 |
Files: | 12,094 |
Messages: | 5,274,354 |