3 WWW::Mechanize::FAQ - Frequently Asked Questions about WWW::Mechanize
5 =head1 How to get help with WWW::Mechanize
7 If your question isn't answered here in the FAQ, please turn to the
12 =item * L<http://perlmonks.org>
14 =item * The libwww-perl mailing list at L<http://lists.perl.org>
20 =head2 I have this web page that has JavaScript on it, and my Mech program doesn't work.
22 That's because WWW::Mechanize doesn't operate on the JavaScript. It only
23 understands the HTML parts of the page.
25 =head2 I thought Mech was supposed to work like a web browser.
27 It does pretty much, but it doesn't support JavaScript.
29 I added some basic attempts at picking up URLs in C<window.open()>
30 calls and return them in C<< $mech->links >>. They work sometimes.
31 Beyond that, there's no support for JavaScript.
33 =head2 Are you going to add JavaScript support?
35 I will if anyone sends me the code to do it. I'm not going to write a
36 JavaScript processor myself.
38 =head2 Wouldn't that be a great thing to have in WWW::Mechanize?
42 =head2 Would it be hard to do?
44 Hard enough that I don't want to deal with it myself. Plus, I don't
45 use JavaScript myself, so I don't have an itch to scratch.
47 =head2 Is anyone working on it?
49 I've heard noises from people every so often over the past couple of
50 years, but nothing you'd pin your hopes on.
52 =head2 It would really help me with a project I'm working on.
56 =head2 Do you know when it might get added?
58 I have no idea if or when such a thing will ever get done. I can
59 guarantee that as soon as there's anything close to JavaScript support
60 I will let everyone know.
62 =head2 Maybe I'll ask around and see if anyone else knows of a solution.
64 If you must, but I doubt that anyone's written JavaScript support for
65 Mechanize and neglected to tell me about it.
67 =head2 So what can I do?
69 Since Javascript is completely visible to the client, it cannot be used
70 to prevent a scraper from following links. But it can make life difficult,
71 and until someone writes a Javascript interpreter for Perl or a Mechanize
72 clone to control Firefox, there will be no general solution. But if
73 you want to scrape specific pages, then a solution is always possible.
75 One typical use of Javascript is to perform argument checking before
76 posting to the server. The URL you want is probably just buried in the
77 Javascript function. Do a regular expression match on
78 C<< $mech->content() >>
79 to find the link that you want and C<< $mech->get >> it directly (this
80 assumes that you know what your are looking for in advance).
82 In more difficult cases, the Javascript is used for URL mangling to
83 satisfy the needs of some middleware. In this case you need to figure
84 out what the Javascript is doing (why are these URLs always really
85 long?). There is probably some function with one or more arguments which
86 calculates the new URL. Step one: using your favorite browser, get the
87 before and after URLs and save them to files. Edit each file, converting
88 the the argument separators ('?', '&' or ';') into newlines. Now it is
89 easy to use diff or comm to find out what Javascript did to the URL.
90 Step 2 - find the function call which created the URL - you will need
91 to parse and interpret its argument list. Using the Javascript Debugger
92 Extension for Firefox may help with the analysis. At this point, it is
93 fairly trivial to write your own function which emulates the Javascript
94 for the pages you want to process.
96 Here's annother approach that answers the question, "It works in Firefox,
97 but why not Mech?" Everything the web server knows about the client is
98 present in the HTTP request. If two requests are identical, the results
99 should be identical. So the real question is "What is different between
100 the mech request and the Firefox request?"
102 The Firefox extension "Tamper Data" is an effective tool for examining
103 the headers of the requests to the server. Compare that with what LWP
104 is sending. Once the two are identical, the action of the server should
107 I say "should", because this is an oversimplification - some values
108 are naturally unique, e.g. a SessionID, but if a SessionID is present,
109 that is probably sufficient, even though the value will be different
110 between the LWP request and the Firefox request. The server could use
111 the session to store information which is troublesome, but that's not
112 the first place to look (and highly unlikely to be relevant when you
113 are requesting the login page of your site).
115 Generally the problem is to be found in missing or incorrect POSTDATA
116 arguments, Cookies, User-Agents, Accepts, etc. If you are using mech,
117 then redirects and cookies should not be a problem, but are listed here
118 for completeness. If you are missing headers, C<< $mech->add_header >>
119 can be used to add the headers that you need.
121 =head1 How do I do X?
123 =head2 Can I do [such-and-such] with WWW::Mechanize?
125 If it's possible with LWP::UserAgent, then yes. WWW::Mechanize is
126 a subclass of L<LWP::UserAgent>, so all the wondrous magic of that
129 =head2 How do I use WWW::Mechanize through a proxy server?
131 See the docs in L<LWP::UserAgent> on how to use the proxy. Short version:
133 $mech->proxy(['http', 'ftp'], 'http://proxy.example.com:8000/');
135 or get the specs from the environment:
139 # Environment set like so:
140 gopher_proxy=http://proxy.my.place/
141 wais_proxy=http://proxy.my.place/
142 no_proxy="localhost,my.domain"
143 export gopher_proxy wais_proxy no_proxy
145 =head2 How can I see what fields are on the forms?
147 Use the mech-dump utility, optionaly installed with Mechanize.
149 $ mech-dump --forms http://search.cpan.org
151 GET http://search.cpan.org/search
153 mode=all (option) [*all|module|dist|author]
154 <NONAME>=CPAN Search (submit)
156 =head2 How do I get Mech to handle authentication?
160 my $agent = WWW::Mechanize->new();
162 Authorization => "Basic " .
163 MIME::Base64::encode( USER . ':' . PASS )
166 $agent->credentials( ADDRESS, REALM, USER, PASS );
167 $agent->get( URL, @args );
169 =head2 How can I get WWW::Mechanize to execute this JavaScript?
171 You can't. JavaScript is entirely client-based, and WWW::Mechanize
172 is a client that doesn't understand JavaScript. See the top part
175 =head2 How do I check a checkbox that doesn't have a value defined?
177 Set it to to the value of "on".
179 $mech->field( my_checkbox => 'on' );
181 =head2 How do I handle frames?
183 You don't deal with them as frames, per se, but as links. Extract
186 my @frame_links = $mech->find_link( tag => "frame" );
188 =head2 How do I get a list of HTTP headers and their values?
190 All L<HTTP::Headers> methods work on a L<HTTP::Response> object which is
191 returned by the I<get()>, I<reload()>, I<response()/res()>, I<click()>,
192 I<submit_form()>, and I<request()> methods.
194 my $mech = WWW::Mechanize->new( autocheck => 1 );
195 $mech->get( 'http://my.site.com' );
196 my $res = $mech->response();
197 for my $key ( $response->header_field_names() ) {
198 print $key, " : ", $response->header( $key ), "\n";
201 =head1 Why doesn't this work: Debugging your Mechanize program
203 =head2 My Mech program doesn't work, but it works in the browser.
205 Mechanize acts like a browser, but apparently something you're doing
206 is not matching the browser's behavior. Maybe it's expecting a
207 certain web client, or maybe you've not handling a field properly.
208 For some reason, your Mech problem isn't doing exactly what the
209 browser is doing, and when you find that, you'll have the answer.
211 =head2 My Mech program gets these 500 errors.
213 A 500 error from the web server says that the program on the server
214 side died. Probably the web server program was expecting certain
215 inputs that you didn't supply, and instead of handling it nicely,
218 Whatever the cause of the 500 error, if it works in the browser,
219 but not in your Mech program, you're not acting like the browser.
220 See the previous question.
222 =head2 Why doesn't my program handle this form correctly?
224 Run F<mech-dump> on your page and see what it says.
226 F<mech-dump> is a marvelous diagnostic tool for figuring out what forms
227 and fields are on the page. Say you're scraping CNN.com, you'd get this:
229 $ mech-dump http://www.cnn.com/
230 GET http://search.cnn.com/cnn/search
231 source=cnn (hidden readonly)
232 invocationType=search/top (hidden readonly)
233 sites=web (radio) [*web/The Web ??|cnn/CNN.com ??]
235 <NONAME>=Search (submit)
237 POST http://cgi.money.cnn.com/servlets/quote_redirect
239 <NONAME>=GET (submit)
241 POST http://polls.cnn.com/poll
242 poll_id=2112 (hidden readonly)
243 question_1=<UNDEF> (radio) [1/Simplistic option|2/VIEW RESULTS]
244 <NONAME>=VOTE (submit)
246 GET http://search.cnn.com/cnn/search
247 source=cnn (hidden readonly)
248 invocationType=search/bottom (hidden readonly)
249 sites=web (radio) [*web/??CNN.com|cnn/??]
251 <NONAME>=Search (submit)
253 Four forms, including the first one duplicated at the end. All the
254 fields, all their defaults, lovingly generated by HTML::Form's C<dump>
257 If you want to run F<mech-dump> on something that doesn't lend itself
258 to a quick URL fetch, then use the C<save_content()> method to write
259 the HTML to a file, and run F<mech-dump> on the file.
261 =head2 Why don't https:// URLs work?
263 You need either L<IO::Socket::SSL> or L<Crypt::SSLeay> installed.
265 =head2 Why do I get "Input 'fieldname' is readonly"?
267 You're trying to change the value of a hidden field and you have
270 First, make sure that you actually mean to change the field that you're
271 changing, and that you don't have a typo. Usually, hidden variables are
272 set by the site you're working on for a reason. If you change the value,
273 you might be breaking some functionality by faking it out.
275 If you really do want to change a hidden value, make the changes in a
276 scope that has warnings turned off:
280 $agent->field( name => $value );
283 =head2 I tried to [such-and-such] and I got this weird error.
285 Are you checking your errors?
289 Are you checking that your action succeeded after every action?
293 For example, if you try this:
295 $mech->get( "http://my.site.com" );
296 $mech->follow_link( "foo" );
298 and the C<get> call fails for some reason, then the Mech internals
299 will be unusable for the C<follow_link> and you'll get a weird
300 error. You B<must>, after every action that GETs or POSTs a page,
301 check that Mech succeeded, or all bets are off.
303 $mech->get( "http://my.site.com" );
304 die "Can't even get the home page: ", $mech->response->status_line
305 unless $mech->success;
307 $mech->follow_link( "foo" );
308 die "Foo link failed: ", $mech->response->status_line
309 unless $mech->success;
311 =head2 How do I figure out why C<< $mech->get($url) >> doesn't work?
313 There are many reasons why a C<< get() >> can fail. The server can take
314 you to someplace you didn't expect. It can generate redirects which are
315 not properly handled. You can get time-outs. Servers are down more often
316 than you think! etc, etc, etc. A couple of places to start:
320 =item 1 Check C<< $mech->status() >> after each call
322 =item 2 Check the URL with C<< $mech->uri() >> to see where you ended up
324 =item 3 Try debugging with C<< LWP::Debug >>.
328 If things are really strange, turn on debugging with
329 C<< use LWP::Debug qw(+); >>
330 Just put this in the main program. This causes LWP to print out a trace
331 of the HTTP traffic between client and server and can be used to figure
332 out what is happening at the protocol level.
334 It is also useful to set many traps to verify that processing is
335 proceeding as expected. A Mech program should always have an "I didn't
336 expect to get here" or "I don't recognize the page that I am processing"
339 Since errors can be transient, by the time you notice that the error
340 has occurred, it might not be possible to reproduce it manually. So
341 for automated processing it is useful to email yourself the following
346 =item * where processing is taking place
348 =item * An Error Message
352 =item * $mech->content
356 You can also save the content of the page with C<< $mech->save_content( 'filename.html' ); >>
358 =head2 I submitted a form, but the server ignored everything! I got an empty form back!
360 The post is handled by application software. It is common for PHP
361 programmers to use the same file both to display a form and to process
362 the arguments returned. So the first task of the application programmer
363 is to decide whether there are arguments to processes. The program can
364 check whether a particular parameter has been set, whether a hidden
365 parameter has been set, or whether the submit button has been clicked.
366 (There are probably other ways that I haven't thought of).
368 In any case, if your form is not setting the parameter (e.g. the submit
369 button) which the web application is keying on (and as an outsider there
370 is no way to know what it is keying on), it will not notice that the form
371 has been submitted. Try using C<< $mech->click() >> instead of
372 C<< $mech->submit() >> or vice-versa.
374 =head2 I've logged in to the server, but I get 500 errors when I try to get to protected content.
376 Some web sites use distributed databases for their processing. It
377 can take a few seconds for the login/session information to percolate
378 through to all the servers. For human users with their slow reaction
379 times, this is not a problem, but a Perl script can outrun the server.
380 So try adding a sleep(5) between logging in and actually doing anything
381 (the optimal delay must be determined experimentally).
383 =head2 Mech is a big memory pig! I'm running out of RAM!
385 Mech keeps a history of every page, and the state it was in. It actually
386 keeps a clone of the full Mech object at every step along the way.
388 You can limit this stack size with the C<stack_depth> parm in the C<new()>
393 Copyright 2005 Andy Lester C<< <andy at petdance.com> >>