Retrieve data from a URL

Problem/Question/Abstract:

How do I fetch text from a URL?

Answer:

This is a follow up to article "Checking if a URL is valid" which returns the data at the web page in a string. If it fails you should get a Status:xxx where xxx is the status or nothing at all if there isn't a web server at the url you try.

It uses the InternetReadfile function to read the data in 4kb chunks. The actual size of the buffer is irrelevant unless (as in this case) it is declared local to the function. This takes up stack space and a large buffer could potentially lead to a stack overflow if there were many functions nested in the call stack. Either move it somewhere non stack based or keep it small.

One thing to watch is the string conversion. If you are using buffer data, make sure you add a #0 on the end and use pchars to convert before ending up with strings.

uses wininet...

function FetchHTML(url: string): string;
var
  databuffer: array[0..4095] of char;
  ResStr: string;
  hSession, hfile, hRequest: hInternet;
  dwindex, dwcodelen, datalen, dwread, dwNumber: cardinal;
  dwcode: array[1..20] of char;
  res: pchar;
  Str: pchar;

begin
  ResStr := '';
  if pos('http://', lowercase(url)) = 0 then
    url := 'http://' + url;
  hSession := InternetOpen('InetURL:/1.0',
    INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);
  if assigned(hsession) then
  begin
    hfile := InternetOpenUrl(
      hsession,
      pchar(url),
      nil,
      0,
      INTERNET_FLAG_RELOAD,
      0);
    dwIndex := 0;
    dwCodeLen := 10;
    HttpQueryInfo(hfile, HTTP_QUERY_STATUS_CODE,
      @dwcode, dwcodeLen, dwIndex);
    res := pchar(@dwcode);
    dwNumber := sizeof(databuffer) - 1;
    if (res = '200') or (res = '302') then
    begin
      while (InternetReadfile(hfile, @databuffer, dwNumber, DwRead)) do
      begin
        if dwRead = 0 then
          break;
        databuffer[dwread] := #0;
        Str := pchar(@databuffer);
        resStr := resStr + Str;
      end;
    end
    else
      ResStr := 'Status:' + res;
    if assigned(hfile) then
      InternetCloseHandle(hfile);
  end;
  InternetCloseHandle(hsession);
  Result := resStr;
end;

This code was written with the help of 'Essential WinInet' by Aaron Skonnard and if you are interested in this subject I strongly suggest you buy the book. ISBN 0-201-37936-8. The code in the book is in C/C++ but isn't too difficult to convert to delphi.