Parse and scrape a web page

    Category: PowerShell
    Hits: 2216

    Parse and scrape a web page

    Invoke-WebRequest:

    Invoke-WebRequest is a built-in PowerShell cmdlet that allows you to send HTTP and HTTPS requests to web services, download files, and interact with web pages. You can use it to perform various operations like getting the content of a web page, downloading files, or making API calls.

    This cmdlet was introduced in Windows PowerShell 3.0.

    Here's the basic syntax for the Invoke-WebRequest cmdlet:

    Invoke-WebRequest -Uri <URL> -Method <HTTP_Method> -Headers <Headers> -Body <Body_Content> -ContentType <Content_Type>

    Here's a brief explanation of the parameters:

    • -Uri: The URL you want to request.
    • -Method: The HTTP method for the request (GET, POST, PUT, DELETE, etc.). The default method is GET.
    • -Headers: A dictionary containing headers to be sent with the request.
    • -Body: The content to be sent in the request body for methods like POST or PUT.
    • -ContentType: The content type of the request body, e.g., "application/json" for JSON data.

     

    Examples:

    Get the content of a web page:

    $response = Invoke-WebRequest -Uri "https://example.com"
Write-Host $response. Content
     
    Download a file:
    Invoke-WebRequest -Uri "https://example.com/file.zip" -OutFile "C:\Downloads\file.zip"
     

    Make a POST request with JSON data:

    $url = "https://api.example.com/data"
$jsonData = @{
    key1 = "value1"
    key2 = "value2"
} | ConvertTo-Json

$response = Invoke-WebRequest -Uri $url -Method POST -Body $jsonData -ContentType "application/json"
Write-Host $response. Content
     
    Make a GET request with custom headers:
    $url = "https://api.example.com/data"
$headers = @{
    "Authorization" = "Bearer your_access_token"
    "Accept" = "application/json"
}

$response = Invoke-WebRequest -Uri $url -Headers $headers
Write-Host $response. Content
     
    These examples demonstrate some of the most common use cases for the Invoke-WebRequest cmdlet in PowerShell. You can adapt and expand these examples for your specific needs.
     

    Invoke-RestMethod:

    Invoke-RestMethod is a PowerShell cmdlet that allows you to send HTTP and HTTPS requests to RESTful web services. It is similar to Invoke-WebRequest but has some differences in how it handles the response data. Invoke-RestMethod automatically converts JSON and XML responses into PowerShell objects, making it easier to work with the returned data.

    Here's the basic syntax for the Invoke-RestMethod cmdlet:

    Invoke-RestMethod -Uri <URL> -Method <HTTP_Method> -Headers <Headers> -Body <Body_Content> -ContentType <Content_Type>

    Here's a brief explanation of the parameters:

    -Uri: The URL you want to request.
    -Method: The HTTP method for the request (GET, POST, PUT, DELETE, etc.). The default method is GET.
    -Headers: A dictionary containing headers to be sent with the request.
    -Body: The content to be sent in the request body for methods like POST or PUT.
    -ContentType: The content type of the request body, e.g., "application/json" for JSON data.

    Examples

    Get JSON data from a RESTful API:

    $url = "https://api.example.com/data"
$response = Invoke-RestMethod -Uri $url
Write-Host $response.key1

    Make a POST request with JSON data:

    $url = "https://api.example.com/data"
$jsonData = @{
    key1 = "value1"
    key2 = "value2"
} | ConvertTo-Json

$response = Invoke-RestMethod -Uri $url -Method POST -Body $jsonData -ContentType "application/json"
Write-Host $response.key1

    Make a GET request with custom headers:

    $url = "https://api.example.com/data"
$headers = @{
    "Authorization" = "Bearer your_access_token"
    "Accept" = "application/json"
}

$response = Invoke-RestMethod -Uri $url -Headers $headers
Write-Host $response.key1

    These examples demonstrate some of the most common use cases for the Invoke-RestMethod cmdlet in PowerShell. You can adapt and expand these examples for your specific needs. The main advantage of Invoke-RestMethod over Invoke-WebRequest is the automatic conversion of JSON and XML responses into PowerShell objects, which makes it easier to manipulate and use the data.

     

    Main Menu