How to download all files from URL?

问题

I'm pretty new with PowerShell and need a script to download all files from the URL: https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/daily/kl/recent/

I already managed to download the files when I store the URLs of the single files in a List (see code). I tried different stuff to generate the list automatically, but I haven't made it.

$list = get-content "D:\ListURL.txt"

foreach($url in $list)
{
    $filename =[System.IO.Path]::GetFileName($url) 
    $file =[System.IO.Path]::Combine($outputdir, $filename) 
    Invoke-WebRequest -Uri $url -OutFile $file

}

Can anyone help me with some code to create the list from the URL? Many thanks in advance.

回答1:

Looking at the file list on that url, this works for me:

$outputdir = 'D:\Downloads'
$url       = 'https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/daily/kl/recent/'

# enable TLS 1.2 and TLS 1.1 protocols
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12, [Net.SecurityProtocolType]::Tls11

$WebResponse = Invoke-WebRequest -Uri $url
# get the list of links, skip the first one ("../") and download the files
$WebResponse.Links | Select-Object -ExpandProperty href -Skip 1 | ForEach-Object {
    Write-Host "Downloading file '$_'"
    $filePath = Join-Path -Path $outputdir -ChildPath $_
    $fileUrl  = '{0}/{1}' -f $url.TrimEnd('/'), $_
    Invoke-WebRequest -Uri $fileUrl -OutFile $filePath
}

回答2:

This should do the trick:

Add-Type -AssemblyName System.Web
Add-Type -AssemblyName System.Web.Extensions

$url       = 'https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/daily/kl/recent/'
$destPath  = 'C:\temp\'

$response  = Invoke-WebRequest -Uri $url -Method GET

$files = @( $response.Content -split [environment]::NewLine | ? { $_ -like '*tageswerte*.zip*' } | % { $_ -replace '^(.*>)(tages.*\.zip)<.*', '$2' } )

foreach( $file in $files ) {

    Invoke-WebRequest -Uri ($url + $file) -Method GET -OutFile ($destPath + $file) | Out-Null
}

来源：https://stackoverflow.com/questions/58523870/how-to-download-all-files-from-url

标签

powershell