3 ways to download files with PowerShell
Perhaps the greatest strength of PowerShell is it's foundation on the .NET framework. The .NET framework enables almost unlimited possibilites inside the scripting realm. This blessing can equally be a curse as things can get complicated. Fast.
This post will describe three methods for downloading files using PowerShell - weighed up with their pros and cons.
Test setup
Today's testing is not highly scientific. The point is to show the difference in execution time and performance.
The test setup consists of PowerShell 4 running on Windows 8.1 x64 with my VDSL connection curently synced at 35.9/10.4 mbps.
I will be downloading a test file from Internode at the following URL: http://mirror.internode.on.net/pub/test/10meg.test. The scripts will be executed 10 times each with the average displayed as the result.
Let's get started!
1. Invoke-WebRequest
The first and most obvious option is the Invoke-WebRequest cmdlet. It is built into PowerShell and can be used in the following method:
$url = "http://mirror.internode.on.net/pub/test/10meg.test"
$output = "$PSScriptRoot\10meg.test"
$start_time = Get-Date
Invoke-WebRequest -Uri $url -OutFile $output
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
Time taken: 27 seconds
Pros
With the cmdlet already available it is super easy to get started and use. Integration with Write-Progress is handy while watching paint dry scripts run (assuming you know the total file size). Cookies can also be persisted between mutiple requests through the use of the -Session and -WebSession parameters.
Cons
Speed. This cmdlet is slow. From what I have observed, the HTTP response stream is buffered into memory. Once the file has been fully loaded, it is flushed to disk. This adds a huge performance hit and potential memory issues for large files. If anyone knows specifics on how this cmdlet operates, let me know!.
Another potentially serious con for this method is the reliance on Internet Explorer. For example, this cmdlet cannot be used on Windows Server Core edition servers as the Internet Explorer binaries are not included by default. In some cases you can use the -UseBasicParsing parameter, but it does not work in all cases.
Verdict
This cmdlet shines when you need to persist cookies across multiple requests (for instance HTTP Forms Auth before downloading the file).
Performance is good enough for small downloads, but there are definitely better options for situations where speed is required. If the script is to be run on a server running Windows Server Core, choose a more universal method.
2. System.Net.WebClient
A common .NET class used for downloading files is the System.Net.WebClient class.
$url = "http://mirror.internode.on.net/pub/test/10meg.test"
$output = "$PSScriptRoot\10meg.test"
$start_time = Get-Date
$wc = New-Object System.Net.WebClient
$wc.DownloadFile($url, $output)
#OR
(New-Object System.Net.WebClient).DownloadFile($url, $output)
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
Time taken: 7 seconds
Pros
This method is also easy to use. Not as syntactically nice as Invoke-RestMethod - yet can still be executed on a single line. Speed is great as the HTTP response stream is buffered to disk throughout the download process.
There is also the option of System.Net.WebClient.DownloadFileAsync(). This can be very handy if you'd like your script to continue while the file downloads in parallel.
Cons
There is no visible progress indicator (or any way to query the progress mid transfer). It essentially blocks the thread until the download completes or fails. This isn't a major con, however sometimes it is handy to know how far through the transfer you are.
Verdict
System.Net.WebClient is my preferred option when file downloads are required. Anything that increases the performance of my scripts is a winner in my books. This method is also fully compatible on Server Core machines and 100% compatible with your Azure Automation runbooks.
3. Start-BitsTransfer
If you haven't heard of BITS before, check this out. BITS is primarily designed for asynchronous file downloads, but works perfectly fine synchronously too (assuming you have BITS enabled).
$url = "http://mirror.internode.on.net/pub/test/10meg.test"
$output = "$PSScriptRoot\10meg.test"
$start_time = Get-Date
Import-Module BitsTransfer
Start-BitsTransfer -Source $url -Destination $output
#OR
Start-BitsTransfer -Source $url -Destination $output -Asynchronous
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
Time taken: 6 seconds
Pros
This method proved to be the fastest in my test cases! Extensive integration with Write-Progress gives you a clear indicator of the file size and progress. The -Asynchronous flag can be used to queue transfers asychronously. This method is also incredibly flexible supporting separate credentials for the destination server AND web proxy, if required.
Personally, the biggest benefit to using the Start-BitsTransfer method is the ability to set retry actions on failure and limiting the amount of bandwidth available to a transfer.
Cons
While BITS is enabled by default on many machines, you can't guarantee it is enabled on all (unless you are actively managing this). Also with the way BITS is designed, if other BITS jobs are running in the background, your job could be queued or run at a later time hindering the execution of your script.
Verdict
This method is perfect for scenarios where you want to limit the bandwidth used in a file download or where time isn't a major issue. I have used this to sync files nightly at full speed and during the day at half speed using Transfer Policies. BITS is also easy to monitor and audit.
Conclusion
We can see the obvious/easiest choice isn't always the best. I would recommend System.Net.WebClient due to it's universal nature and performance. BITS my second choice due to it's flexibility and managability.
Do you know other methods? Let me know!
https://blog.jourdant.me/post/3-ways-to-download-files-with-powershell
3 ways to download files with PowerShell的更多相关文章
- Asp.net core 学习笔记 ( upload/download files 文件上传与下载 )
更新 : 2018-01-22 之前漏掉了一个 image 优化, 就是 progressive jpg refer : http://techslides.com/demos/progressi ...
- 安装Laravel遇到You must enable the openssl extension to download files via https问题
刚看了一篇文章说了2014年最火的10个php框架,看到了Laravel,于是便自己试试,孰料刚安装便遇到了一个问题(由于一不小心关掉了cmd,此处无法截图显示),便是如文章标题中所说的那样,goog ...
- [AWS - EC2] 如何向 Amazon Linux 2 实例传输文件,下载文件。How to send/ download files from Amazon Linux 2 Instance
1. 需要: 安装 WinSCP 2. 需要: PuTTY 生成的ppk格式密钥, 没有的话请移步此文章,完成1, 2, 3步即可. 3. 打开 WinSCP , 如果提示已经有PuTTY配置是否导入 ...
- How To Use XDOLoader to Manage, Download and Upload Files? (文档 ID 469585.1)
Applies to: BI Publisher (formerly XML Publisher) - Version 5.6.3 to 5.6.3 [Release 5] Information ...
- How To Use XDOLoader to Manage, Download and Upload Files? (DOC ID 469585.1)
In this Document Goal Fix Downloading Files Uploading Files References Applies to: BI Publishe ...
- SharePoint 2013 – Workflow Manager 1.0 offline download
[http://sharepointdeal.wordpress.com/2013/03/13/sharepoint-2013-workflow-manager-1-0-offline-downloa ...
- How To Download Youtube Videos Without any software
https://www.quora.com/What-is-the-best-way-to-download-YouTube-videos-for-free There are various met ...
- Download/Attach source-code/java-docs with maven dependencies
I am using Maven in my projects from last couple of years, and the automatically downloading the Jar ...
- 通过PowerShell命令给Azure VM添加CustomScriptExtension
Azure的VM提供了一种管理工具叫Azure VM Extension.它实现了一些管理虚拟机所需要的重要功能,比如:重设密码.设置RDP参数.以及许多其他关键的功能,并且Azure VM一直在添加 ...
随机推荐
- 卡特兰数(Catalan)及其应用
卡特兰数 大佬博客https://blog.csdn.net/doc_sgl/article/details/8880468 卡特兰数是组合数学中一个常出现在各种计数问题中出现的数列. 卡特兰数前几项 ...
- Java 学习笔记 ------第三章 基础语法
本章学习目标: 认识类型与变量 学习运算符的基本使用 了解类型转换细节 运用基本流程语法 一.类型(基本类型) 所谓基本类型,就是在使用时,得考虑一下数据用多少内存长度存比较经济,利用程序语法告诉JV ...
- 《JavaScript》web客户端存储
Web存储: 兼容IE8在内的所有主流浏览器,不兼容早期浏览器:支持大容量但非无限量 LocalStorage和sessionStorage是window对象的两个属性,这两个属性都代表同一个stor ...
- 缓存-System.Web.Caching.Cache
实现 Web 应用程序的缓存. 每个应用程序域创建一个此类的实例,只要应用程序域将保持活动状态,保持有效. 有关此类的实例的信息,请通过Cache的属性HttpContext对象或Cache属性的Pa ...
- Sql server 中关闭ID自增字段(SQL取消ID自动增长)
sql server在导入数据的时候,有时候要考虑id不变,就要先取消自动增长再导入数据,导完后恢复自增. 比如网站改版从旧数据库导入新数据库,数据库结构不相同,可能会使用insert into xx ...
- inotify 工具 是一种强大的、细粒度的、异步文件系统监控机制
前言:Inotify是一种强大的.细粒度的.异步文件系统监控机制,它满足各种各样的文件监控需要,可以监控文件系统的访问属性.读写属性.权限属性.删除创建.移动等操作,也就是可以监控文件发生的一切变化. ...
- DELL服务器PXE前期处理
thaks:https://www.cnblogs.com/520ZXL/ PXE批量推系统,服务器要具备条件:raid处理好,设置为pxe启动,与PXE服务器网络要通 先进入磁盘阵列(ctrl+R) ...
- php中扩展pecl与pear
要为大家分享的内容是PECL 和 PEAR 他们之间的不同和相同之处. PEAR 是“PHP Extension and Application Repository”的缩写,即PHP扩展和应用仓库. ...
- HDU4803_Poor Warehouse Keeper
题目很有意思,我想说其实我在比赛的时候就看过了一下这个题目,今天才这么快搞出来吧. 其实总共按上键的次数不会超过10个,我们可以每次假设相隔按两次上键之间按了xi次下键,由于上键的次数是确定的,所以最 ...
- 【Java】SVN下载maven项目到eclipse之后,项目红叉,pom.xml出现Missing artifact fakepath:dubbo:jar:2.8.5等缺少jar包情况
刚入公司,从svn上把代码弄下来之后导入eclipse,一般是maven项目,往往项目都会有红叉.如果排除代码本身问题,一般是jar包没有. 鼠标点开pom.xml文件,在约束那里一般有红叉,鼠标放上 ...