【Azure Developer】使用REST API获取Activity Logs、传入Data Lake的数据格式问题

问题一：.  如何在用REST API获取活动日志时，控制输出的项？

【答】参考REST API对于获取活动日志的说明接口，在参数是$filter和$select中可以分别控制过滤条件和输出项

GET https://management.chinacloudapi.cn/subscriptions/{subscriptionId}/providers/microsoft.insights/eventtypes/management/values?api-version=2015-04-01&$filter={$filter}&$select={$select}

注：management.chinacloudapi.cn为中国区Azure的管理终结点，management.azure.com为全球Azure的管理终结点

URI Parameters

Name	Description
$filter	Reduces the set of data collected. This argument is required and it also requires at least the start date/time. The $filter argument is very restricted and allows only the following patterns. - List events for a resource group: $filter=eventTimestamp ge '2014-07-16T04:36:37.6407898Z' and eventTimestamp le '2014-07-20T04:36:37.6407898Z' and resourceGroupName eq 'resourceGroupName'. - List events for resource: $filter=eventTimestamp ge '2014-07-16T04:36:37.6407898Z' and eventTimestamp le '2014-07-20T04:36:37.6407898Z' and resourceUri eq 'resourceURI'. - List events for a subscription in a time range: $filter=eventTimestamp ge '2014-07-16T04:36:37.6407898Z' and eventTimestamp le '2014-07-20T04:36:37.6407898Z'. - List events for a resource provider: $filter=eventTimestamp ge '2014-07-16T04:36:37.6407898Z' and eventTimestamp le '2014-07-20T04:36:37.6407898Z' and resourceProvider eq 'resourceProviderName'. - List events for a correlation Id: $filter=eventTimestamp ge '2014-07-16T04:36:37.6407898Z' and eventTimestamp le '2014-07-20T04:36:37.6407898Z' and correlationId eq 'correlationID'. NOTE: No other syntax is allowed.
$select	Used to fetch events with only the given properties. The $select argument is a comma separated list of property names to be returned. Possible values are: authorization, claims, correlationId, description, eventDataId, eventName, eventTimestamp, httpRequest, level, operationId, operationName, properties, resourceGroupName, resourceProviderName, resourceId, status, submissionTimestamp, subStatus, subscriptionId

在Filter和select可以配置的每一项说明,请查看文末的附录一。

问题二：是否可以往Data Lake中注入Json格式文件？

【答】根据官方文档示例，如果使用Databricks进行（SQL查询语句）进行数据分析，是支持Json格式的数据，示例中使用的Json格式如下图所示：

参考使用 Databricks 分析数据一文: https://docs.azure.cn/zh-cn/storage/blobs/data-lake-storage-quickstart-create-databricks-account#ingest-sample-data

问题三：如何用C#向Data Lake中插入数据文件？

【答】可以，需要先安装 Azure.Storage.Files.DataLake NuGet 包。然后添加如下引用

using Azure.Storage.Files.DataLake;
using Azure.Storage.Files.DataLake.Models;
using Azure.Storage;
using System.IO;
using Azure;

将文件上传到目录，则参考以下代码即可：

• 通过创建 DataLakeFileClient 类的实例，在目标目录中创建文件引用。
• 通过调用 DataLakeFileClient.AppendAsync 方法上传文件。
• 确保通过调用 DataLakeFileClient.FlushAsync 方法完成上传。

public async Task UploadFile(DataLakeFileSystemClient fileSystemClient)
{
    DataLakeDirectoryClient directoryClient =
        fileSystemClient.GetDirectoryClient("my-directory");

    DataLakeFileClient fileClient = await directoryClient.CreateFileAsync("uploaded-file.txt");

    FileStream fileStream =
        File.OpenRead("C:\\file-to-upload.txt");

    long fileSize = fileStream.Length;

    await fileClient.AppendAsync(fileStream, offset: 0);

    await fileClient.FlushAsync(position: fileSize);

}

完整实例参考使用 .NET 管理 Azure Data Lake Storage Gen2：https://docs.azure.cn/zh-cn/storage/blobs/data-lake-storage-directory-file-acl-dotnet#upload-a-file-to-a-directory

附录一：在$Filter和$select可配置的选择项的EventData属性

The Azure event log entries are of type EventData

Name	Type	Description
authorization	SenderAuthorization	The sender authorization information.
caller	string	the email address of the user who has performed the operation, the UPN claim or SPN claim based on availability.
category	LocalizableString	the event category.
claims	object	key value pairs to identify ARM permissions.
correlationId	string	the correlation Id, usually a GUID in the string format. The correlation Id is shared among the events that belong to the same uber operation.
description	string	the description of the event.
eventDataId	string	the event data Id. This is a unique identifier for an event.
eventName	LocalizableString	the event name. This value should not be confused with OperationName. For practical purposes, OperationName might be more appealing to end users.
eventTimestamp	string	the timestamp of when the event was generated by the Azure service processing the request corresponding the event. It in ISO 8601 format.
httpRequest	HttpRequestInfo	the HTTP request info. Usually includes the 'clientRequestId', 'clientIpAddress' (IP address of the user who initiated the event) and 'method' (HTTP method e.g. PUT).
id	string	the Id of this event as required by ARM for RBAC. It contains the EventDataID and a timestamp information.
level	EventLevel	the event level
operationId	string	It is usually a GUID shared among the events corresponding to single operation. This value should not be confused with EventName.
operationName	LocalizableString	the operation name.
properties	object	the set of <Key, Value> pairs (usually a Dictionary<String, String>) that includes details about the event.
resourceGroupName	string	the resource group name of the impacted resource.
resourceId	string	the resource uri that uniquely identifies the resource that caused this event.
resourceProviderName	LocalizableString	the resource provider name of the impacted resource.
resourceType	LocalizableString	the resource type
status	LocalizableString	a string describing the status of the operation. Some typical values are: Started, In progress, Succeeded, Failed, Resolved.
subStatus	LocalizableString	the event sub status. Most of the time, when included, this captures the HTTP status code of the REST call. Common values are: OK (HTTP Status Code: 200), Created (HTTP Status Code: 201), Accepted (HTTP Status Code: 202), No Content (HTTP Status Code: 204), Bad Request(HTTP Status Code: 400), Not Found (HTTP Status Code: 404), Conflict (HTTP Status Code: 409), Internal Server Error (HTTP Status Code: 500), Service Unavailable (HTTP Status Code:503), Gateway Timeout (HTTP Status Code: 504)
submissionTimestamp	string	the timestamp of when the event became available for querying via this API. It is in ISO 8601 format. This value should not be confused eventTimestamp. As there might be a delay between the occurrence time of the event, and the time that the event is submitted to the Azure logging infrastructure.
subscriptionId	string	the Azure subscription Id usually a GUID.
tenantId	string	the Azure tenant Id