Read Parquet file from Azure blob with out downloading it locally c# .net(从Azure blob读取拼图文件，而无需将其下载到本地c#.net)

本文介绍了从Azure blob读取拼图文件，而无需将其下载到本地c#.net的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我们有一个拼花格式化文件(500MB)，它位于Azure BLOB中。如何直接从BLOB中读取该文件并将其保存在c#的内存中，例如：DataTable。

我可以使用以下代码读取实际位于文件夹中的镶木地板文件。

public void ReadParqueFile()
    {
         using (Stream fileStream = System.IO.File.OpenRead("D:/../userdata1.parquet"))     
        {
            using (var parquetReader = new ParquetReader(fileStream))
            {
                DataField[] dataFields = parquetReader.Schema.GetDataFields();

                for (int i = 0; i < parquetReader.RowGroupCount; i++)
                {

                    using (ParquetRowGroupReader groupReader = parquetReader.OpenRowGroupReader(i))
                    {
                        DataColumn[] columns = dataFields.Select(groupReader.ReadColumn).ToArray();

                        DataColumn firstColumn = columns[0];

                        Array data = firstColumn.Data;
                        //int[] ids = (int[])data;
                    }
                }
           }
        }

    }
}

(我可以使用SourestStream直接从BLOB读取CSV文件)。请推荐一种直接从BLOB读取镶木地板文件的最快方法

推荐答案

根据我的经验，直接从BLOB中读取拼图文件的解决方案是先用sas Token生成blob url，然后用sas从url中获取HttpClient的流，最后通过ParquetReader读取http响应流。

首先，请参考官方文档Create a service SAS for a container or blob with .NET使用Azure Blob Storage SDK for.NET Core的Create a service SAS for a blob部分下面的示例代码。

private static string GetBlobSasUri(CloudBlobContainer container, string blobName, string policyName = null)
{
    string sasBlobToken;

    // Get a reference to a blob within the container.
    // Note that the blob may not exist yet, but a SAS can still be created for it.
    CloudBlockBlob blob = container.GetBlockBlobReference(blobName);

    if (policyName == null)
    {
        // Create a new access policy and define its constraints.
        // Note that the SharedAccessBlobPolicy class is used both to define the parameters of an ad hoc SAS, and
        // to construct a shared access policy that is saved to the container's shared access policies.
        SharedAccessBlobPolicy adHocSAS = new SharedAccessBlobPolicy()
        {
            // When the start time for the SAS is omitted, the start time is assumed to be the time when the storage service receives the request.
            // Omitting the start time for a SAS that is effective immediately helps to avoid clock skew.
            SharedAccessExpiryTime = DateTime.UtcNow.AddHours(24),
            Permissions = SharedAccessBlobPermissions.Read | SharedAccessBlobPermissions.Write | SharedAccessBlobPermissions.Create
        };

        // Generate the shared access signature on the blob, setting the constraints directly on the signature.
        sasBlobToken = blob.GetSharedAccessSignature(adHocSAS);

        Console.WriteLine("SAS for blob (ad hoc): {0}", sasBlobToken);
        Console.WriteLine();
    }
    else
    {
        // Generate the shared access signature on the blob. In this case, all of the constraints for the
        // shared access signature are specified on the container's stored access policy.
        sasBlobToken = blob.GetSharedAccessSignature(null, policyName);

        Console.WriteLine("SAS for blob (stored access policy): {0}", sasBlobToken);
        Console.WriteLine();
    }

    // Return the URI string for the container, including the SAS token.
    return blob.Uri + sasBlobToken;
}

然后使用SAS令牌从URL获取HttpClient的http响应流。

var blobUrlWithSAS = GetBlobSasUri(container, blobName);
var client = new HttpClient();
var stream = await client.GetStreamAsync(blobUrlWithSAS);

最后通过ParquetReader读取，代码来自GitHub回购aloneguid/parquet-dotnet的Reading Data。

var options = new ParquetOptions { TreatByteArrayAsString = true };
var reader = new ParquetReader(stream, options);

这篇关于从Azure blob读取拼图文件，而无需将其下载到本地c#.net的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持编程学习网！