将 CSV 文件上传到 azure 表存储不会上传所有项目

Question

我正在尝试通过 powershell 脚本将 CSV 文件上传到 Azure 表存储。 CSV 有 4776 条记录，但实际上只有大约 3042 项迁移到 Azure 表存储。我做错了什么/可以采取什么不同的做法？这是我的代码：

function Add-Entity()
{
 [CmdletBinding()]

 param
 (
    $table, 
    [string] $partitionKey, 
    [string] $rowKey, 
    [string] $propOne,
    [string] $proptwo

 )
 $assemblySN = $table.CloudTable.GetType().Assembly.FullName
 $entity = New-Object -TypeName "Microsoft.WindowsAzure.Storage.Table.DynamicTableEntity,$assemblySN" -ArgumentList $partitionKey, $rowKey
 $entity.Properties.Add("FirstProperty",$propOne)
 $entity.Properties.Add("SecondProperty",$propTwo)

 $result = $table.CloudTable.ExecuteAsync((invoke-expression "[Microsoft.WindowsAzure.Storage.Table.TableOperation,$assemblySN]::InsertOrReplace(`$entity)"))
}

Clear-Host
$subscriptionName = "mysubscription" 
$resourceGroupName = "myrg"
$storageAccountName = "mystorage" 
$location = "East US"
$tableName = "mytable"

# Log on to Azure and set the active subscription
Select-AzureRmSubscription -SubscriptionName $subscriptionName

# Get the storage key for the storage account
$storageAccountKey = (Get-AzureRmStorageAccountKey -ResourceGroupName $resourceGroupName -Name $storageAccountName).Value[0]

# Get a storage context
$ctx = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey

# Get a reference to the table
$table = Get-AzureStorageTable -Name $tableName -Context $ctx 

$csv = Import-CSV <file path>

ForEach ($line in $csv)   
{
 Add-Entity -Table $table -partitionKey $line.Partitionkey -rowKey $line.Rowkey -FirstProperty $line.propOne -SecondProperty $line.proptwo 
}

Answer 1

如果您确定 .csv 文件中没有 [Partitionkey+Rowkey] 的复制，那么可能的原因是您的

invalid characters

中存在一些

partition key or row key

。通过使用您的脚本，如果表中包含无效字符，则这些项目将不会添加到表中。

请按照此

doc

检查您的

partition key

或row key中是否包含以下字符。

无效字符截图：

Answer 2

根据我的测试，我们可以使用以下PowerShell脚本将csv文件导入到Azure表存储中。

$storageAccountKey = (Get-AzureRmStorageAccountKey -ResourceGroupName $resourceGroupName -Name $storageAccountName).Value[0]
$ctx = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey
$table = Get-AzureStorageTable -Name $tableName -Context $ctx 

$CsvContents = Import-Csv -Path $Path
$CsvHeaders = ($CsvContents[0] | Get-Member -MemberType NoteProperty).Name | Where{$_ -ne "RowKey" -and $_ -ne "PartitionKey"}

Foreach($CsvContent in $CsvContents)
{
    $PartitionKey = $CsvContent.PartitionKey
    $RowKey = $CsvContent.RowKey
    $Entity = New-Object "Microsoft.WindowsAzure.Storage.Table.DynamicTableEntity" "$PartitionKey", "$RowKey"

    Foreach($CsvHeader in $CsvHeaders)
    {
        $Value = $CsvContent.$CsvHeader
        $Entity.Properties.Add($CsvHeader, $Value)
    }
    Write-Verbose "Inserting the entity into table storage."
    $result = $Table.Execute([Microsoft.WindowsAzure.Storage.Table.TableOperation]::Insert($Entity))
}

Answer 3

为了子孙后代...谢谢你提出这个问题。它帮助了我。

某些任务未进入存储帐户的原因可能是因为异步任务尚未完成。

当您调用

$table.CloudTable.ExecuteAsync

时，它会返回一个必须等待的任务。您可以使用

$result.GetAwaiter().GetResult()

等待它，我认为发生的情况是最后几个任务在进程关闭之前尚未完成。

这是我的实现供参考。我可以弄清楚如何自动创建类型，所以我只是切换了它。

function Add-Entity() {
    [CmdletBinding()]
    param
    (
        $table, 
        $data
    )

    $assembly = $table.CloudTable.GetType().Assembly.FullName

    $entity = New-Object -TypeName "Microsoft.WindowsAzure.Storage.Table.DynamicTableEntity,$assembly" -ArgumentList $data.PartitionKey, $data.RowKey
    $data | Get-Member -MemberType NoteProperty | ? Name -NotLike '*@*' | ? Name -NE PartitionKey | ? Name -NE RowKey | ? { $data."$($_.Name)@type" } | % {
        $type = $data."$($_.Name)@type"
        $name = $_.Name
        $value = $data.$($_.Name)

        $property = switch ($type) {
            "String" { New-Object -TypeName "Microsoft.WindowsAzure.Storage.Table.EntityProperty,$assembly" -ArgumentList ([string]$value) }
            "Boolean" { New-Object -TypeName "Microsoft.WindowsAzure.Storage.Table.EntityProperty,$assembly" -ArgumentList ([bool]$value) }
            "DateTime" { New-Object -TypeName "Microsoft.WindowsAzure.Storage.Table.EntityProperty,$assembly" -ArgumentList ([DateTime]::Parse($value)) }
            "Double" { New-Object -TypeName "Microsoft.WindowsAzure.Storage.Table.EntityProperty,$assembly" -ArgumentList ([Double]$value) }
            "Guid" { New-Object -TypeName "Microsoft.WindowsAzure.Storage.Table.EntityProperty,$assembly" -ArgumentList ([Guid]::Parse($value)) }
            "Int32" { New-Object -TypeName "Microsoft.WindowsAzure.Storage.Table.EntityProperty,$assembly" -ArgumentList ([Int32]$value) }
            "Int64" { New-Object -TypeName "Microsoft.WindowsAzure.Storage.Table.EntityProperty,$assembly" -ArgumentList ([Int64]$value) }
            "Binary" { throw "nope" }
            Default { throw "Unknown Type '$type" }
        }

        $entity.Properties.Add($name, $property)
    }

    $task = $table.CloudTable.ExecuteAsync((Invoke-Expression "[Microsoft.WindowsAzure.Storage.Table.TableOperation,$assembly]::InsertOrReplace(`$entity)"))
    $task.GetAwaiter().GetResult()
}