403 and 400 errors using AWS S3 Multipart Upload

Addy R. Adeosun

Hello,

One of our software engineers is having some trouble with the AWS S3 PHP SDK. He's getting an error when trying to create a multipart uploader using this process: https://support.panopto.com/s/article/Upload-API

He's able to create an upload session, store the upload session info, and initialize the S3 client. It fails on the next step which is to imitate a multipart upload using the client. He's either getting a 400 or 403 error depending on the endpoint URL he passes to the S3 client:

Using https://temple.hosted.panopto.com/Panoptoas the endpoint results in a 400 bad request error and the S3 client outputs a URL that looks like this with /Panopto doubled: https://temple.hosted.panopto.com/Panopto/Panopto/Upload/22222222-2222-2222-2222-222222222222/filename.ext?uploads

Using https://temple.hosted.panopto.com/ as the endpoint, results in a 403 forbidden error and the S3 client outputs a URL like this with /Panopto not there at allhttps://temple.hosted.panopto.com/Upload/22222222-2222-2222-2222-222222222222/filename.ext?uploads

Neither of the URLs it outputs seem right and below is the code for this part which he based on Panopto's Python example:

Any thoughts and suggestions would be helpful!

$elements = explode('/', $upload_target);
$service_endpoint = implode('/', array_slice($elements, 0, -2));
$bucket = current(array_slice($elements, -2, 1));
$prefix = current(array_slice($elements, -1, 1));
$object_key = $prefix . '/' . $file_name;

// Create S3 client with custom endpoint
// Panopto doesn't use access key or secret, but the library needs some values or a no credentials error is thrown.
$s3 = new S3Client([
    'region' => 'us-east-2',
    'version' => '2006-03-01',
    'endpoint' => $service_endpoint,
    'credentials' => false
]);

// Initiate multipart upload
$mpu = $s3->createMultipartUpload([
    'Bucket' => $bucket,
    'Key' => $object_key
]);
$mpu_id = $mpu['UploadId'];

// Iterate through parts
$parts = [];
$uploaded_bytes = 0;
$file = fopen($file_path, 'rb');
$part_number = 1;
while(true) {
    $data = fread($file, $this->part_size);
    if(!strlen($data))
        break;
    $part = $s3->uploadPart(['Body' => $data, 'Bucket' => $bucket, 'Key' => $object_key, 'UploadId' => $mpu_id, 'PartNumber' => $part_number]);
    array_push($parts, ['PartNumber' => $part_number, 'ETag' => $part['ETag']]);
    $uploaded_bytes += strlen($data);
    //echo "  -- $uploaded_bytes of $file_size bytes uploaded";
    $part_number++;
}
fclose($file_path);

# Copmlete the upload
$result = $s3->completeMultipartUpload([
    'Bucket' => $bucket,
    'Key' => $object_key,
    'UploadId' => $mpu_id,
    'MultipartUpload' => [
        'Parts' => $parts
    ]
]);

Kevin Baum

Hi Addy,

The $service_endpoint should be https://temple.hosted.panopto.com/Panopto/ for your example. Is it possible that the bucket is accidentally including "/Panopto" and adding that to the URL?

I would suggest temporarily hard-coding the $service_endpoint to https://temple.hosted.panopto.com/Panopto/ , and verify that the $bucket parameter is just set to Upload and see whether that changes anything. Please let me know if after hard-coding the service endpoint and verifying the bucket name you still continue to see an issue and I'll continue to look into it.

Please let me know if you have any other questions.

Thanks,

Kevin

Addy R. Adeosun

Thank you for the response Kevin!

This is the code he's using to split the upload target:

echo "upload target = $upload_target\n\r";
        $elements = explode('/', $upload_target);
        $service_endpoint = implode('/', array_slice($elements, 0, -2));
        $bucket = current(array_slice($elements, -2, 1));
        $prefix = current(array_slice($elements, -1, 1));
        $object_key = $prefix . '/' . $file_name;

        echo "endpoint = $service_endpoint\r\n";
        echo "bucket = $bucket\r\n";
        echo "prefix = $prefix\r\n";
        echo "object key = $object_key\r\n";

This is the output:

image.png

Kevin Baum

Hi Addy,

Can you try making sure that the service endpoint ends with a "/" so that the result is "https://temple.hosted.panopto.com/Panopto/"? The missing slash may be triggering it to add that again.

Please let me know if that doesn't work, or if you have any other questions.

Thanks,

Kevin

Addy R. Adeosun

Hi Kevin, this is the output he's getting after adding a "/" to the service endpoint:

Do you have any additional suggestions?

Kevin Baum

Hi Addy,

Thank you for trying that for me. I'm not sure off the top of my head what may be causing this, and may need to investigate it further. Would you be able to open a support ticket with Panopto, or ask your POC to open a support ticket on your behalf so I can look into this further for you?

Thanks,

Kevin

Addy R. Adeosun

Thank you, I put in a ticket yesterday. I appreciate your help!

Josh Mckibbin

Hi!

I wanted to comment here because I am having the same exact issue on the same exact server even. I'm just wondering if you ever came up with a solution. The weird thing is that I had this working previously, but after I updated the AWS SDK libraries, it broke.

If I make https://temple.hosted.panopto.com the service endpoint it omits the Panopto directory and I get this error:

Error executing "PutObject" on "https://temple.hosted.panopto.com/Upload/7ab4cc9c-c911-4fa7-b0a7-b0fc00fc9069/ba-2104-basic-computations-part-1.xml"

However, if I make https://temple.hosted.panopto.com/Panopto the service endpoint it doubles up the Panopto directory and I get this error:

Error executing "PutObject" on "https://temple.hosted.panopto.com/Panopto/Panopto/Upload/0abb414d-0f3d-4084-b656-b0fc01087b85/ba-2104-basic-computations-part-1.xml"

Thanks,

Josh

Josh Mckibbin

Hi again!

I was actually able to figure out the issue. It wasn't the $service_endpoint. It was the $bucket and $prefix.

Here is the relevant code that solved the problem:

$elements = explode('/', $upload_target);
$session_dir = array_pop($elements); // This is the session directory
$prefix = array_pop($elements) . '/' . $session_dir; // This is the upload location
$bucket = array_pop($elements); // This is the bucket: Panopto
$service_endpoint = implode('/', $elements);

$file_name = basename($file_path);
$object_key = $prefix . '/' . $file_name;

        $s3Client = new S3Client(array(
            'endpoint' => $service_endpoint,
            'region'  => 'us-east-2',
            'version' => '2006-03-01',
            'credentials' => false
        ));

In short, the bucket actually needed to be 'Panopto' and the prefix needed to be 'Upload/{SESSION DIRECTORY}'.

Hopefully this can help anyone else that may have this issue.