volcano-sdk-js/docs/storage.md at main · Kong/volcano-sdk-js

title	Storage
description	Volcano Storage provides secure file storage with access control. Upload user avatars, documents, media files, and more with built-in security policies.

Volcano Storage provides secure file storage with access control. Upload user avatars, documents, media files, and more with built-in security policies.

When the current session has a usable refresh token, authenticated storage requests refresh it after an HTTP 401 and retry the request once. Upload retries preserve the original file bytes. Network failures do not trigger an upload retry.

Overview

The storage module offers:

File Operations - Upload, download, list, delete, move, and copy files
Access Control - Private by default, with per-file public/private settings
Resumable Uploads - Handle large files reliably with chunked uploads
Bucket Organization - Organize files into logical buckets

Basic Concepts

Buckets

Files are organized into buckets. A bucket is a top-level container, similar to a folder. You might have buckets like:

avatars - User profile pictures
documents - PDF reports and files
uploads - General user uploads

Buckets are created in the Volcano dashboard and referenced by name in the SDK.

Paths

Within a bucket, files are identified by their path. Paths can include subdirectories:

avatars/user-123/profile.jpg
documents/reports/2024/q1-summary.pdf
uploads/images/photo-001.png

Path Rules

The SDK sends the bucket name and path as URL path segments. URL parsers resolve . and .. segments, so a path such as ../../functions/x would leave the storage API. Every storage method checks the bucket name and each object path before it sends anything. When a check fails, the method returns its usual error result, such as { data: null, error }, without contacting Volcano. The list() prefix is a query parameter, which URL parsing does not resolve, so these rules do not apply to it.

A path is refused when it:

is empty or is not a string
has a . or .. segment, such as ../secret.txt or reports/./q1.pdf
has an empty segment: a trailing /, a repeated //, or more than one leading /
has a lone UTF-16 surrogate, which cannot be percent-encoded

One leading / is allowed and dropped, so /avatars/user-123.jpg and avatars/user-123.jpg name the same object in every method that takes a path. Object keys never start with /, so pass list() a prefix without one.

A dot inside a segment is part of the name, as in .hidden/notes.txt or v1.2/notes.txt. Reserved characters such as %, ?, # and \ are percent-encoded, so a literal %2e%2e can never act as a parent folder. Volcano also refuses any object key that contains .., even inside a segment.

A bucket name is refused when it is blank, is . or .., contains /, or has a lone UTF-16 surrogate. Numbers, bigints and booleans are converted to strings; other non-string values are refused. The rules cover move() and copy() paths and every path passed to remove().

const { data, error } = await volcano.storage.from('documents').download('../secret.txt');
// data: null
// error.message: 'Storage path cannot contain empty, ".", or ".." segments'

Access Control

Files are private by default. Private files require authentication to download. You can make individual files public, allowing anyone to access them via a public URL.

Selecting a Bucket

All storage operations start by selecting a bucket:

const avatars = volcano.storage.from('avatars');
const documents = volcano.storage.from('documents');

This returns a StorageFileApi object with methods for file operations.

Uploading Files

Basic Upload

Upload a file from an <input type="file"> element:

const fileInput = document.querySelector('input[type="file"]');
const file = fileInput.files[0];

const { data, error } = await volcano.storage.from('avatars').upload('user-123/profile.jpg', file);

if (error) {
  console.error('Upload failed:', error.message);
  return;
}

console.log('Uploaded:', data.name);
console.log('Size:', data.size, 'bytes');
console.log('Type:', data.mime_type);

Upload with Options

Specify content type:

const { data, error } = await volcano.storage.from('documents').upload('reports/annual-2024.pdf', file, {
  contentType: 'application/pdf',
});

Upload from Blob or ArrayBuffer

You can upload any binary data:

// From Blob
const blob = new Blob(['Hello, World!'], { type: 'text/plain' });
await volcano.storage.from('uploads').upload('notes/hello.txt', blob);

// From ArrayBuffer
const buffer = await fetchSomeData();
await volcano.storage.from('uploads').upload('data/export.bin', buffer, {
  contentType: 'application/octet-stream',
});

Upload Path Patterns

Organize files with meaningful paths:

// User-specific files
const userId = user.id;
await storage.upload(`${userId}/avatar.jpg`, file);
await storage.upload(`${userId}/documents/${docId}.pdf`, file);

// Date-organized files
const date = new Date().toISOString().split('T')[0];
await storage.upload(`uploads/${date}/${file.name}`, file);

Downloading Files

Basic Download

const { data: blob, error } = await volcano.storage.from('documents').download('reports/annual-2024.pdf');

if (error) {
  console.error('Download failed:', error.message);
  return;
}

// Create a download link
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = 'annual-2024.pdf';
a.click();

// Clean up
URL.revokeObjectURL(url);

Display in Browser

For images and other displayable content:

const { data: blob } = await volcano.storage.from('avatars').download('user-123/profile.jpg');

const url = URL.createObjectURL(blob);
document.getElementById('avatar').src = url;

Partial Download (Range Requests)

Download only a portion of a file:

// Download first 1KB
const { data: blob } = await volcano.storage.from('documents').download('large-file.zip', {
  range: 'bytes=0-1023',
});

This is useful for streaming media or resuming interrupted downloads.

Listing Files

List All Files in Bucket

const { data: files, error } = await volcano.storage.from('uploads').list();

files.forEach((file) => {
  console.log(`${file.name} - ${file.size} bytes`);
});

List with Prefix

Filter files by path prefix:

// List files in a specific "folder"
const { data: files } = await volcano.storage.from('uploads').list('images/');

// List user's files
const { data: files } = await volcano.storage.from('documents').list(`${userId}/`);

Paginated Listing

For buckets with many files:

const { data: files, nextCursor } = await volcano.storage.from('uploads').list('', { limit: 100 });

console.log(`Found ${files.length} files`);

if (nextCursor) {
  // Fetch next page
  const { data: moreFiles } = await volcano.storage.from('uploads').list('', { limit: 100, cursor: nextCursor });
}

File Metadata

Listed files include metadata:

const { data: files } = await volcano.storage.from('uploads').list();

files.forEach((file) => {
  console.log('Name:', file.name);
  console.log('Size:', file.size);
  console.log('Type:', file.mime_type);
  console.log('Public:', file.is_public);
  console.log('Created:', file.created_at);
  console.log('Owner:', file.owner_id);

  if (file.is_public && file.public_url) {
    console.log('Public URL:', file.public_url);
  }
});

Deleting Files

Delete Single File

const { data, error } = await volcano.storage.from('uploads').remove('old-file.txt');

if (!error) {
  console.log('File deleted');
}

Delete Multiple Files

const { data, error } = await volcano.storage
  .from('uploads')
  .remove(['temp/file1.txt', 'temp/file2.txt', 'temp/file3.txt']);

if (data) {
  console.log(`Deleted ${data.deleted.length} files`);
}

Moving and Copying Files

Move (Rename)

const { data, error } = await volcano.storage.from('documents').move('drafts/report.pdf', 'published/report.pdf');

if (data) {
  console.log('File moved to:', data.name);
}

Copy

const { data, error } = await volcano.storage.from('documents').copy('templates/invoice.pdf', 'invoices/2024-001.pdf');

if (data) {
  console.log('File copied to:', data.name);
}

Public and Private Files

Default: Private

Files are private by default. They require an authenticated user session to download.

Make File Public

const { data, error } = await volcano.storage.from('avatars').updateVisibility('user-123/profile.jpg', true);

if (data) {
  console.log('File is now public');
  console.log('Public URL:', data.public_url);
}

Make File Private

const { error } = await volcano.storage.from('avatars').updateVisibility('user-123/profile.jpg', false);

if (!error) {
  console.log('File is now private');
}

Public URLs

Public files have a URL that anyone can access without authentication:

// After making a file public
const { data } = await volcano.storage.from('avatars').updateVisibility('profile.jpg', true);

// Use the public URL
const publicUrl = data.public_url;
// https://api.yourproject.volcano.dev/public/project-id/avatars/profile.jpg

// Use in HTML
<img src={publicUrl} alt="Avatar" />;

// Share via email, embed in external sites, etc.

The public URL:

Requires no authentication
Works in any browser
Can be cached by CDNs
Returns 404 if the file is made private later

Get Public URL

For files you know are public, get their URL directly:

const { data, error } = volcano.storage.from('avatars').getPublicUrl('profile.jpg');

if (data) {
  console.log('Public URL:', data.publicUrl);
}

Note: This constructs the URL locally and doesn't verify the file is actually public. Use list() or updateVisibility() to get the server-confirmed URL. The path must follow the path rules.

Resumable Uploads

For large files (over 100MB) or unreliable connections, use resumable uploads.

Simple Resumable Upload

The uploadResumable method handles everything automatically:

const { data, error } = await volcano.storage.from('uploads').uploadResumable('large-video.mp4', file, {
  onProgress: (uploaded, total) => {
    const percent = Math.round((uploaded / total) * 100);
    console.log(`Progress: ${percent}%`);
  },
});

if (data) {
  console.log('Upload complete:', data.object.name);
}

Manual Resumable Upload

For more control, manage the upload session yourself:

// 1. Create upload session
const { data: session } = await volcano.storage.from('uploads').createUploadSession('large-video.mp4', {
  totalSize: file.size,
  contentType: 'video/mp4',
  partSize: 10 * 1024 * 1024, // 10MB parts
});

console.log(`Session created: ${session.session_id}`);
console.log(`Will upload in ${session.total_parts} parts`);

// 2. Upload each part
for (let i = 1; i <= session.total_parts; i++) {
  const start = (i - 1) * session.part_size;
  const end = Math.min(start + session.part_size, file.size);
  const partData = file.slice(start, end);

  const { error } = await volcano.storage
    .from('uploads')
    .uploadPart('large-video.mp4', session.session_id, i, partData);

  if (error) {
    // Retain the session ID to resume the failed part later.
    throw error;
  }

  console.log(`Part ${i}/${session.total_parts} uploaded`);
}

// 3. Complete the upload
const { data, error } = await volcano.storage
  .from('uploads')
  .completeUploadSession('large-video.mp4', session.session_id);

if (data) {
  console.log('Upload complete!', data.object.name);
}

Resume Interrupted Upload

createUploadSession() returns session_id, part_size, total_parts, and expires_at. Read path and total_size from getUploadSession(), not the creation response.

If an upload is interrupted, you can resume it later:

// Check session status
const { data: status } = await volcano.storage.from('uploads').getUploadSession('large-video.mp4', sessionId);

console.log(`${status.parts_uploaded}/${status.total_parts} parts uploaded`);
console.log(`${status.bytes_uploaded}/${status.total_size} bytes`);
const uploadedParts = new Set(status.parts.map((part) => part.part_number));
const missingParts = Array.from({ length: status.total_parts }, (_, index) => index + 1).filter(
  (partNumber) => !uploadedParts.has(partNumber),
);
console.log('Missing parts:', missingParts);

// Upload only the missing parts
for (const partNumber of missingParts) {
  const start = (partNumber - 1) * status.part_size;
  const end = Math.min(start + status.part_size, file.size);
  const partData = file.slice(start, end);

  await volcano.storage.from('uploads').uploadPart('large-video.mp4', sessionId, partNumber, partData);
}

// Complete when all parts are uploaded
await volcano.storage.from('uploads').completeUploadSession('large-video.mp4', sessionId);

Abort Upload

Cancel an in-progress upload and discard its parts without publishing an object. Further session-status requests report not found:

const { error } = await volcano.storage.from('uploads').abortUploadSession('large-video.mp4', sessionId);

if (!error) {
  console.log('Upload cancelled');
}

Resumable Upload Limits

Minimum part size: 5 MB
Maximum part size: 25 MB (default)
Maximum parts: 10,000
Session expiry: 7 days

Error Handling

Storage operations return errors rather than throwing:

const { data, error } = await volcano.storage.from('uploads').upload('file.txt', file);

if (error) {
  // Common errors:
  // - "Storage path ..." or "Bucket name ..." - Breaks the path rules; nothing was sent
  // - "No active session" - User not signed in
  // - "File not found" - File doesn't exist
  // - "Permission denied" - Access policy violation
  // - "Bucket not found" - Invalid bucket name
  // - "File too large" - Exceeds size limit

  console.error('Upload failed:', error.message);
}

Access Policies

Storage access is controlled by policies defined in the Volcano dashboard. Typical patterns:

User's Own Files

Users can only access files in their own "folder":

// Policy: path starts with user's ID
await storage.upload(`${user.id}/avatar.jpg`, file); // Allowed
await storage.upload(`other-user-id/avatar.jpg`, file); // Denied

Public Read, Authenticated Write

Anyone can download, but only authenticated users can upload:

// Download works without auth (if file is public)
await storage.download('public/logo.png');

// Upload requires authentication
await storage.upload('public/new-file.png', file);

Role-Based Access

Admins can access all files, regular users have restrictions:

// Policy checks user's role from JWT
// Admin: full access
// User: only own files

Best Practices

Use Meaningful Paths

Organize files with structured paths:

// Good
`users/${userId}/documents/${year}/${filename}``projects/${projectId}/assets/${assetType}/${filename}`
// Avoid
`file-${Date.now()}.pdf``${Math.random()}.jpg`;

Validate Before Upload

Check file type and size client-side:

function validateFile(file) {
  const maxSize = 10 * 1024 * 1024; // 10MB
  const allowedTypes = ['image/jpeg', 'image/png', 'image/gif'];

  if (file.size > maxSize) {
    return 'File too large (max 10MB)';
  }

  if (!allowedTypes.includes(file.type)) {
    return 'Invalid file type';
  }

  return null;
}

const error = validateFile(file);
if (error) {
  showError(error);
  return;
}

await storage.upload('images/photo.jpg', file);

Handle Large Files

Use resumable uploads for files over 100MB:

const useResumable = file.size > 100 * 1024 * 1024;

if (useResumable) {
  await storage.uploadResumable(path, file, { onProgress });
} else {
  await storage.upload(path, file);
}

Clean Up Object URLs

When creating URLs from blobs, revoke them when done:

const url = URL.createObjectURL(blob);
img.src = url;

// Later, when the image is no longer needed:
URL.revokeObjectURL(url);

Show Upload Progress

For better UX with large files:

await volcano.storage.from('uploads').uploadResumable('video.mp4', file, {
  onProgress: (uploaded, total) => {
    const percent = Math.round((uploaded / total) * 100);
    progressBar.style.width = `${percent}%`;
    progressText.textContent = `${percent}%`;
  },
});

Next Steps

Database - Store file metadata in your database
Realtime - Get notified when files are uploaded
Functions - Process files with serverless functions

Storage HTTP errors preserve the response status as error.status and any server error code as error.code. An aborted or missing upload session returns 404. completeUploadSession() and uploadResumable() return completed metadata inside data.object; a single-request upload() returns it directly in data.

For remove(), top-level error metadata describes the first failed path. error.failures preserves each failed path and its original error; data.deleted contains the paths successfully removed. Local validation errors have no failure list. If any path in the list breaks the path rules, remove() deletes nothing and returns { data: null, error }.

FilesExpand file tree

storage.md

Latest commit

History